Process Multiple files:Process Each file in each run

krishna81 · Post by **krishna81** » Sun May 10, 2009 2:14 pm

I have a list of filenames in a file and i want to pass these file names in parameter and want to run my job for each file by keep in loop using job sequencer.

in each files header is record count and and following are data.
example:

file1timestamp.txt
-----------------------
3
1 rec1
2 rec2
3 rec3

file2timestamp.txt
--------------------
4
1 rec1
3 rec2
4 rec3
2 rec5

i need to run my job for each file.
i guess file pattern doesn't works in this scenario.And i need to process each file in each run.
Though i am invoking my DS job thru KSH, want to avoid loop step in my korn shell.
please advise me in this matter.

Thanks in Advance
kris

krishna81 · Post by **krishna81** » Sun May 10, 2009 2:19 pm

example:
file contains list of filenames

Filenamelist.txt
----------------
file1timestamp.txt
file2timestamp.txt

chulett · Post by **chulett** » Sun May 10, 2009 6:18 pm

So, what have you tried? The UserVariables stage can be used to fetch the list of filenames and then the Start Loop stage can take one entry off that list for each iteration of the loop and pass it as a job parameter to your processing job.

The only 'tricky' part would be building a routine that returns the filenames from your file in a delimited fashion that the stage supports. You could, for example, 'cat' the file capturing the output and then Convert() the delimiters from field marks to commas. Or perhaps script the "pivoting" of the names.

krishna81 · Post by **krishna81** » Mon May 11, 2009 7:29 am

what exactly i have to give in expression editor in UserVariables stage.

chulett · Post by **chulett** » Mon May 11, 2009 7:47 am

The name of the routine you wrote, as one example.