Looping through the files in directory

mab_arif16 · Post by **mab_arif16** » Fri May 05, 2006 10:33 am

I have a job which needs to loop through the files in a directory ,read all the files sequentially in the order which they were created,These files have a time stamp in their name .
Does any body has an idea about it.
Thanks for the help
Arif

ArndW · Post by **ArndW** » Fri May 05, 2006 10:56 am

There are at least two basic approaches. One way is to use a UNIX command one liner to get the filelist and cat all the results to stdin for the DataStage PX job. Another is to work through a DataStage Sequence and get the directory contents and call up your processing job once for each input file.

I like the Job Sequence method, but a PX job will allow you execute the input stream command easily. Also, you can use a file pattern to automatically load all the files in the sequetial stage, but you cannot order the files.

mab_arif16 · Post by **mab_arif16** » Fri May 05, 2006 11:14 am

Hi
Thanks for your help can you explain the sequence method in more detail ,how to get a list of files and how to how to loop the job processing the files.
Thanks
Arif

ray.wurlod · Post by **ray.wurlod** » Fri May 05, 2006 3:37 pm

The StartLoop activity can be set to process a "list of things". The list can be generated by any means, such as a regular expression or a script. Read the manual chapter or on-line help on StartLoop activity.

splayer · Post by **splayer** » Thu Dec 07, 2006 4:25 pm

ray, if you had a list of files in a folder, how would you go through the files to process them in the folder using this activity? According to the manual, you can hardcode values but you cannot specify *.txt in a folder or something like that.

ray.wurlod · Post by **ray.wurlod** » Thu Dec 07, 2006 7:49 pm

Use an Execute Command activity to generate a delimited list of file names.
Use its output as the delimited list in the Start Loop activity, and specify there that it's a "list of things" loop and the appropriate delimiter.
Downstream of that, the index variable ($Control, or something like that) will contain one file name that can be delivered as a parameter. Put in all your other required activities, bring them into an End Loop activity and link that back to the Start Loop activity to complete the loop. An output trigger (unconditional) from the End Loop activity is where control goes once the loop has been completed.