DSXchange

My Job is doing some processing on the input file provided by 40 sources.
I have to run my Job as soon as any of the files from these sources comes.the files can be arrived anytime,even when the processing of the previous is not being completed.
What we discussed is that we would maintain a sequential file having the names of all the sources,Control flag for each sources and a status to check each source file processing status.
Now i have to keep on looking at this sequential file.As soon as the first source puts it's control flag as ready then i have to take the file name of the files from that source into the Job parameters and start the process.
And when the processing get finish...i have to again look into the file and get the very next file names with the same logic.

Please tell me how we can read the Sequential file in this manner.

My idea was :

Start Loop Activity -> ****** -> Job -> End Loop
How i can achieve '*****' Activity.

Thanx and Regards
arun

Hi Arun,

Do you have one source file arriving into staging file-system? Or 40 source files from 40 sources? What is the frequency of source file(s)?

If you have n-number of source files arriving daily/weekly/monthly/yearly/etc., keep track of those incoming files into a status file (through a script) with the flag as you mentioned. Based on file status and priority, you kick your job one at a time. Immediately after finishing the job, change the status of the file to "Loaded/Finished" (Through before/after job routine scripts). Then kick the next one from the status file, based on priority.

Hi satish,
I have 40 sources giving 40 files.Each day each source will produce one file.So 40 files a day.Yes we are also planning to proceed in the same manner which u suggests..but i donot find how to read the sequential file in this manner.Like to find out the row which has it's status as ready.Hope you understand.

Thanx

Do each of these 40 files trigger 40 different jobs or could be the same job? Give us more details. What is this job, is it a single job or a batch. Are there any dependencies between the files?

If for all 40 files, you need to run a single job then this would be useful.

first create an empty file containing following columns - Filename and Date. Key columns will be Filename and Date.

Design the job that needs to be executed for all 40 files as multiple instance. So it should be enabled for multi-instance as well as should have features for multi instance.

Create one more server job which will have following design: .
Folder stage will read the directory where u will get all files. It will have only one column that will contain the filename. Using the filename and today's date do a lookup on hash file. If the lookup is not successful, call the datastage job designed to process files. Else do not do anything. Once the job call is successful, update the hash file with filename and today'ss date.

So what is happening here is in hash file u r maintain list of files processed and date. So if the lookup is successful means the file is processed today, else process the file and add one entry in hash file to indicate that it is processed.

Schedule this jobs on timesly basis and its done.