Job Sequence

theverma · Post by **theverma** » Mon Dec 11, 2006 11:39 am

My Job is doing some processing on the input file provided by 40 sources.
I have to run my Job as soon as any of the files from these sources comes.the files can be arrived anytime,even when the processing of the previous is not being completed.
What we discussed is that we would maintain a sequential file having the names of all the sources,Control flag for each sources and a status to check each source file processing status.
Now i have to keep on looking at this sequential file.As soon as the first source puts it's control flag as ready then i have to take the file name of the files from that source into the Job parameters and start the process.
And when the processing get finish...i have to again look into the file and get the very next file names with the same logic.

Please tell me how we can read the Sequential file in this manner.

My idea was :

Start Loop Activity -> ****** -> Job -> End Loop
How i can achieve '*****' Activity.

Thanx and Regards
arun

satish_valavala · Post by **satish_valavala** » Mon Dec 11, 2006 12:01 pm

Hi Arun,

Do you have one source file arriving into staging file-system? Or 40 source files from 40 sources? What is the frequency of source file(s)?

If you have n-number of source files arriving daily/weekly/monthly/yearly/etc., keep track of those incoming files into a status file (through a script) with the flag as you mentioned. Based on file status and priority, you kick your job one at a time. Immediately after finishing the job, change the status of the file to "Loaded/Finished" (Through before/after job routine scripts). Then kick the next one from the status file, based on priority.

theverma · Post by **theverma** » Mon Dec 11, 2006 12:26 pm

Hi satish,
I have 40 sources giving 40 files.Each day each source will produce one file.So 40 files a day.Yes we are also planning to proceed in the same manner which u suggests..but i donot find how to read the sequential file in this manner.Like to find out the row which has it's status as ready.Hope you understand.

Thanx

DSguru2B · Post by **DSguru2B** » Mon Dec 11, 2006 1:12 pm

Do each of these 40 files trigger 40 different jobs or could be the same job? Give us more details. What is this job, is it a single job or a batch. Are there any dependencies between the files?

Kirtikumar · Post by **Kirtikumar** » Tue Dec 12, 2006 1:12 am

If for all 40 files, you need to run a single job then this would be useful.

first create an empty file containing following columns - Filename and Date. Key columns will be Filename and Date.

Design the job that needs to be executed for all 40 files as multiple instance. So it should be enabled for multi-instance as well as should have features for multi instance.

Create one more server job which will have following design:

Code: Select all

       Hash file
        |
folder stage--->transformer--->transformer---->hashfile

.
Folder stage will read the directory where u will get all files. It will have only one column that will contain the filename. Using the filename and today's date do a lookup on hash file. If the lookup is not successful, call the datastage job designed to process files. Else do not do anything. Once the job call is successful, update the hash file with filename and today'ss date.

So what is happening here is in hash file u r maintain list of files processed and date. So if the lookup is successful means the file is processed today, else process the file and add one entry in hash file to indicate that it is processed.

Schedule this jobs on timesly basis and its done.