wait for file (by autosys or with datastage)

[vsi]

Hi,

I have 3 files generated by datastage (through 3 sequences). The next sequence (4th) should start after all these 3 files are available.

These 3 files could come within a span of 1 month.


Q1) Is it advisable to

a) use wait for file activity (3 for each files) in the 4th sequence. The 4th sequence being kicked off along with any one of the 3 sequences?
b) or use autosys to wait for all these 3 files (generated by datastage)?

Q2)
a) Can any1 provide me the link (a tutorial ) for autosys/control-M, to wait for file/fileS. I made lot of searches, but doesnt get information for coding.

Thanks

[chulett]

I have 3 files generated by datastage (through 3 sequences). The next sequence (4th) should start after all these 3 files are available.

These 3 files could come within a span of 1 month.

Q1) Is it advisable to

a) use wait for file activity (3 for each files) in the 4th sequence. The 4th sequence being kicked off along with any one of the 3 sequences?
b) or use autosys to wait for all these 3 files (generated by datastage)?

You want to have the jobs running and waiting for these files for up to a month? Not the approach I would choose.

Assuming you don't need to run the second all three files are there, I'd just do a one time (or possibly more) sweep for the three files each day and trigger the fourth the day you see all three files.

Q2)
a) Can any1 provide me the link (a tutorial ) for autosys/control-M, to wait for file/fileS. I made lot of searches, but doesnt get information for coding.

You don't need an autosys/control-M tutorial. Technically, you don't really need to know anything about them. I'm sure you've got people there whose job it is to support whatever Enterprise Scheduler your company uses. Sometimes all you need to do is tell them what you need and they set it all up for you.

In this case you'd probably need to provide them with a shell script to check for all files and pass back an appropriate status. They can then chain your two activities together - run the script, and if it says all three files are there they can then run the 4th job, otherwise pass.

[DSguru2B]

You can also look for the files for a specific time period in a day. If all files are found then execute sequence 4 , else end the job. Schedule this to run daily.
Or you can also go with Control-M. With control-M, the control wont be in datastage, it will be with the scheduler. As Craig suggested, talk to the scheduling folks, usually they have scripts like these already in place, as this is a common functionality thats widely used.

[vsi]

Thanks for the response. I would like to have few clarifications to know whether i am heading the right way.

Immediately after the 4th sequence gets kicked off, I plan to move those 3 files, so that the scheduling tool wont find the 3 files when it looks for them the next day.

Is it good to have the scheduler look for the files daily even if they are found and moved? or is it better to have the scheduler not look for a few days (if known how many days) after the files were found and the fourth sequence got triggered?

The 3 files which are generated out of the 3 sequences is for synchonizing purpose (passing information as well, but mostly for synchronizing purpose).

Is this a good approach. Because the three sequences are mutually exclusive sequences which gets triggered due to different events. This is the best way i was able to think of to synchronize.

[chulett]

It's a perfectly fine approach. The 'cost' to look for those files will be minimal and I always prefer the simplest approach that satisfies the requirements. For what it's worth, I've done things very similiar to this in the past, well before I learned how to spell DataStage.

If it was me, I'd have it run every day.

[vsi]

Thanks a lot.

[lgharis]

The capability of Autosys is to create a "Wait" job. This job will wait for a file to appear. You can have the job wait for a specified amount of time. If the file does not appear, you can have the job fail. You can also have it wait for the file to reach a certain minimum size or until the filesize has not changed after a specified number of seconds.

You could setup a Autosys command job to execute if the wait job completes successfully. The time you want to start the jobs is up to you.