Check for File arrival

[ashish10mca]

Hi,


Please tell me how can i do this.. i have to process a file whenever its arrived
to our server in a day.If two file arrives i have to funnel them.

Problem i am facing that i dont know which time the file arrive so whenever file arrive i have to run the Job.

Scheduling the job 24 hours in a day is an option but it will degrade the os performance and datastage performance so suggest some optimal way.


Thanks

[qt_ky]

whenever file arrive i have to run the Job.

In that case, you're not likely to need to funnel two files in the same job at the same time.

Schedule a sequence job and use the Wait For File stage. It's not going to pose a significant burden on the OS or DataStage performance. If you design it differently, perhaps using polling and creative scheduling, then your going to spend more development time, which gets more expensive.

If your server hardware and OS are already at the limit of what they can handle, then you and your team have bigger problems to solve before adding more jobs of any kind.

[qt_ky]

Scheduling the job 24 hours in a day is an option but it will degrade the os performance and datastage performance so suggest some optimal way.

Try testing that option to find out exactly how much it degrades performance. I think it would be difficult to measure.

Then load test it. How many such jobs does it takes to actually degrade performance?

[jerome_rajan]

Hi,

Why don't you leave that job to the scheduler. So if you are using AutoSys to schedule your jobs, you may create a file watcher job in AutoSys that can run for as much time as you want. The moment it finds the files you are looking for, it will go into success state. Then you can set your job to trigger on the success of the file watcher AutoSys job.

[chulett]

Assuming they have a decent Enterprise scheduler, that is. We have to assume, unless specifically noted otherwise, that the 'scheduler' in question here is DataStage itself and for a repeating schedule on a UNIX system that means via cron.

[jerome_rajan]

Not too well versed with the native scheduler but does it not have a 'File Watcher' equivalent? Pardon my lack of knowledge with the scheduling department.

[qt_ky]

That would be the Wait For File stage I mentioned on the first reply... It's built into DataStage, so there's no need for an external product to do the same thing.

[chulett]

There is no 'native' DataStage scheduler. What is has is an interlude / interface to the standard scheduling utilities of the underlying O/S. So, for a UNIX server, when you are looking at or manipulating the schedule in Director, you are seeing the cron entries it has automagically created for you. Well... cron for any repeating schedules or at for any one-time schedules. On a Windows server it leverages the standard Windows at scheduler.

Eric is talking about scheduling in a Sequence job and having the first object in that sequence be a Wait For File stage.

[ray.wurlod]

You're right that there's no DataStage scheduler in version 7, which is the version indicated by the OP.

However, Information Server does have a scheduling service in version 8. But that scheduling service does not include a "file watcher" equivalent.

Other than multiple Wait For File activities in a sequence, there's no real built-in capability to wait for multiple files.

[chulett]

However, Information Server does have a scheduling service in version 8.

Since the original release, or was it added along the way?

[ray.wurlod]

8.0 as far as I am aware.