Wait-for-file should run continuously

[dsscholar]

My jobs should be triggered if I get a file in a particular directory. For that I should be writing a file-watcher job that is always running and looking periodically for files. Once the file is there it should trigger the main sequencer.
As per my company policy I cant use crontab or any other scheduler to schedule this job. They need a job which should be kicked off once and then keeps running forever.
Please advise.

[chulett]

As per my company policy I cant use crontab or any other scheduler to schedule this job.

My advise - push back on the policy. I'm adding this to my list of 'silly ass policies' we see here way too often that make absolutely no sense.

Do you have an Enterprise Scheduler like Control-M? It should be doing the 'continuous periodic' file watching and then kicking off the appropriate process when they show. You really don't want a DataStage job that runs 24x7 unless we're talking RTI/SOA. IMHO.

ps. You could do this with a Sequence job, I would think. The latest version of the WFF stage can wait 'forever' and you could build it into a Loop construct. Still would advise against it, however.

[ray.wurlod]

What happens if you need to restart the server?

[dsscholar]

In that case, I think we would be re-running the job but that would not be an everyday activity. I just want to explore if there's any way at all to run a never ending loop which keeps looking for a file.

Initially I was think thinking of scheduling this job to run at regular intervals to look for the file, but they dont want to schedule it.

[jhmckeever]

You could always write a script which does an infinite loop like ...

Code: Select all

if FILE found then
   dsjob MY_SEQUENCE
fi

sleep for NN seconds

You could start this script (nohup MyScript &) when your server comes up and leave it looking for your file.

J.

[chulett]

As noted, don't schedule 'the job'. Schedule a small script that looks for the files and trigger the job(s) based on the result. Use an Enterprise Scheduler if available.

[ray.wurlod]

Of course you can have forever-running jobs. We just don't think they are a great idea.

The Wait For File activty in a job sequence can wait forever, but how would you shut that down gracefully?

You could build a loop in a job sequence that checks for the file (Wait For File with a short interval), checks for notification to shut down (Wait for some other file?), and sleeps for a while (to conserve CPU resources) if the file is not found. This won't be able to kick off your job immediately the file arrives, only when it next wakes.

Perhaps some compromise, where the Wait For File activity waits a long time (12 hours?). The loop would only be re-entered once every 12 hours, and any stop request might take as long as 12 hours to be fulfilled.

Otherwise you're up for writing some signal handling or semaphore manipulation code.

[dsscholar]

Thanks for all the responses..I think I will be able to tackle this now !

[ray.wurlod]

Still push back on company policy, or at least ask their reasons (and post them here so we can arm you with the appropriate refutation).

[dsscholar]

They agreed to schedule the job to run at regular intervals to check the presence of the file. :D

[ray.wurlod]

Congratulations! It is often worth challenging from a position of relevant knowledge.

[gateleys]

I was just wondering about the SIZE of the log for such a job... the one that runs infinitely .. or, for a very long time.

Or is there a way to purge the log even when the job is in a running status (like when it is in a 'nap' state in between the polling)? Or get DataStage to STOP writing anything at all to the log?

Just curious.

[DSguru2B]

Well if the job is just waiting for a file there will be a lock on its RT_LOG and nothing will be logged.
If the job is running indefinately and after completing a process comes back to wait for the file then I doubt anyone would be able to purge the logs externally. It will need to be coded inside the job.

[gateleys]

It will need to be coded inside the job.

But the process will already have exclusive lock on the RT_LOGnnn file. How will you code from inside the job to purge the log or something equivalent?

[ravibabu]

Hi


Please use the StartLoopactivity and Endloopactivity for this problem