Wait-for-file should run continuously
Moderators: chulett, rschirm, roy
Wait-for-file should run continuously
My jobs should be triggered if I get a file in a particular directory. For that I should be writing a file-watcher job that is always running and looking periodically for files. Once the file is there it should trigger the main sequencer.
As per my company policy I cant use crontab or any other scheduler to schedule this job. They need a job which should be kicked off once and then keeps running forever.
Please advise.
As per my company policy I cant use crontab or any other scheduler to schedule this job. They need a job which should be kicked off once and then keeps running forever.
Please advise.
Re: Wait-for-file should run continuously
My advise - push back on the policy. I'm adding this to my list of 'silly ass policies' we see here way too often that make absolutely no sense.dsscholar wrote:As per my company policy I cant use crontab or any other scheduler to schedule this job.
Do you have an Enterprise Scheduler like Control-M? It should be doing the 'continuous periodic' file watching and then kicking off the appropriate process when they show. You really don't want a DataStage job that runs 24x7 unless we're talking RTI/SOA. IMHO.
ps. You could do this with a Sequence job, I would think. The latest version of the WFF stage can wait 'forever' and you could build it into a Loop construct. Still would advise against it, however.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
In that case, I think we would be re-running the job but that would not be an everyday activity. I just want to explore if there's any way at all to run a never ending loop which keeps looking for a file.
Initially I was think thinking of scheduling this job to run at regular intervals to look for the file, but they dont want to schedule it.
Initially I was think thinking of scheduling this job to run at regular intervals to look for the file, but they dont want to schedule it.
-
- Premium Member
- Posts: 301
- Joined: Thu Jul 14, 2005 10:27 am
- Location: Melbourne, Australia
- Contact:
You could always write a script which does an infinite loop like ...
You could start this script (nohup MyScript &) when your server comes up and leave it looking for your file.
J.
Code: Select all
if FILE found then
dsjob MY_SEQUENCE
fi
sleep for NN seconds
J.
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Of course you can have forever-running jobs. We just don't think they are a great idea.
The Wait For File activty in a job sequence can wait forever, but how would you shut that down gracefully?
You could build a loop in a job sequence that checks for the file (Wait For File with a short interval), checks for notification to shut down (Wait for some other file?), and sleeps for a while (to conserve CPU resources) if the file is not found. This won't be able to kick off your job immediately the file arrives, only when it next wakes.
Perhaps some compromise, where the Wait For File activity waits a long time (12 hours?). The loop would only be re-entered once every 12 hours, and any stop request might take as long as 12 hours to be fulfilled.
Otherwise you're up for writing some signal handling or semaphore manipulation code.
The Wait For File activty in a job sequence can wait forever, but how would you shut that down gracefully?
You could build a loop in a job sequence that checks for the file (Wait For File with a short interval), checks for notification to shut down (Wait for some other file?), and sleeps for a while (to conserve CPU resources) if the file is not found. This won't be able to kick off your job immediately the file arrives, only when it next wakes.
Perhaps some compromise, where the Wait For File activity waits a long time (12 hours?). The loop would only be re-entered once every 12 hours, and any stop request might take as long as 12 hours to be fulfilled.
Otherwise you're up for writing some signal handling or semaphore manipulation code.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I was just wondering about the SIZE of the log for such a job... the one that runs infinitely .. or, for a very long time.
Or is there a way to purge the log even when the job is in a running status (like when it is in a 'nap' state in between the polling)? Or get DataStage to STOP writing anything at all to the log?
Just curious.
Or is there a way to purge the log even when the job is in a running status (like when it is in a 'nap' state in between the polling)? Or get DataStage to STOP writing anything at all to the log?
Just curious.
gateleys
Well if the job is just waiting for a file there will be a lock on its RT_LOG and nothing will be logged.
If the job is running indefinately and after completing a process comes back to wait for the file then I doubt anyone would be able to purge the logs externally. It will need to be coded inside the job.
If the job is running indefinately and after completing a process comes back to wait for the file then I doubt anyone would be able to purge the logs externally. It will need to be coded inside the job.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.