Event based scheduling

nilotpalr · Post by **nilotpalr** » Wed Nov 20, 2002 1:26 am

Can any one suggest how to do start a job on an Event. I need to monitor a folder. When I get a file to process in that folder my job for processing that file should be invoked. I do not want time based scheduling where my job polls that folder and starts when it finds a file there.
Thanks in advance,
Nilotpal

ray.wurlod · Post by **ray.wurlod** » Wed Nov 20, 2002 1:47 am

What version of DataStage?
In release 5.0 and later, you can construct a Sequence, and use a WaitForFileActivity. You can have separate output links from this activity, for example one that is triggered (and runs your job) if the file appears (or disappears), the other the is triggered if the wait times out (which is a property of the activity).

chulett · Post by **chulett** » Wed Nov 20, 2002 7:42 am

Well, if you don't want "time based scheduling" then you need to look at having the job that delivers the file launch the DataStage job. Check out the Command Line Interface section of the Server Jobs manual and see if creating a post-delivery script might work for you. We have a couple of jobs that do that, but most do the "polling" thing.

-craig

vbalacts · Post by **vbalacts** » Wed Nov 20, 2002 9:07 am

We want to design job in such a way that the job has to poll for a table with flag as 'Y'. We are using datastage V5.2. We dont want to spool it in file and then use the WaitforfileActivity. Is it possible to poll for a table?

Thanks,
Bala

Well, if you don't want "time based scheduling" then you need to look at having the job that delivers the file launch the DataStage job. Check out the Command Line Interface section of the Server Jobs manual and see if creating a post-delivery script might work for you. We have a couple of jobs that do that, but most do the "polling" thing.

-craig

chulett · Post by **chulett** » Wed Nov 20, 2002 9:37 am

We want to design job in such a way that the job has to poll for a table with flag as 'Y'. We are using datastage V5.2. We dont want to spool it in file and then use the WaitforfileActivity. Is it possible to poll for a table?

Sure... and you can make it as simple or as complicated as you like. I have a job that polls for unprocessed rows in a control table, something like you indicate above. If found, I process some information to hash that the following jobs use. This initial job is launched from a 'Batch' in a loop that checks for processed rows using DSGetLinkInfo. If none are processed, it sleeps and loops back around for another go until it either finds what it expects or exceeds its polling window. Sleep time and number of times to poll are Job Parameters, and I get paged if nothing is found or the process is running 'late'. Once the flag(s) are found, the batch passes control on to the next series of jobs. Somewhere along the line (where appropriate) there is a job that resets the flags found.

Hope this helps,

-craig

vbalacts · Post by **vbalacts** » Wed Nov 20, 2002 10:48 pm

Thanks for the reply. Is it possible to achieve the same thing thro mappings? Because, our client want everything in terms of mapping and they are not encouraging the coding part.

Thanks,
Bala

We want to design job in such a way that the job has to poll for a table with flag as 'Y'. We are using datastage V5.2. We dont want to spool it in file and then use the WaitforfileActivity. Is it possible to poll for a table?
Sure... and you can make it as simple or as complicated as you like. I have a job that polls for unprocessed rows in a control table, something like you indicate above. If found, I process some information to hash that the following jobs use. This initial job is launched from a 'Batch' in a loop that checks for processed rows using DSGetLinkInfo. If none are processed, it sleeps and loops back around for another go until it either finds what it expects or exceeds its polling window. Sleep time and number of times to poll are Job Parameters, and I get paged if nothing is found or the process is running 'late'. Once the flag(s) are found, the batch passes control on to the next series of jobs. Somewhere along the line (where appropriate) there is a job that resets the flags found.

Hope this helps,

-craig

ray.wurlod · Post by **ray.wurlod** » Thu Nov 21, 2002 2:20 am

The original poster (Nilotpalr) wanted to start a DataStage job based on an event.
We showed you the WaitForFile activity.

But then the rules changed. Bala came back with 'we want to poll for a table with flag = "Y" and not spool to a file and use a WaitForFile activity'. Should we have begun a new thread here?
We suggested two ways to poll for a flag.

Now Bala asks whether it is possible to achive this through mappings. Well, basically no. Your hierarchy of jobs would be very convoluted, as you attempted to communicate the result from the polling job back to its controlling job so that it can decide whether to run the real job.
That said, it's not impossible. But you WILL need a sequence, which creates DataStage BASIC code anyway, so the code will be there. To avoid looking at the code entirely, the sequence would need to run the polling job, which creates a file if the "Y" flag is found. A WaitForFile activity can then decide whether to run the other job. But that's the thing you said you didn't want to do!

Other communication mechanisms exist that you can explore, including:
- named pipes between jobs (with timeout) using sequential file stage
- DataStage job's user status area
- UniVerse numbered semaphores (see LOCK in the BASIC manual)
- artificial entries in the lock table
All of these, of course, involve some coding.

I take it you seek an approach that doesn't have the job aborting if the flag is not "Y".