DataStage complex job schedule

calvinlo · Post by **calvinlo** » Mon Jul 07, 2003 11:58 pm

Dear All,

I would like to do some complex scheduling job control and i've written some script using "loop repeat sleep..." but i find that i cannot use director to stop these infinite loop jobs.
and then i tried to read from a config file to see if a paramater is true or false to trigger the job run. But if one hopes to stop a job...he needs to edit the config file instead of easily pressing stop button in director....and it seems have many dependencies...like the config file...the control job...the script.....
am i doing the right thing?? or any other suggestion?

thanks

Cal

kcbland · Post by **kcbland** » Tue Jul 08, 2003 6:28 am

What I have done is put external file watching into my infinite loops. I've also put time limits on the infinite loops if one can be determined. As for the file watching, periodically check a polling directory for the presence of certain files. I used files name STOP.flg, KILL.flg, PAUSE.flg, RESUME.flg, and DEBUG.flg. Stop means exit from the loop. Kill means kill your current executing jobs and exit from the loop. Pause means don't do anything, just keep cycling, whereas Resume would start running jobs again. Debug would output messages as to the current run information.

Good luck!
-Ken

Kenneth Bland

chulett · Post by **chulett** » Tue Jul 08, 2003 6:56 am

I've done the same as Ken and it works well. Time or loop limits should always (IMHO) be part of the process so you don't have 'infinite loop' jobs. Also, a quick check for a stop flag of some sort during each iteration in case you need to interrupt things is easy to code. On a Unix server, a simple 'touch' of the file would suffice as you would only be checking for its existence, or a script could be written to create the appropriate file for you.

-craig

calvinlo · Post by **calvinlo** » Tue Jul 08, 2003 9:26 pm

Thank you very much!
and it means that i can make some control jobs to create those flag files automatically so that i can easily stop/pause the job by the director, rite?
how about the time limit, how u do that? create a config file and the job read parameter from the file? one problem is that if a job loop to read parameter from the config file, another job is not able to modify it since the file is locked (loop infinitely to read by anothe job), by vi i can modify it but by datastage job i cannot.
the major concern is that my client dun want to monitor the job with large effort (e.g. edit config file to change the time limit). they just hope to use Director to control all the jobs in GUI.
Hope you all could give me a hand....thank you!

Cal

chulett · Post by **chulett** » Tue Jul 08, 2003 11:03 pm

Yes, you certainly could build jobs to create the flag files if you want to do everything from the Director.

As to the time limits, there are multiple choices there as well, all typically handled as Job Parameters. We do fine with passing in as parameters the number of times to loop and the sleep time between loops. Between this and the start time we schedule the job for, we know the 'window' the job will run in.

Others have posted (but I couldn't seem to find) code to take the run time of the jobs inside your loop into account. That way, if you want something to happen every hour let's say, and it takes three minutes to run the jobs in the loop, it will know to sleep 57 minutes this iteration of the loop.

Lastly, I would think you could pass a time in as a parameter as your absolute limit for job execution and stop when you have iterated past this time.

As you've noted, job parameters cannot be changed while the job is running. I'm not sure I'm following your concerns, but if you are really worried about being able to dynamically change the nature of the 'looping' you are doing, then I'd look into putting those parameters into a flat file (for example) and reading the file each time thru to know how to handle the loop that particular time. As long as you open and close the file each time, you could sneak in and change it while the job is running. If it doesn't need to be dynamic, then a simple 'reschedule' of the job via Director will allow an easy change of the parameters to use the next time the job runs.

-craig

calvinlo · Post by **calvinlo** » Wed Jul 09, 2003 12:17 am

Thanks so much!!
You've solved my problem. The time limit should be set as job paremeter. If one hopes to change the schedule must stop the job and start again to enter the new paremeter. Maybe I thought too much and actually it is straight forward. Anyway, thank you.

Cal

calvinlo · Post by **calvinlo** » Wed Jul 09, 2003 12:28 am

Dear all,

I encounter another problem. If one sets the time limit to say every 300 minutes to run the job and in my script there is a statement like SLEEP 300mins * 60 - job run time, and then someone hopes to stop and reschedule to run the job every 10 minutes immediately, how do i cope with this situation??? I need to wait the sleep function finish until i can restart the job.
Thank you for your help!

Cal

ray.wurlod · Post by **ray.wurlod** » Wed Jul 09, 2003 6:10 am

SLEEP hh:mm
This form of the SLEEP statement will sleep until the specified time (on the server!).

What you seem to require, however, is a loop containing short sleeps, checking periodically whether to execute some control function (such as finishing) and checking at a longer interval whether to start any queued jobs.

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

calvinlo · Post by **calvinlo** » Wed Jul 09, 2003 8:13 pm

Thank you for your help!
by the way, if there are hundreds of jobs running on server, will it affect the performance?? Since most jobs are LOOPING ...SLEEPING...CHECKING FLAG...all the time.

Cal

kcbland · Post by **kcbland** » Wed Jul 09, 2003 8:25 pm

You must have an interesting set of requirements. It sounds like you are using DataStage as some sort of EAI implementation or data bridge.

As far as jobs sleeping, there's almost no cpu time involved. Checking for a file existence adds a little overhead while looping, but if you're doing sleeps in between then you should be okay. One neat trick to reading a sequential text file really easily is to use a "cat" UNIX statement or a "type" NT statement to send a file into a variable.

For Unix, try these lines:
UNIXcmd = "cat JobControlLoopingFile.txt"
CALL DSExecute("UNIX", UNIXcmd, ScreenOutput, ReturnCode)

This statement will echo the contents of the file, and since DSExecute returns in the third argument the screen output, your file is contained in the variable ScreenOutput! Now, you can do simple parsing to retrieve runtime values. If someone vi'd the file, or you ran a Batch job that updated this file based on user parameters, it's immediately picked up. You don't have to mess with sequential reads, file locks, etc.

Good luck!

Kenneth Bland

ray.wurlod · Post by **ray.wurlod** » Thu Jul 10, 2003 5:55 pm

I tend to use the BASIC statement ReadV to test for the existence of "flag" files, since they can be zero length. If the ReadV statement is successful, then its Then clause is taken, otherwise its Else clause is taken. The directory in which the file is to exist or not is opened with a regular OPEN statement.

Common /Files/FileDir.fvar

* Obtain name of flag file from job parameter
FileName = DSGetParamInfo(DSJ.ME, "FlagFile", DSJ.PARAMVALUE)

* Decompose into directory and entry names
FileDir = ""
EntryName = ""
Call !GET.PATHNAME(FileName, FileDir, EntryName, Status)

* Open file if not already open
Ans = @NULL
If FileInfo(FileDir.fvar, 0) = 0
Then
Open FileDir To FileDir.fvar
Else
GoTo MainExit
End
End ; * end of If statement

* Determine whether file name exists in directory.
* ReadV with 0 as field number argument is fast.
ReadV Test From FileDir.fvar, FileMame, 0
Then FileExists = @TRUE
Else FileExists = @FALSE
Ans = FileExists

MainExit:

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

calvinlo · Post by **calvinlo** » Thu Jul 10, 2003 7:37 pm

Thank you!
I use openSeq to test if the file exists, is it a solution too?

kcbland · Post by **kcbland** » Thu Jul 10, 2003 8:53 pm

You can use openseq to test for files, but one some OS's this can give sporadic false positives (Sun 2.8 for example). Your best choice is to use a unix call to a shell script doing "test -r filename"

Good luck!

Kenneth Bland