I would like to iterate through a list of files and call a datastage job to process those files simultaneously using the filename as the invocation id. I have created a sequence that does this. However, the loop is currently waiting for the datastage job to complete before moving to the next file in the list. What I'd like to see is N jobs running simultaneously with their invocation id. Is there a way to do this? Is there something in the trigger that I need to set properly?
I had the job activity inline; that did not work. So I tried adding a sequencer that would fork off to the job activity. That does not work either.
---read_list_file----|
|
v
end loop------>loop----set_parms
| (for each file) |
| |
^------sequencer-----<
|
|
V
job_activity
(invocation_id: filename from list)
| |
| |
v v
email_success email_fail
The Job Activity stage runs a job and then waits for it to complete, that functionality cannot be changed. You'd have to "roll your own", say something like a routine does all of the same steps except the DSWaitForJob() call or a script that leverages dsjob without either of the wait options.
The only issue would be monitoring the status of all the jobs...
-craig
"You can never have too many knives" -- Logan Nine Fingers
Does the Loop have to wait for the Job Activity to finish even if I "fork" the flow to continue? So, File1 MUST complete before File2 is even processed.
The Job Activity does a 'Wait for File' So, I don't mind the Master Sequence waiting for all the children to finish before it finishes; That would be a nice to have to tie up the work package. I just want to have all the jobs waiting for their individual files at the same time since the files will not generally arrive in order. File5 may arrive before File3 which arrives after File1. And it could be several hours/days before any one file arrives. The only information I have is the list of files to process as soon as they arrive.
If you have any suggestions on a better way to implement, I'd greatly appreciate it.
Routine is easy to write. You need to do a couple things to get parameters to work unless all jobs have the same parameter list.
When you attach to a job you need to get the valid parameters for that job. Next you can only set a parameter that exists in the job you are about to run. I can probably post some of this code if you need it. I have posted similar code done in a shell script.
ray.wurlod wrote:Use sub-sequences each with its own WaitForFile activity and Job activity.
Ray- how would this be different from what I have currently designed? The job i'm calling from the master is a sequence with a Wait For File inside. The problem I'm having is that the master sequence will not invoke the subsequence and then continue to loop without waiting for the subsequence to finish so that I have more than one subsequence running at the same time.
kduke wrote:Routine is easy to write. You need to do a couple things to get parameters to work unless all jobs have the same parameter list.
When you attach to a job you need to get the valid parameters for that job. Next you can only set a parameter that exists in the job you are about to run. I can probably post some of this code if you need it. I have posted similar code done in a shell script.
If you can post an example or point me to one of your prior posts that might be helpful. I would have thought this would be a fairly common use for the looping construct.
The master sequence always waits for a sub-sequence. You can take the sub-sequence code and turn it into a routine that returns immediately (remove the call to DSWaitForJob), but you would then need somehow to code for waiting for all the sub-sequences to finish.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Ray is correct on how to build a routine. You could also start with SDK utility run job routine. That might help. Lots of posts on here about using it.
Thanks Uncle Kim for the script.. I have stripped it down and added a flag.
Still seems like the Job Activity should have a switch that disables the wait for finish.
Maybe the logic should handle $RETURN_VALUE -eq 0 separately. This means that the job start request was successful and that the job is still running. You have no idea when it will end.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ray.wurlod wrote:Maybe the logic should handle $RETURN_VALUE -eq 0 separately. This means that the job start request was successful and that the job is still running. You have no idea when it will end.
