How do I create a basic Sequencer?

[Tampa_Indy]

Trying to create a sequencer that kicks off 40 jobs. Technically speaking, there is no need for any sort of 'sequence.' All it needs to do is run all 40 jobs at once. What is the best way to accomplish this?

FYI, I have brought in all 40 DataStage jobs into the Sequence file, but do not know what to from this point forward.

Many thanks in advance!

[chulett]

All it needs to do is run all 40 jobs at once.

Are you sure this is wise, as in you've done this before and you know your system can handle it?

If literally all you need to do is start 40 jobs and not have any monitoring in place, I'd look into scripting a loop around dsjob. Iterate through a list of job names, fire them off one at a time without a "wait" option, sleep for a short period of time. Lather, rinse, repeat until you run out of jobs. Or until the system goes kaboom.

[Tampa_Indy]

...scripting a loop around dsjob...

How do you do this? Novice here, step by step instructions would be ideal.

[chulett]

Is your ETL server truly on Windows? If so that would mean a DOS batch file, something that's not my strong suit. Or (question to the general audience) can one still craft shell scripts and take advantage of the MKS Toolkit to run it?

The problem with dumping them all in a Sequence job and using the Job Activity stage to run them is they will all run one at a time in a serial fashion as that stage waits for the job to complete before it moves on.

[Tampa_Indy]

Is your ETL server truly on Windows

No it is not. It is actually UNIX/AIX, my mistake. How do I take advantage of the MKS Toolkit. All posts I am coming across are suggesting using a sequencer (or scheduler, which we would prefer not to use) for this.

[austin_316]

I'd look into scripting a loop around dsjob. Iterate through a list of job names, fire them off one at a time without a "wait" option, sleep for a short period of time. Lather, rinse, repeat until you run out of jobs. Or until the system goes kaboom.

I would go with the option Craig mentions. We have similar scenarios in our project. We have around 200 jobs that needed to be split into 4 sets(Sequences) and there is no dependency between each job. In the initial implementation we went with 4 Sequences included the required jobs inside sequences and then exected them. Even though there is no dependency between jobs to avoid loading the server we kept them in a sequence but as Craig mentioned this would mean that one job waits for other to finish(either success or failure).

To avoid this dependency, we maintined the list of jobs in a file and started looping through the file to execute 10 jobs at a time using dsjob command. And if there are any jobs that aborted we look for the status of the job and maintain another file aborted_list and then execute them again.

May be below shell script code snippet can help you achieve this

while read extr_rec; do

export JOBNM=$(echo ${extr_rec})

JMOPTIONS="-domain ${SERVER_PORTID} -user ${DS_USER} -password ${DS_PWD} -server ${DS_SRVR}"

RunJobsBackEnd &

done < Jobs_Details.dat

RunJobsBackEnd is a function inside shell script which will be execute at the backend

JPROCCMD="${DSENGINE}/bin/dsjob ${JMOPTIONS} -run -warn 0 ${PROJNAME} ${JOBNM}"

$(echo ${JPROCCMD})

job_sts=$(${DSENGINE}/bin/dsjob -jobinfo ${PROJNAME} ${JOBNM} | head -1 | cut -d "(" -f 2 | cut -d ")" -f 1)

## If the job_sts value is '0' i.e., if the job status has not yet been updated in the DS system
while [[ ${job_sts} -eq 0 ]]; do
sleep 2
job_sts=$(${DSENGINE}/bin/dsjob -jobinfo ${PROJNAME} ${JOBNM} | head -1 | cut -d "(" -f 2 | cut -d ")" -f 1)
done

## Check if the job is aborted or not finished
if [[ ${job_sts} -ge 3 || ${job_sts} -eq -2 ]]; then
echo "aborted"
echo "${JOBNM}" >> Incomplete_Details.dat
else
echo "finished"
fi

This is not the exact script but you can tweak this as per your requirement and execute.

But just be careful about the load or else definitely

Or until the system goes kaboom.

[Tampa_Indy]

I'd look into scripting a loop around dsjob.

I would go with the option Craig mentions.

RunJobsBackEnd is a function inside shell script which will be execute at the backend

A what? Novice here, novice. I am not familiar with running shell scripts or even creating them. Provided I have no idea on how to build these, should I 'learn' or simply figure out a way to make the sequencer work?

[chulett]

Never mind about the Toolkit reference, that was only valid on a Windows server. Since you are on a UNIX one, you have the Power Of Shell Scripts at your disposal! Is there anyone you work with that can help you with crafting one? It's not something to pick up from a forum post or two, unfortunately.

As noted, examples have been posted here in the past, one such example is here. Hopefully it helps.

As noted earlier, the Sequence job will run everything in a serial fashion - unless you have a wrapper script, in which case you could use the Execute Command stage to run the script 40 times. Again, without any waiting to see what happens, just crank one up and move on. That would take the 'iteration' out of the script you could write and simplify the task... somewhat. Basically the Sequence could loop through the list of job names for you.

[ray.wurlod]

You don't want to run 40 jobs at the same time - it will overload your machine (unless, of course, you are on a version in which Workload Manager is enabled - that you're in version 8.x suggests that you don't have WLM).

Let's imagine you want to run no more than eight jobs at a time (and even that might be a lot). Create a sequence job with five streams of job activities, each with eight job activities in the stream.

Otherwise create 40 isolated job activities in the sequence (with no links between them). They won't all start at exactly the same time, but their start requests will be fairly close together. This will super-saturate your server, and may throw "unable to start in time" errors.

[chulett]

Ray, for your last suggestion - won't all the jobs still run in a serial fashion? The Job Activity stage forces a wait, so nothing in the sequence will happen until the first job finishes at which point the second will start. Yes?

I'm remembering several conversations on this topic here over the years but that could just be my mind playing tricks on me. Again.

[Tampa_Indy]

So this is the latest:

We are unable to access the DataStage server. Therefore, shell scripting is not an option. I believe I have successfully created the sequencer.

[chulett]

We are unable to access the DataStage server. Therefore, shell scripting is not an option.

Can shell scripts be created but then 'installed' and/or run on the server by someone else, an admin perhaps? If they are completely off the table, you've lost a very important tool from your ETL toolbox. IMHO.

Let us know how the Sequence job works out for you.

[Tampa_Indy]

Can shell scripts be created but then 'installed' and/or run on the server by someone else, an admin perhaps?

Yes it can, but we do not have access to that person. Unfortunately, I think the sequencer is our only option. We are testing jobs right now to see if it will work. Thank you all for the feedback!

[ray.wurlod]

You can use a sequence. Try using job activities without links, to see whether they wait - the sequence is supposed to start as many jobs as are eligible and wait for any of them to finish. I just checked this on a version 8.5 system and, indeed, that's what happens - three jobs running simultaneously and the sequence job log contains "Waiting for Job1~Job2~Job3".

Otherwise use a sequence of Execute Command activities that start dsjob as a nohup background process. These commands will return immediately, meaning that your sequence will not notice the actual finish of the jobs.

[vamsi_4a6]

chulett, could you please explain why we have to use without a "wait" option, sleep for a short period of time before triggering next job?