job run as per the sequence available

jeet_81 · Post by **jeet_81** » Thu Nov 03, 2011 9:25 am

I have a text file which contains the names of few job sequences new line delimiter.

Now I am developing a sequence job which will only run those sequences whose names are retrieved from the text file.

(i) I call an script which gives me the names of the sequences.

(ii) I store all the names of in UserVariable.

(iii) Then I start a loop activity which takes the names one by one.

(iv) I applied a job activity stage,but cannot get the value i.e the sequence name to run from the file in the Job Name tab of the job activity.

Could you please suggest me whether the design is correct or it has to be done in some other way?

Hope I am able to explain the requirement.

ray.wurlod · Post by **ray.wurlod** » Thu Nov 03, 2011 12:41 pm

HOW does the StartLoop activity "take the names one by one"? I suspect the logic at that point may be faulty. Also, what is the output of the script, particularly delimiters?

kduke · Post by **kduke** » Thu Nov 03, 2011 4:05 pm

I tihnk he is correct, I think it will not let you use a variable for the job name. You need to use a routine activity and call UtilityRunJob in the SDK. You may have an issue with feeding parameters into jobs ran.

ray.wurlod · Post by **ray.wurlod** » Thu Nov 03, 2011 4:14 pm

It should be quite possible to use DSUtilityRunJob from a routine activity, and to construct its parameters string either in that routine or in a User Variables activity.

jeet_81 · Post by **jeet_81** » Fri Nov 04, 2011 3:25 am

Hi all,
The file is comma delimited which has got all the sequences to run. Now I ran a shell script which gives me all the names of the sequence jobs which is comma delimited and is stored in a variable in UserVariableActivity.Now in StartLoopActivity I get the sequence name which I use Routine UtilityRunJob with Arg1 = Name of the sequence
Arg2 = ''(because when it will call the sequence, it will get all the parameters to run in that particular sequence)
Arg3 = 0 and Arg4 =1.
Now it runs fine for the 1st iteration but it does not complete all its iterations. I have got sequence names in the file to run,but it only runs the 1st one and finishes with the log ' StartLoop_Activity_10' was started, but did not complete all its iterations'

Please help me.

chulett · Post by **chulett** » Fri Nov 04, 2011 6:09 am

It sounds like the first Sequence you ran inside the loop did not run gracefully. What is in its log? What is your trigger expression from there to the End Loop stage?

jeet_81 · Post by **jeet_81** » Fri Nov 04, 2011 7:43 am

Well the first sequence which ran inside the loop ran absolutely fine.The logs are fine. Trigger from the routine to EndLoopStage is 'ok Conditional'

I am attaching the job control code:

* Tokens were replaced below as follows:
* IdV0S9%%UserVar%%0 <= UserVariables_Activity_9.Seq_Name
* IdV0S3%%Result1%%1 <= Script_to_get_seq.$ReturnValue
* IdV0S3%%Name%%2 <= Script_to_get_seq.$CommandName
* IdV0S3%%Result2%%3 <= Script_to_get_seq.$CommandOutput
* IdV0S10%%Counter%%4 <= StartLoop_Activity_10.$Counter
* IdV0S22%%Result1%%5 <= Routine_Activity_22.$ReturnValue
* IdV0S22%%Name%%6 <= Routine_Activity_22.$RoutineName
* IdV0S22%%Param%%7 <= Routine_Activity_22.Arg1
* IdV0S22%%Param%%8 <= Routine_Activity_22.Arg2
* IdV0S22%%Param%%9 <= Routine_Activity_22.Arg3
* IdV0S22%%Param%%10 <= Routine_Activity_22.Arg4
*
*** [Generated at 2011-11-04 10:48:06 - 8.1.0.0]
$INCLUDE DSINCLUDE DSJ_XFUNCS.H
EQU Time$$ Lit "Oconv(Time(), 'MTS:'):': '"
****************************************
* Graphical Sequencer generated code for Job dummy
****************************************

b$V0S3$autoreset = @False
loop$V0S10$entered = @False
loop$V0S10$done = @False
loop$V0S16$triggered = @False
seq$V0S24$count = 0
seq$V0S24$status = 0
gosub$stack = 0
b$CPRestarted = @False
cp$loops = ""
loop$prevcontexts = ""
b$InErrorRef = @False
summary$restarting = @False
*** Create new checkpoint for sequence if not restarting
If DSCheckPointExists(DSJ.ME, cp$dtm) Then GoTo L$RESTART
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0015\Starting new run of checkpointed Sequence job", ""), "@Coordinator")
If (Not(DSCheckPointValidate(DSJ.ME, cp$msg))) Then
err$source = "@Coordinator"; err$number = -1
msg$ = DSMakeMsg("DSTAGE_JSG_M_0016\Error creating checkpoint record<L>%1", cp$msg)
msg$id = "@Coordinator"; GoTo L$ERROR
End
GoTo L$MAINSTART
L$RESTART:
*** Validate restarted job matches prior state
summary$restarting = @True
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0017\Sequence job is being restarted after failure<L>Checkpoint record last modified at %1", cp$dtm), "@Coordinator")
b$CPRestarted = @True
If Not(DSCheckPointValidate(DSJ.ME, cp$msg)) Then
Call DSLogWarn(cp$msg, "@Coordinator")
End
L$MAINSTART:
*** Sequence start point
summary$ = DSMakeMsg("DSTAGE_JSG_M_0048\Summary of sequence run", "")
If summary$restarting Then
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0049\Sequence restarted after failure", ""))
End Else
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0050\Sequence started (checkpointing on)", ""))
End
loop$context = 1
cp$loops = "X"
GoTo L$V0S3$START
L$RETURN$1:
GoTo L$FINISH
**************************************************
L$V0S3$START:
*** Activity "Script_to_get_seq": Execute command
x$V0S3$cp = "x$V0S3+":Convert(@FM, "_", cp$loops)
If DSCheckPointGetValue(DSJ.ME, x$V0S3$cp, cp$value) = DSJE.NOERROR Then
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0023\Skipped execution of command '%1' on restart", "sh ":pcscriptDirectory), "@Script_to_get_seq")
x$V0S3 = Raise(cp$value<1>)
x$V0S3$output = Raise(cp$value<2>)
IdV0S3%%Result1%%1 = x$V0S3
IdV0S3%%Name%%2 = "sh ":pcscriptDirectory
IdV0S3%%Result2%%3 = x$V0S3$output
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0062\%1 (COMMAND %2) skipped", "Script_to_get_seq":@FM:"sh ":pcscriptDirectory))
Goto L$V0S3$SKIPPED
End
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0059\%1 (COMMAND %2) started", "Script_to_get_seq":@FM:"sh ":pcscriptDirectory))
p$V0S3$1 = (pcSequenceFile)
x$cmd = "sh ":pcscriptDirectory:" ":p$V0S3$1
x$type = (If System(91) = 1 Then "NT" Else "UNIX")
Call DSExecute(x$type, x$cmd, x$V0S3$output, x$V0S3)
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0007\Executed: %1<L>Reply=%2<L>Output from command ====><L>%3", x$cmd:@FM:x$V0S3:@FM:Lower(x$V0S3$output)), "@Script_to_get_seq")
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0064\%1 finished, reply=%2", "Script_to_get_seq":@FM:x$V0S3))
IdV0S3%%Result1%%1 = x$V0S3
IdV0S3%%Name%%2 = "sh ":pcscriptDirectory
IdV0S3%%Result2%%3 = x$V0S3$output
If b$InErrorRef Or (x$V0S3 <> 0) Then Goto L$V0S3$SKIPPED
cp$value = Lower(x$V0S3)
cp$value<2> = Lower(x$V0S3$output)
dummy$ = DSCheckPointSetValue(DSJ.ME, x$V0S3$cp, cp$value)
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0027\Checkpointed execution of command '%1'", "sh ":pcscriptDirectory), "@Script_to_get_seq")
**************************************************
L$V0S3$SKIPPED:
GoTo L$V0S9$START
L$V0S3$RETURN$1:
GoTo L$RETURN$1
**************************************************
L$V0S9$START:
*** Set user variables: "UserVariables_Activity_9"
IdV0S9%%UserVar%%0 = (IdV0S3%%Result2%%3)
GoTo L$V0S10$START
L$V0S9$RETURN$1:
GoTo L$V0S3$RETURN$1
**************************************************
L$V0S10$START:
*** Start of loop "StartLoop_Activity_10": Initialize loop control variables
loop$V0S10 = 0
loop$V0S10$sep = Char(0)
loop$V0S10$list = EReplace(IdV0S9%%UserVar%%0, ",", loop$V0S10$sep)
loop$V0S10$final = DCount(loop$V0S10$list, loop$V0S10$sep)
If (loop$V0S10$sep = '') Then
err$source = "StartLoop_Activity_10"; err$number = -1
msg$ = DSMakeMsg("DSTAGE_JSG_M_0033\Loop specifies empty list separator - cannot execute", "")
msg$id = "@StartLoop_Activity_10"; GoTo L$ERROR
End
If (loop$V0S10$list = '') Then
err$source = "StartLoop_Activity_10"; err$number = -1
msg$ = DSMakeMsg("DSTAGE_JSG_M_0034\Loop specifies empty iteration list - cannot execute", "")
msg$id = "@StartLoop_Activity_10"; GoTo L$ERROR
End
loop$V0S10$entered = @True
loop$V0S10$done = @False
loop$V0S10$returned = @False
loop$V0S16$triggered = @False
loop$prevcontexts<1> = loop$context
loop$context = 2
**************************************************
L$V0S10$REPEAT:
* Test loop variables to see if repeat needed
Loop While gosub$stack > 1
Return To L$V0S10$UNWIND
L$V0S10$UNWIND:
gosub$stack -= 1
Repeat
loop$V0S10 += 1
If loop$V0S10 > loop$V0S10$final Then loop$V0S10$done = @True
If loop$V0S10$done Then GoTo L$V0S16$COMPLETE
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0036\Loop through list (%1 items): iteration %2", loop$V0S10$final:@FM:loop$V0S10), "@StartLoop_Activity_10")
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0070\%1 loop iteration %2 started", "StartLoop_Activity_10":@FM:loop$V0S10))
IdV0S10%%Counter%%4 = Field(loop$V0S10$list, loop$V0S10$sep, loop$V0S10)
seq$V0S24$count<loop$context> = 0
cp$loops<1> = loop$V0S10
GoTo L$V0S22$START
L$V0S10$RETURN$1:
L$V0S10$FINISHED:
If loop$V0S10$returned Then
GoTo L$FINISH
End Else
loop$V0S10$returned = @True
GoTo L$V0S9$RETURN$1
End
**************************************************
L$V0S16$START:
*** End of loop "EndLoop_Activity_16": Iterate until finished
GoTo L$V0S10$REPEAT
**************************************************
L$V0S16$COMPLETE:
loop$context = loop$prevcontexts<1>
If loop$V0S16$triggered Then GoTo L$V0S10$FINISHED
loop$V0S16$triggered = @True
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0037\Loop finished", ""), "@EndLoop_Activity_16")
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0071\%1 loop finished", "EndLoop_Activity_16"))
GoTo L$V0S10$FINISHED
**************************************************
L$V0S22$START:
*** Activity "Routine_Activity_22": Call routine
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0058\%1 (ROUTINE %2) started", "Routine_Activity_22":@FM:"DSX.UTILITYRUNJOB"))
rtn$ok = DSCheckRoutine("DSX.UTILITYRUNJOB")
If (Not(rtn$ok)) Then
err$source = "Routine_Activity_22"; err$number = -1
msg$ = DSMakeMsg("DSTAGE_JSG_M_0005\BASIC routine is not cataloged: %1", "DSX.UTILITYRUNJOB")
msg$id = "@Routine_Activity_22"; GoTo L$ERROR
End
p$V0S22$1 = (IdV0S10%%Counter%%4)
p$V0S22$2 = ('')
p$V0S22$3 = (0)
p$V0S22$4 = (1)
Call 'DSX.UTILITYRUNJOB'(r$V0S22, p$V0S22$1, p$V0S22$2, p$V0S22$3, p$V0S22$4)
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0064\%1 finished, reply=%2", "Routine_Activity_22":@FM:r$V0S22))
IdV0S22%%Result1%%5 = r$V0S22
IdV0S22%%Name%%6 = "DSX.UTILITYRUNJOB"
IdV0S22%%Param%%7 = (p$V0S22$1)
IdV0S22%%Param%%8 = (p$V0S22$2)
IdV0S22%%Param%%9 = (p$V0S22$3)
IdV0S22%%Param%%10 = (p$V0S22$4)
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0039\Omitted checkpoint for call of routine '%1'", "DSX.UTILITYRUNJOB"), "@Routine_Activity_22")
b$V0S22else = @True
If b$V0S22else Then GoSub L$V0S24$START
If (r$V0S22 = 0) Then b$V0S22else = @False; GoTo L$V0S16$START
L$V0S22$RETURN$1:
GoTo L$V0S10$RETURN$1
**************************************************
L$V0S24$START:
*** Sequencer "Sequencer_24": wait until inputs ready
seq$V0S24$status = 1
seq$V0S24$count<loop$context> += 1
If Sum(seq$V0S24$count) < 1 Then Return
seq$V0S24$status = 2
gosub$stack += 1
gosub$stack -= 1
Return
**************************************************
L$ERROR:
Call DSLogWarn(DSMakeMsg("DSTAGE_JSG_M_0009\Controller problem: %1", msg$), msg$id)
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0052\Exception raised: %1", msg$id:", ":msg$))
If Not(b$InErrorRef) Then
dummy$ = DSCheckPointSetValue(DSJ.ME, "!", DSMakeMsg("DSTAGE_JSG_M_0031\Failed at: %1, Reason=%2", err$source:@FM:msg$))
End
L$ABORT:
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0055\Sequence failed (restartable)", ""))
Call DSLogInfo(summary$, "@Coordinator")
Call DSLogFatal(DSMakeMsg("DSTAGE_JSG_M_0014\Sequence job (restartable) will abort due to previous unrecoverable errors", ""), "@Coordinator")
**************************************************
L$FINISH:
*** At successful exit remove checkpoint record
If Not(b$InErrorRef) And DSCheckPointExists(DSJ.ME, cp$dtm) Then
dummy$ = DSCheckPointDelete(DSJ.ME)
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0018\Removed checkpoint record at successful completion of sequence", ""), "@Coordinator")
End
If loop$V0S10$entered And Not(loop$V0S10$done) Then
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0075\Note: Loop '%1' was started, but did not complete all its iterations", "StartLoop_Activity_10"), "@Coordinator")
End
If (seq$V0S24$status = 1) Then
Call DSLogInfo(DSMakeMsg("DSTAGE_JSG_M_0072\Note: Sequencer '%1' was entered, but never exited", "Sequencer_24"), "@Coordinator")
End
summary$<1,-1> = Time$$:Convert(@VM, " ", DSMakeMsg("DSTAGE_JSG_M_0054\Sequence finished OK", ""))
Call DSLogInfo(summary$, "@Coordinator")
L$EXIT: Return To L$EXIT

chulett · Post by **chulett** » Fri Nov 04, 2011 9:01 am

What does UTILITYRUNJOB return? If it does 1 = Success, 2 = Warnings (etc) then that trigger will not consider that "OK" as it needs to return a 0 for OK.

jeet_81 · Post by **jeet_81** » Fri Nov 04, 2011 9:29 am

here are the logs which I have:

(i)dummy..JobControl (@Coordinator): Starting new run of checkpointed Sequence job
(ii)dummy..JobControl (@Script_to_get_seq): Executed: sh /home/dsdeveloper/dataarea/Dev/Staging/S2l/Emcv/temp/Biswajeet/Script_to_get_Jobnames.sh /home/dsdeveloper/dataarea/Dev/Staging/S2l/Emcv/temp/Biswajeet/Sequence_to_run.txt
Reply=0
Output from command ====>
js_0201100_ExtrLoadTran_WklyPetData,
js_1310011_travel_sequence
(iii)dummy..JobControl (@Script_to_get_seq): Checkpointed execution of command 'sh /home/dsdeveloper/dataarea/Dev/Staging/S2l/Emcv/temp/Biswajeet/Script_to_get_Jobnames.sh'
(iv) dummy..JobControl (@StartLoop_Activity_10): Loop through list (2 items): iteration 1
(v) dummy..JobControl (UtilityRunJob): Attaching job for processing - js_0201100_ExtrLoadTran_WklyPetData - Status of Attachment = 2
(vi) dummy -> (js_0201100_ExtrLoadTran_WklyPetData): Job run requested
Mode (row/warn limits) = 0/1
Job Parameters --->
DSJobController=dummy
(vii)dummy -> (js_0201100_ExtrLoadTran_WklyPetData): Job run requested
Mode (row/warn limits) = 0/1
Job Parameters --->
DSJobController=dummy
(viii)dummy..JobControl (DSRunJob): Waiting for job js_0201100_ExtrLoadTran_WklyPetData to start
(ix) dummy..JobControl (DSWaitForJob): Waiting for job js_0201100_ExtrLoadTran_WklyPetData to finish
(x)dummy..JobControl (DSWaitForJob): Job js_0201100_ExtrLoadTran_WklyPetData has finished, status = 1 (Finished OK)
(Xi)dummy..JobControl (UtilityRunJob): Getting job statistics
(xii)dummy..JobControl (UtilityRunJob): List of Stages=%1
(xiii)dummy..JobControl (UtilityRunJob): RunJob Status=js_0201100_ExtrLoadTran_WklyPetData
(Xiv)dummy..JobControl (@Routine_Activity_22): Omitted checkpoint for call of routine 'DSX.UTILITYRUNJOB'
(XV)dummy..JobControl (@Coordinator): Removed checkpoint record at successful completion of sequence
(xvi)dummy..JobControl (@Coordinator): Note: Loop 'StartLoop_Activity_10' was started, but did not complete all its iterations
(xvii)dummy..JobControl (@Coordinator): Summary of sequence run
13:58:17: Sequence started (checkpointing on)
13:58:17: Script_to_get_seq (COMMAND sh /home/dsdeveloper/dataarea/Dev/Staging/S2l/Emcv/temp/Biswajeet/Script_to_get_Jobnames.sh) started
13:58:19: Script_to_get_seq finished, reply=0
13:58:19: StartLoop_Activity_10 loop iteration 1 started
13:58:19: Routine_Activity_22 (ROUTINE DSX.UTILITYRUNJOB) started
14:00:06: Routine_Activity_22 finished, reply=js_0201100_ExtrLoadTran_WklyPetData=1
14:00:06: Sequence finished OK
(XViii)Finished Job dummy.

I hope one can distinguish between the job names

ray.wurlod · Post by **ray.wurlod** » Fri Nov 04, 2011 2:59 pm

DSUtilityRunJob has returned 1. Look in its job log for warnings.