Staggered start in a sequence

[PhilHibbs]

I want three jobs to run independently of each other, but I want them to start in a staggered sequence - the first one immediately, the second one after 1 minute, and the third one after 2 minutes. How can I do this? A while ago I did find a stage - it might have been an Execute Command - that I could use to implement a time delay, but I found that this time delay kicked in first and prevented anything else from happening until it had completed, and then the first and second jobs both started simultaneously.

[mouthou]

yes. Execute command will do the trick. But what do you mean by it prevented other things. are you running other jobs in parallel etc.

if you introduce a delay between two jobs, it will be only for the two.

[PhilHibbs]

yes. Execute command will do the trick. But what do you mean by it prevented other things. are you running other jobs in parallel etc.

What I expected was that it would start J1 and also the Execute Command stage both in parallel, and then start J2 when the Execute Command finished. What actually happens is that it runs the Execute Command, waits for it to finish, and then starts J2 and J1. It doesn't "wait for" in the same sense that it loops round monitoring a job status waiting for it to finish - it shells out to the command specified and only resumes doing anything else when the shell returns.

Code: Select all

        +----+        +-----+
		  | J1 |        | Cmd |
        +----+        +-----+
								 |
								 |
							 +-----+
							 |  J2 |
							 +-----+

[chulett]

You won't be able to do this with stages in a Sequence, you'll need hand code. Use DSRunJob() to start the first job but then don't immediately call DSWaitForJob() like the generated code does but sleep then start your second job. After both have started you can go back to the 'wait for job' call for both.

Or, if there are downstream activities dependant on these two and you want it to move along as soon as possible, build a loop to check the status of both and exit when both checks show completion.

[PhilHibbs]

You won't be able to do this with stages in a Sequence, you'll need hand code.

I don't like that idea. I think a DataStage job that runs a sleep command in a BeforeJobRoutine would be better. That way, it's clear what is happening when someone opens the job design, and the sequence can be amended and added to just like any other sequence.

[chulett]

If that 'delay' works for you. You still won't be able to manage a true 'staggered start' in the Sequence job, however. You could, however, arrange for both to start at the same time and then delay one as you noted. Only 'downside' I can see is the fact that it will always have that built in delay, even when run stand-alone.

[PhilHibbs]

You could, however, arrange for both to start at the same time and then delay one as you noted. Only 'downside' I can see is the fact that it will always have that built in delay, even when run stand-alone.

I meant a separate Server Job whose purpose is only to do the sleep command.

[chulett]

Ah... no need for that, the answer is so simply I'm ashamed it took so long for it to pop into my brain. Back to two jobs and a routine...

Code: Select all

         +---RA----JA
Seqr ----|
         +---JA

Staggered start.

[PhilHibbs]

So just drop a Sequencer stage in before J1 and the Execute Command stage? That fails to compile, "This activity must be triggered by another activity".

*Update*: Oh, RA is Routine Activity?

*Update2* Routine Activity behaves the same as Execute Command, they suspend execution of the Sequence Job until the timer expires, so if I have J1 immediate, J2 after a 10 second delay, and J3 after a 20 second delay, all on three independent activity chains, J3 occurs after 30 seconds not 20.

[chulett]

Yes, RA is Routine Activity. And if it wants something upstream from the Sequencer, put a Nested Condition there with a Unconditional trigger.

[PhilHibbs]

Code: Select all

+----------------------------+
|		  Nested Condition     |
+----------------------------+
	|           |           |
	|           |           |
+----+      +-----+     +-----+
| J1 |      | RA1 |     | RA2 |
+----+      +-----+     +-----+
				   |           |  
				   |           |  
				+----+      +----+
				| J2 |      | J3 |
				+----+      +----+

Still no good. The first thing that happens now is RA2 (sleep 20), then J3, then RA1 (sleep10), then J2 and J1. I want J1 to run first and immediately, but in practice it runs after a 30 second delay, and 10 seconds after J3. Routine Activity and Execute Command behave identically in this regard. A Server Job that just calls sleep as a BeforeJobActivity works fine, though.

Incidentally, how do I get a Server Job to compile if it has no active components? I've dropped a Transformer on sending a single row to /dev/null to get it to compile.

[chulett]

You skipped the Sequencer, something you need to get this to work. That's what "Seqr" meant, btw, a "Sequencer" stage, not that this was the start point for a "Sequence" job. Hence the "r".

The transformer needs a stage variable to compile. You don't need to use it, it just needs to be there. I use 'svDummy' for that. And if you don't really need that single row, a constraint set to @FALSE works just fine. Really depends on exactly what you are doing there. If it's just so you can have an 'empty' canvas so a Before-job routine can be the only thing run, try putting a comment or a call to DSLogInfo() in the Job Control tab instead. Of course, an annotation on the canvas stating all that is going on would be good, too.

[ArndW]

Why not put an execute command stage for "sleep 60" and "sleep 120" into the stream for each job that should be staggered?

[PhilHibbs]

You skipped the Sequencer, something you need to get this to work. That's what "Seqr" meant, btw, a "Sequencer" stage, not that this was the start point for a "Sequence" job. Hence the "r".

Like this?

Code: Select all

+-----------------------------+
|        Nested Condition     |
+-----------------------------+
   |           |           |
+-----+     +-----+     +-----+
| Sqr |     | Sqr |     | Sqr |
+-----+     +-----+     +-----+
   |           |           |
+----+      +-----+     +-----+
| J1 |      | RA1 |     | RA2 |
+----+      +-----+     +-----+
               |           | 
            +----+      +----+
            | J2 |      | J3 |
            +----+      +----+

The Sequencers makes absolutely no difference. I still get 20 seconds, then J3, then 10 seconds, then J2 and J1. The problem is that the Routine Activity stops the Sequence job dead in its tracks, nothing can happen until the timer elapses, and all three activities J1, RA2, and RA2 happen in an arbitrary order which presumably depends on the order in which I linked up the stages.

If it's just so you can have an 'empty' canvas so a Before-job routine can be the only thing run, try putting a comment or a call to DSLogInfo() in the Job Control tab instead.

Ah, a single * in the Job Control does the trick. Thanks.

*Update:*
This doesn't work either:

Code: Select all

+-----------------------------+
|        Nested Condition     |
+-----------------------------+
               |
+-----------------------------+
|          Sequencer          |
+-----------------------------+
   |           |           |
+----+      +-----+     +-----+
| J1 |      | RA1 |     | RA2 |
+----+      +-----+     +-----+
               |           | 
            +----+      +----+
            | J2 |      | J3 |
            +----+      +----+

[PhilHibbs]

all three activities J1, RA2, and RA2 happen in an arbitrary order which presumably depends on the order in which I linked up the stages...

Indeed, detaching the links to RA1 and J1 and re-attaching them in reverse order fixes my problem. I get J1 and a 10 second delay, then J2 and a 20 second delay, then J3. So RA2 only needs to be 10 seconds, as it has to wait for RA1 to finish before it starts. This is still not ideal, though, as any disruption to the link ordering will break it.

*Update:* Marking as resolved, as I can use a Server Job that calls the sleep command. It appears that there is no way to do this with Execute Command or Routine Activity stages, as they both suspend execution of the Sequence job until they have elapsed. Also the Sequencer and Nested Condition stages are not needed:

Code: Select all

+----+  +----+  +----+
| J1 |  | S1 |  | S2 |
+----+  +----+  +----+
           |       |
        +----+  +----+
        | J2 |  | J3 |
        +----+  +----+

S1 and S2 are the Server Job that calls the sleep as a BeforeJobRoutine, with 10 and 20 seconds respectively. I assume the job has to Allow Multiple Instances, so I ticked that.