DSXchange

Hi -

We have a job-sequence which is created in the graphical job-sequencer. There are around 150 jobs in this sequence.

The problem is, that the generated code is wrong. It executes the jobs in a completely wrong order and terminates after around 20 jobs - with no errors. Has anyone experienced similar problems ?

I am going to look deeper into the generated code but it clearly is wrong. I should also mention that the way this sequence was created was to drag all the jobs on to the canvas and afterwards apply the sequences and links. It seems like this might have "confused" the code generator.

Thanks

Regards
Peter

Peter can you post the part of the code which u think might be wrong. Are there any warnings in your 20th job. This might not trigger the 21st job if u r trigger conditions are set to 'Ok Conditional'

IHTH

peterbaun wrote:We have a job-sequence which is created in the graphical job-sequencer. There are around 150 jobs in this sequence.

You really should consider creating a set of 'sub-sequences' that control a smaller amount of jobs apiece and then have a 'master' sequence to run them in order.

The problem is, that the generated code is wrong. It executes the jobs in a completely wrong order and terminates after around 20 jobs - with no errors. Has anyone experienced similar problems ?

I am going to look deeper into the generated code but it clearly is wrong. I should also mention that the way this sequence was created was to drag all the jobs on to the canvas and afterwards apply the sequences and links. It seems like this might have "confused" the code generator.

Haven't seen this, but I didn't really start to use Sequence jobs until the 6.x version was released. This might be part of your issue, 5.x was the first release of Sequence Jobs (IIRC) and may have some issues. Can you upgrade to 6? I think you'll find things (all things actually) much nicer there than when running 5.

I've also routinely layed out all of the jobs on the canvas and then added the links and other stages later, sometimes in no particular order. Never had it run jobs in the wrong order.

Nobody wants to touch this one. 150 jobs? It probably couldn't generate the job control code properly because this most likely exceeded internal capacities in the design logic. There's probably a practical limit at about 50 stages in a canvas.

Let me opine here in my most humble sagely manner. I would look for modularity, reusability, a standard library of common functions and routines. It sounds like you put your entire jobstream into a single sequence. You really need to break this into manageable pieces. Layer your design, build small black boxes that can be individually tested and verified, then structure them together to form an application.

peterbaun wrote:We have a job-sequence which is created in the graphical job-sequencer. There are around 150 jobs in this sequence.

What happens if Job #47 failed? How do Production recover from that?

The bigger the jobs and sequencers are, the more likely unusual errors would appears. It is doubtful Ascential would fix it considering that you're on 5.x -- they would recommend that you upgrade to at least 6.x.

-T.J.

Hi -

Thought that I should had added a comment to why we do this - so we don't look like complete DataStage novices.

We currently have a number of job-sequences all connected in a "master" sequence - so we only have to schedule this one sequence. It is very unsophisticated but works just fine. What we are doing now is going away from using the graphical job sequences to use a much more dynamic and sophisticated job-execution and error-handling system which is coded in Basic. So what we did was to create this one huge sequence job from the original job-sequences - substituting the sequences in the master-sequence with the atomic jobs. The purpose of creating this monster job-sequence was to be able to simulate the structure of the whole job sequence which would be a structural copy of the new meta-data driven way we are going to run jobs sot that we could test and verify that the two methods of executing jobs are doing the same.

So it is not a job that we intend to use in production. As Mr. Bland correctly guesses - yes we put our entire job stream into one job, but it is entirely for testing purposes. And just to emphasize - I wouldn't normally dream of creating a job like this - it would definately be modularized and broken down to manageable pieces - in fact that is how the original jobs were designed

Hope my explanation clarifies the purpose of this.

I think that the comments show that we can expect problems with a job-sequence this large. Anyway I will just have to find another way of testing. The easiest thing will obviously be to break this sequence into smaller parts and execute these one by one in the correct order to preserve the dependencies that exist. It should give me the same result at the end. However I would still think that it should work so I will open a case with Ascential support to hear their view of this.

Thanks for your input.

Regards
Peter

We're not beating up on you. I'm glad you've discovered what a lot of us have been saying about a complex job organization using Sequencers. Even Sequences of nested Sequences of nested Sequences is UGLY.

A metadata driven solution is the best way, IMHO, to tackle jobstreams with hundreds of jobs. My last client has 40 jobstreams for the 40 subject areas in the EDW. Some of the jobstreams have 200-300 jobs. Sequencing is not an option.

We're glad you are only doing this as a stop-gap measure until your ultimate solution is in place. Decomposing your sequence is the right approach. Either way, you probably had resynchronization points in the stream anyway so those become natural breaking points.