Ideal sequence of jobs in a sequence

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
splayer
Charter Member
Charter Member
Posts: 502
Joined: Mon Apr 12, 2004 5:01 pm

Ideal sequence of jobs in a sequence

Post by splayer »

I have 30+ jobs and none of them are dependent on another. Some of them take a long time to finish while others finish in seconds. What would be the ideal way to place them in a job sequence so that performance for the overall sequence is best?
mk_ds09
Participant
Posts: 72
Joined: Sun Jan 25, 2009 4:50 pm
Location: Pune

Post by mk_ds09 »

L ->
| M
L -> | Max
| M
L -> | Max
| M
L - >


L :- Job takes minimum time...
M :- Medium time
Max :- Max time..

If you can have a design to start the jobs which takes less time simultanously and the job which takes the long time can run when there is less processing to do !

It will help in such way that the job which will require maximum time will have all the resources to use !
-----------------------------------
Regards
MK

What would you attempt to do if you knew you could not fail?

-----------------------------------
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

What do you mean by "performance" in this scenario?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
splayer
Charter Member
Charter Member
Posts: 502
Joined: Mon Apr 12, 2004 5:01 pm

Post by splayer »

By performance, I mean I want the entire sequence to finish as fast possible.
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:

Post by kduke »

I would not put 30 jobs in one sequence. I would run in multiple sequences. Most of this is trial and error. You need to figure out how many jobs can run at the same time. The idea is keep the server at capacity without paging. Monitor your server.

Usually some jobs have to run before others. Like a parent table needs to load before a child table. Figure that order out first.
Mamu Kim
DSguru2B
Charter Member
Charter Member
Posts: 6854
Joined: Wed Feb 09, 2005 3:44 pm
Location: Houston, TX

Post by DSguru2B »

Considering these are parallel jobs I would take Kim's advice on monitoring the server load. Do a bunch of runs with different sequence. Hey if your server can take it you can create multiple sequences and fire all of them at once if there is no dependency.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Since you have version 8, you can use the resource estimator to indicate the likely resources to be consumed by each job. Then assemble them so that any dependencies are addressed, and the total demand for resources remains under 100% of available resources.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply