Improving Sequence Compilation times

SIHM · Post by **SIHM** » Wed Feb 18, 2015 2:00 am

I'm getting slow responses from compiling my Datastage job sequences. They take about 30 minutes.

I have removed the calling of routines via the routine stage and put them in a user variable stage instead. This helps but still they are pretty slow

Admittedly, there are 40 jobs in the sequence with 80 routine calls. We are obliged to keep the job sequences big as they are grouped by functionality.

qt_ky · Post by **qt_ky** » Wed Feb 18, 2015 6:01 am

That is disturbing. Is your client by any chance very, very remote from the server, perhaps using an old style dial up connection?

Have you checked on Fix Central or with Support to see if there is any patch available?

SIHM · Post by **SIHM** » Wed Feb 18, 2015 6:13 am

We have looked at a few patches but they only seem to relate to PX jobs not Job Sequences as they mention BUILDOP flags.

There were others but they were for 8.7 DB2 connectors, which again we dont have.

qt_ky · Post by **qt_ky** » Wed Feb 18, 2015 6:14 am

What version are you on?

How does your compile time compare to a client that is "right next door" (on the network) to your DataStage server?

SIHM · Post by **SIHM** » Wed Feb 18, 2015 6:17 am

We are using version 9.1.0.1 but as I mentioned the PX jobs with many stages in compile in no time at all its only the job sequences.

We thought it might be a project issue so we imported the objects into another project but still have the same issue.

arvind_ds · Post by **arvind_ds** » Wed Feb 18, 2015 6:24 am

Raise a ticket with your official support provider.

chulett · Post by **chulett** » Wed Feb 18, 2015 8:28 am

Since this is specifically about Sequence jobs, pushed this over to the General forum. Carry on.

Question: Does this behavior occur for every developer? Is this something that changed as in was fine but then went sideways or is this an always been this way kind of thing?

vibwipro · Post by **vibwipro** » Wed Feb 18, 2015 9:05 am

Hi,

If project server is working fine, I will suggest to split your Seq may be at least in 2 to 3 parts. Holding around 120+ stages on a single canvas is bit hard and you may land up with such problems.

Regards.

SIHM · Post by **SIHM** » Wed Feb 18, 2015 9:08 am

Ah, yes, Apologies. Force of habit I'm afraid

Yes, every developer is suffering at the moment. I have requested that we keep the JS to a maximum of 10 jobs and we will just have to break the functional rule.

It does seem a little bizarre though. I have never experienced this elsewhere.

chulett · Post by **chulett** » Wed Feb 18, 2015 10:30 am

Hopefully you've involved support in this, yes?

ray.wurlod · Post by **ray.wurlod** » Wed Feb 18, 2015 2:29 pm

Sequences are compiled differently than regular DataStage jobs, with some of the effort actually occurring on the client. Perhaps a faster connection between client and engine would help you.

qt_ky · Post by **qt_ky** » Wed Feb 18, 2015 2:35 pm

Again, how does your compile time compare to a client that is "right next door" (on the network) to your DataStage server?

chulett · Post by **chulett** » Wed Feb 18, 2015 3:44 pm

I thought practically all of a Sequence job's compilation 'effort' happened on the client.

ray.wurlod · Post by **ray.wurlod** » Wed Feb 18, 2015 4:27 pm

chulett wrote:I thought practically all of a Sequence job's compilation 'effort' happened on the client.

Yes, but all of the objects to be compiled have to be transferred from the engine, and various resulting objects need to be transferred back to the engine. A slow network plays havoc with compilation time.

SIHM · Post by **SIHM** » Thu Feb 19, 2015 8:52 am

I'll have to find out where the machine is! We dont seem to have any other network issues though.