Always-on job using MQ stages

thebird · Post by **thebird** » Wed May 12, 2010 9:15 am

Hi Ernie...

I haven't touched anything on the interprocess row buffering settings. And I am not sure how to go about it either... Can you throw some light on it, so that I can check it out here from my side.

eostic · Post by **eostic** » Wed May 12, 2010 11:25 am

Go to Edit Job Properties...performance settings...in there are settings for interprocess row buffering.... I want you to have "none".

Also... any chance you are validating the xml?

Ernie

thebird · Post by **thebird** » Wed May 12, 2010 12:02 pm

Ernie!!!!

That seemed to have done the trick!!! :D

The interprocess row buffering, when i disabled it, messages are moving out to the target queue... I am still testing it to make sure that I aint day dreaming...

Will update here in sometime!

Aneesh

eostic · Post by **eostic** » Wed May 12, 2010 12:08 pm

I suspected as much....it had to be "something" that was impacting the behavior of row shipment. I knew I've used xmlInput in real time jobs a ton of times......

...and for something like this, you won't need interprocess row buffering. I suppose someone at your site felt it was necessary, but I personally am very against having it ever be used as a project default. It changes the behavior of DataStage in ways that are important for 'some' jobs needing the performance boost --- but it should only be used with extreme awareness.

Keep us posted.

Ernie

thebird · Post by **thebird** » Wed May 12, 2010 4:42 pm

Works like a charm!!!

Tested it out having the XML Output before the final queue as well, and it worked.

Thanks for holding on till the end, Ernie.

Now before I mark this as resolved, I have a couple of questions regarding the always-on topology.

a) Prior to v8.x - can we not have an always-on Parallel job involving multiple stages, because the end-of-wave issue is going to be playing in then.

b) If not - how is the eow issue over come at all? Is going the Server way the only solution? Can the eow be generated by another application - say like Message Broker or MQ Series itself - an application that is putting messages into a queue - so that Parallel jobs can be brought into the picture?

b) Wouldnt the eow issue also be applicable to Parallel RTI jobs? I would assume so - considering that the behaviour of the stages do should not change be it RTI job or a batch job. Is that a fair statement?

Thanks!

Aneesh

chulett · Post by **chulett** » Wed May 12, 2010 5:55 pm

eostic wrote:I suppose someone at your site felt it was necessary, but I personally am very against having it ever be used as a project default. It changes the behavior of DataStage in ways that are important for 'some' jobs needing the performance boost --- but it should only be used with extreme awareness.

This. Times 1000. It is not a Silver Bullet, folks.

eostic · Post by **eostic** » Thu May 13, 2010 5:18 am

First.... Congrats! Glad you stuck it out too. DS can do some wonderful things in real time.

a) EE jobs were designed for heavy volume. Major batch stuff. Grid, etc. They have buffers built into the natural pipeline processing that occurs by default. When we run in real time, we have to somehow "override" those buffers to "flush" data thru the job. No way to flush them without End of Wave......so........prior to the new MQ Connector and the End of Wave operator, most non-RTI real time EE jobs will hang. Huge messages might exceed the buffer, but it is a guessing game pre MQ Connnector and EOW Operator.

b) There is no way to force an End of Wave...it's an internal concept that makes the job "think" that it is finished....sends sort of the same signal that goes thru the job when the starting stage finally hits end of file.

c) RTI forces EOW......RTI was the first tooling 8 years ago that "invented" the concept. Then, in v8 we added it explicitly for real time that is crafted by the developer (MQ or your own personal real time stages for JMS or other custom operators). End of Wave has to be recognized by all operators in EE....it wasn't until release 8 that most all operators were recognizing it fully.

Server is invaluable for real time jobs, in v7 and in v8, for architectural reasons that go beyond end of wave. Keep it in your back pocket for special requirements, but certainly EE can do real time now also, which is critical when you are doing QualityStage with real time, or sharing large amounts of DataStage logic with a heavy batch process.

Ernie

thebird · Post by **thebird** » Thu May 13, 2010 10:00 am

wow!! That was a lot of information...Helped to set things in the right perspective. Thanks Ernie.

Let me take sometime to digest and think over this. Meanwhile, marking this thread resovled.

Aneesh