Message Queue's Wait time

rakesh.nc · Post by **rakesh.nc** » Wed Mar 12, 2008 10:13 pm

I need a help related to Message Queue stage in a DataStage Job. Agenda is to make it continuous run, for this in MQ stage under the tab output the value for the Wait time is '-1'. if this is the case jobs are taking more time to run and to process number of records as well as to commit it to the target database.
So I need a help
1.Is there is any other ways so that jobs can process the records coming form MQ fast as well as commit in the similar manner rather than using '-1'.
2. If wait time '-1' is the only one option, how can I change the nature of the job to process records fast as well as to commit the records to the database.

ArndW · Post by **ArndW** » Thu Mar 13, 2008 3:26 am

I've used -1 and see no detectable slowdown in read speeds. How did you measure this?

rakesh.nc · Post by **rakesh.nc** » Thu Mar 13, 2008 6:39 am

ArndW wrote:I've used -1 and see no detectable slowdown in read speeds. How did you measure this? ...

This is what my observation:

The total messages in the queue is 480.

scenario-1:

-----------

When i run the job with message limit=480 (non-continuous mode), the job is getting completed within 1 minute.

scenario-2:

-----------

Again when i run the same job with wait time=-1 and message limit=0 (continuos mode), the job is taking more than 45 minutes to process 430 records and the remaining 50 records are not fetched from the MQ. The total records went into target is 390. But the job is still running.

eostic · Post by **eostic** » Thu Mar 13, 2008 7:56 am

...not sure what you are saying here...when you run in a continuous fashion, the job NEVER finishes....that is by design....the job "finishing" and "rows being processed" are two different things. Do you have your commit point = 1 at the target? The rows are probably going thru perfectly (unless you have a blocking stage like aggregator or XMLOutput or something in the middle)....

Ernie

ArndW · Post by **ArndW** » Wed Mar 26, 2008 8:44 am

Change your output to a dataset. Is the behaviour the same? If not, then the cause is on the database side.

eostic · Post by **eostic** » Fri Mar 28, 2008 6:45 am

Arnd is right...there is some other details missing here...dataset, sequential file...it shouldn't matter...do you lose rows then? of course you can't "look" at them while the job is running, but when it finally finishes, are they all there? Try another relational target...with commit = 1....do you get all the rows? is it a 1:1 scenario? (one message equals one row) There's more to this mystery still...

Ernie

eostic · Post by **eostic** » Sun Mar 30, 2008 10:52 am

The more I think about it, and now see also that this is a 7.x implemetation (sorry I missed that before), it simply may be the need for end-of-wave in an EE scenario. EE is designed at it's core to handle incredibly large volumes of data. As part of that strategy, it does buffering during pipeline parallelism.....that can be an issue for real-time jobs that need to "force" data thru the process.... so...in v8 we added the ability to force "end of wave" to the MQ stage --- but in 7.x, it could be that your "missing rows" are still "in the buffer"........

I guess the real question above is......are the rows missing even after you "cleanly" terminate the job? Or are they only only missing when you check things during an intermediate time while the job is still running?

Ernie

rakesh.nc · Post by **rakesh.nc** » Tue Apr 01, 2008 10:54 pm

eostic wrote:The more I think about it, and now see also that this is a 7.x implemetation (sorry I missed that before), it simply may be the need for end-of-wave in an EE scenario. EE is designed at it's core to handle incredibly large volumes of data. As part of that strategy, it does buffering during pipeline parallelism.....that can be an issue for real-time jobs that need to "force" data thru the process.... so...in v8 we added the ability to force "end of wave" to the MQ stage --- but in 7.x, it could be that your "missing rows" are still "in the buffer"........

I guess the real question above is......are the rows missing even after you "cleanly" terminate the job? Or are they only only missing when you check things during an intermediate time while the job is still running?

Ernie

yes, rows are missing even after i cleanly terminate the job. if it is not a continuous run all the records will get process and will committ into the target.

eostic · Post by **eostic** » Wed Apr 02, 2008 5:59 am

...and if you have a Sequential or dataset target, do you lose records?

Ernie

rakesh.nc · Post by **rakesh.nc** » Wed Apr 02, 2008 10:19 pm

eostic wrote:...and if you have a Sequential or dataset target, do you lose records?

Ernie

Yes, same behaviour.

eostic · Post by **eostic** » Thu Apr 03, 2008 6:00 am

Ok...so now it's time to better understand the job itself. I'll assume it's more than just "MQ to Sequential". What kind of stages do you have in there? Any "blocking" stage types (Aggregator, call to QualityStage, XMLOutput) in there? Anything custom, like a custom operator or buildOp? Something is holding up the rows, and pipelining could still be the culprit, but it's hard to tell at this point.

Ernie

sjfearnside · Post by **sjfearnside** » Wed Apr 22, 2009 2:03 pm

Did this issue ever get solved and the resolution not posted or did the originator of the post just give up and do something different? If there was a solution, please post the answer, if not is there any value to leaving this post for others?

eostic · Post by **eostic** » Wed Apr 22, 2009 2:39 pm

My assumption is that it was resolved, but hopefully we'll hear from the original author. The symptoms were sounding just like jobs in EE that naturally "hold" rows in buffers. Putting a limit on the job simply caused it to go thru "shut down" mode sooner, flushing all of the rows. A Server Job wouldn't have had this problem, and an 8.x job in EE with MQ has an end-of-wave option that will also flush the buffers.

Ernie