Message Queue's Wait time
Moderators: chulett, rschirm, roy
Message Queue's Wait time
I need a help related to Message Queue stage in a DataStage Job. Agenda is to make it continuous run, for this in MQ stage under the tab output the value for the Wait time is '-1'. if this is the case jobs are taking more time to run and to process number of records as well as to commit it to the target database.
So I need a help
1.Is there is any other ways so that jobs can process the records coming form MQ fast as well as commit in the similar manner rather than using '-1'.
2. If wait time '-1' is the only one option, how can I change the nature of the job to process records fast as well as to commit the records to the database.
So I need a help
1.Is there is any other ways so that jobs can process the records coming form MQ fast as well as commit in the similar manner rather than using '-1'.
2. If wait time '-1' is the only one option, how can I change the nature of the job to process records fast as well as to commit the records to the database.
regards,
Rakesh
Rakesh
I've used -1 and see no detectable slowdown in read speeds. How did you measure this?
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
This is what my observation:ArndW wrote:I've used -1 and see no detectable slowdown in read speeds. How did you measure this? ...
The total messages in the queue is 480.
scenario-1:
-----------
When i run the job with message limit=480 (non-continuous mode), the job is getting completed within 1 minute.
scenario-2:
-----------
Again when i run the same job with wait time=-1 and message limit=0 (continuos mode), the job is taking more than 45 minutes to process 430 records and the remaining 50 records are not fetched from the MQ. The total records went into target is 390. But the job is still running.
regards,
Rakesh
Rakesh
...not sure what you are saying here...when you run in a continuous fashion, the job NEVER finishes....that is by design....the job "finishing" and "rows being processed" are two different things. Do you have your commit point = 1 at the target? The rows are probably going thru perfectly (unless you have a blocking stage like aggregator or XMLOutput or something in the middle)....
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Change your output to a dataset. Is the behaviour the same? If not, then the cause is on the database side.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Arnd is right...there is some other details missing here...dataset, sequential file...it shouldn't matter...do you lose rows then? of course you can't "look" at them while the job is running, but when it finally finishes, are they all there? Try another relational target...with commit = 1....do you get all the rows? is it a 1:1 scenario? (one message equals one row) There's more to this mystery still...
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
The more I think about it, and now see also that this is a 7.x implemetation (sorry I missed that before), it simply may be the need for end-of-wave in an EE scenario. EE is designed at it's core to handle incredibly large volumes of data. As part of that strategy, it does buffering during pipeline parallelism.....that can be an issue for real-time jobs that need to "force" data thru the process.... so...in v8 we added the ability to force "end of wave" to the MQ stage --- but in 7.x, it could be that your "missing rows" are still "in the buffer"........
I guess the real question above is......are the rows missing even after you "cleanly" terminate the job? Or are they only only missing when you check things during an intermediate time while the job is still running?
Ernie
I guess the real question above is......are the rows missing even after you "cleanly" terminate the job? Or are they only only missing when you check things during an intermediate time while the job is still running?
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
eostic wrote:The more I think about it, and now see also that this is a 7.x implemetation (sorry I missed that before), it simply may be the need for end-of-wave in an EE scenario. EE is designed at it's core to handle incredibly large volumes of data. As part of that strategy, it does buffering during pipeline parallelism.....that can be an issue for real-time jobs that need to "force" data thru the process.... so...in v8 we added the ability to force "end of wave" to the MQ stage --- but in 7.x, it could be that your "missing rows" are still "in the buffer"........
I guess the real question above is......are the rows missing even after you "cleanly" terminate the job? Or are they only only missing when you check things during an intermediate time while the job is still running?
Ernie
yes, rows are missing even after i cleanly terminate the job. if it is not a continuous run all the records will get process and will committ into the target.
regards,
Rakesh
Rakesh
...and if you have a Sequential or dataset target, do you lose records?
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Ok...so now it's time to better understand the job itself. I'll assume it's more than just "MQ to Sequential". What kind of stages do you have in there? Any "blocking" stage types (Aggregator, call to QualityStage, XMLOutput) in there? Anything custom, like a custom operator or buildOp? Something is holding up the rows, and pipelining could still be the culprit, but it's hard to tell at this point.
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
-
- Premium Member
- Posts: 278
- Joined: Wed Oct 03, 2007 8:45 am
My assumption is that it was resolved, but hopefully we'll hear from the original author. The symptoms were sounding just like jobs in EE that naturally "hold" rows in buffers. Putting a limit on the job simply caused it to go thru "shut down" mode sooner, flushing all of the rows. A Server Job wouldn't have had this problem, and an 8.x job in EE with MQ has an end-of-wave option that will also flush the buffers.
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>