Reading MQ Messages using DataStage
Moderators: chulett, rschirm, roy
Reading MQ Messages using DataStage
We are building a sample prototype to test reading MQ messages and
processing through DS.
We have installed MQ 6 on the DataStage Server (7.5.2).
Setup the Queues.
Cretead a test job to read the message from MQ.
Job is able to connect to the Queue manager, but not able to read the
message, its waiting indefinitely.
Are there any configurations that need to be done, to read MQ messages using DataStage, MQ stage?
processing through DS.
We have installed MQ 6 on the DataStage Server (7.5.2).
Setup the Queues.
Cretead a test job to read the message from MQ.
Job is able to connect to the Queue manager, but not able to read the
message, its waiting indefinitely.
Are there any configurations that need to be done, to read MQ messages using DataStage, MQ stage?
Suren V
I would do a couple of things:
a) put in a bogus queue manager name. and a bogus queue name. Be sure you get clean MQ failure messages in the log. This would at least absolutly ensure that you are "talking" with the MQ API properly. 2035 errors and such happen when queue details are wrong.
b) test with a single large varchar column as your payload. Nothing else in initial tests.
c) be sure you have a simple character message in your queue.
d) try writing a message as a target
When doing testing, always set the wait time to something like 10 or 20 initially, until you understand the behavior. do NOT leave it at the default -1. I also always set the message limit to 10 initially when doing config testing.
Ernie
a) put in a bogus queue manager name. and a bogus queue name. Be sure you get clean MQ failure messages in the log. This would at least absolutly ensure that you are "talking" with the MQ API properly. 2035 errors and such happen when queue details are wrong.
b) test with a single large varchar column as your payload. Nothing else in initial tests.
c) be sure you have a simple character message in your queue.
d) try writing a message as a target
When doing testing, always set the wait time to something like 10 or 20 initially, until you understand the behavior. do NOT leave it at the default -1. I also always set the message limit to 10 initially when doing config testing.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Hi Ernie,
Thanks for the response.
I have tried these options:
When i specify incorrect Queue name, i get the 2035 errors.
When i specify timeout value and the correct Queue name, job writes some garbage output to the file, may be that is result of timeout. But i don't see the actual message being written to the file.
Thanks in Advance.
Thanks for the response.
I have tried these options:
When i specify incorrect Queue name, i get the 2035 errors.
When i specify timeout value and the correct Queue name, job writes some garbage output to the file, may be that is result of timeout. But i don't see the actual message being written to the file.
Thanks in Advance.
Suren V
Make sure that "you" are entering the messages into the queue when doing such validation. Use DataStage, or amqsput (MQ command line utility -- do some searches --- you'll find the syntax)... Put "Hello World" into a message. Be sure that it is a simple datagram message --- pure text, nothing else, no response messages or anything complex or conversational, nothing from a JMS client, etc.....
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
And you only have one column in your job that reads the queue? A large varchar? ...and the Job is something like MQ to Transformer to Sequential?
Make sure it is that simple. Also make sure that your message is still in the queue.
Ernie
Make sure it is that simple. Also make sure that your message is still in the queue.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
I was able to make the job work, by defining the 2 meta data columns. one with Message ID and the other as Message.
I was able to read the messages.
Iam still running into the following issues:
Issue 1: DS job is not able to read the message when the wait time is set to unlimited within the job. Job keeps waiting forever even though there are messages in the Queue that need to be read. (We want the wait time to be unlimited within the DS job, so that DS can keep looking for messages arrival and process the messages as and when they arrive in the Queue)
Issue 2: Once i specify some wait time (i specified 60 seconds), DS is successfully reading the message as expected and job is getting finished and coming out of the waiting loop.
We want the DS job to wait forever for reason specified in Issue 1.
Issue 3: If there are multiple messages waiting in the Queue, DS is reading all of them at once (as one big message).
I was able to read the messages.
Iam still running into the following issues:
Issue 1: DS job is not able to read the message when the wait time is set to unlimited within the job. Job keeps waiting forever even though there are messages in the Queue that need to be read. (We want the wait time to be unlimited within the DS job, so that DS can keep looking for messages arrival and process the messages as and when they arrive in the Queue)
Issue 2: Once i specify some wait time (i specified 60 seconds), DS is successfully reading the message as expected and job is getting finished and coming out of the waiting loop.
We want the DS job to wait forever for reason specified in Issue 1.
Issue 3: If there are multiple messages waiting in the Queue, DS is reading all of them at once (as one big message).
Suren V
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
In version 7 the MQ Series server must be co-resident with the DataStage server. Is this requirement satisfied? (The requirement is relaxed when you upgrade to version 8 DataStage.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
You are experiencing the classic end-of-wave issue. The problem occurs in Parallel Jobs because the buffers generally can't be set small enough to force data thru the entire Job, and it will happen in Server Jobs (and Parallel Jobs) if use are using a blocking Stage such as Aggregator that requires all the rows to be processed before allowing a row to leave.....
...if it's the Aggregator (or Sort or XMLOutput or QSMatch --- any stage that wants to touch "all the rows" before releasing any), then remove it if possible --- if the Job is "straight" (just transforms and such), and you are still on release 7, consider using Server, or else move to Release 8 where you have an end-of-wave option on the MQ Connector.
End-of-Wave forces all downsteam stages to act as though the job is complete, and flushes things while allowing the job to continue to run. I have a description of it on my blog that goes into more details.
Ernie
...if it's the Aggregator (or Sort or XMLOutput or QSMatch --- any stage that wants to touch "all the rows" before releasing any), then remove it if possible --- if the Job is "straight" (just transforms and such), and you are still on release 7, consider using Server, or else move to Release 8 where you have an end-of-wave option on the MQ Connector.
End-of-Wave forces all downsteam stages to act as though the job is complete, and flushes things while allowing the job to continue to run. I have a description of it on my blog that goes into more details.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Transaction semantics and messages are two separate things.....quite often they are indeed 1:1, but sometimes they are 1:many, many:1 and even many:many --- it depends on the payload of the message.
Transaction semantics is a whole separate dialog in itself.....as for messages, however, and to your question, DS always reads one message at a time...it can "browse" for messages, and even search for them, but it always reads one message to put on an output link ---- it can, however, chop that message up if in fact it contains multiple "records". This isn't too common, but happens every once in awhile if a message is really a "file" with embedded CRLF's or LF's. Then, for example, DataStage will read one message, but spit out "n" 'rows' on the output link. There is a property on the Stage that specifies this...
Ernie
Transaction semantics is a whole separate dialog in itself.....as for messages, however, and to your question, DS always reads one message at a time...it can "browse" for messages, and even search for them, but it always reads one message to put on an output link ---- it can, however, chop that message up if in fact it contains multiple "records". This isn't too common, but happens every once in awhile if a message is really a "file" with embedded CRLF's or LF's. Then, for example, DataStage will read one message, but spit out "n" 'rows' on the output link. There is a property on the Stage that specifies this...
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
I was more meaning is there a setting that will control whether DataStage processes a message as it receives it or whether it waits for X number of messages or an EOW (to try explain why DataStage is refusing to read the messages unless a timeout is specified).
Looking through the options... "Message quantity" and "Record count" have promise to this... although how it handles different values of this I cannot check.
Of course, make sure no message composition options are set either otherwise it may be looking for a missing message...
Looking through the options... "Message quantity" and "Record count" have promise to this... although how it handles different values of this I cannot check.
Of course, make sure no message composition options are set either otherwise it may be looking for a missing message...