Reading MQ Messages using DataStage

CGI_SUREN · Post by **CGI_SUREN** » Thu Nov 05, 2009 10:40 am

We are building a sample prototype to test reading MQ messages and
processing through DS.

We have installed MQ 6 on the DataStage Server (7.5.2).

Setup the Queues.

Cretead a test job to read the message from MQ.

Job is able to connect to the Queue manager, but not able to read the
message, its waiting indefinitely.

Are there any configurations that need to be done, to read MQ messages using DataStage, MQ stage?

eostic · Post by **eostic** » Thu Nov 05, 2009 10:53 am

I would do a couple of things:

a) put in a bogus queue manager name. and a bogus queue name. Be sure you get clean MQ failure messages in the log. This would at least absolutly ensure that you are "talking" with the MQ API properly. 2035 errors and such happen when queue details are wrong.

b) test with a single large varchar column as your payload. Nothing else in initial tests.

c) be sure you have a simple character message in your queue.

d) try writing a message as a target

When doing testing, always set the wait time to something like 10 or 20 initially, until you understand the behavior. do NOT leave it at the default -1. I also always set the message limit to 10 initially when doing config testing.

Ernie

CGI_SUREN · Post by **CGI_SUREN** » Thu Nov 05, 2009 2:12 pm

Hi Ernie,

Thanks for the response.
I have tried these options:

When i specify incorrect Queue name, i get the 2035 errors.

When i specify timeout value and the correct Queue name, job writes some garbage output to the file, may be that is result of timeout. But i don't see the actual message being written to the file.

Thanks in Advance.

eostic · Post by **eostic** » Thu Nov 05, 2009 2:39 pm

Make sure that "you" are entering the messages into the queue when doing such validation. Use DataStage, or amqsput (MQ command line utility -- do some searches --- you'll find the syntax)... Put "Hello World" into a message. Be sure that it is a simple datagram message --- pure text, nothing else, no response messages or anything complex or conversational, nothing from a JMS client, etc.....

Ernie

CGI_SUREN · Post by **CGI_SUREN** » Thu Nov 05, 2009 5:04 pm

I have entered the message into the queue using 'amqsput' command and its a pure text.
I still having the same problem.
Do i have to specify the 'Message Type' property?

I was able to write a message from DataStage to MQ. This part works fine.

Thanks in Advance.

eostic · Post by **eostic** » Fri Nov 06, 2009 6:51 am

And you only have one column in your job that reads the queue? A large varchar? ...and the Job is something like MQ to Transformer to Sequential?

Make sure it is that simple. Also make sure that your message is still in the queue.

Ernie

CGI_SUREN · Post by **CGI_SUREN** » Fri Nov 13, 2009 2:48 pm

I was able to make the job work, by defining the 2 meta data columns. one with Message ID and the other as Message.
I was able to read the messages.

Iam still running into the following issues:

Issue 1: DS job is not able to read the message when the wait time is set to unlimited within the job. Job keeps waiting forever even though there are messages in the Queue that need to be read. (We want the wait time to be unlimited within the DS job, so that DS can keep looking for messages arrival and process the messages as and when they arrive in the Queue)

Issue 2: Once i specify some wait time (i specified 60 seconds), DS is successfully reading the message as expected and job is getting finished and coming out of the waiting loop.
We want the DS job to wait forever for reason specified in Issue 1.

Issue 3: If there are multiple messages waiting in the Queue, DS is reading all of them at once (as one big message).

ray.wurlod · Post by **ray.wurlod** » Fri Nov 13, 2009 3:09 pm

In version 7 the MQ Series server must be co-resident with the DataStage server. Is this requirement satisfied? (The requirement is relaxed when you upgrade to version 8 DataStage.)

eostic · Post by **eostic** » Sat Nov 14, 2009 12:14 pm

You are experiencing the classic end-of-wave issue. The problem occurs in Parallel Jobs because the buffers generally can't be set small enough to force data thru the entire Job, and it will happen in Server Jobs (and Parallel Jobs) if use are using a blocking Stage such as Aggregator that requires all the rows to be processed before allowing a row to leave.....

...if it's the Aggregator (or Sort or XMLOutput or QSMatch --- any stage that wants to touch "all the rows" before releasing any), then remove it if possible --- if the Job is "straight" (just transforms and such), and you are still on release 7, consider using Server, or else move to Release 8 where you have an end-of-wave option on the MQ Connector.

End-of-Wave forces all downsteam stages to act as though the job is complete, and flushes things while allowing the job to continue to run. I have a description of it on my blog that goes into more details.

Ernie

CGI_SUREN · Post by **CGI_SUREN** » Mon Nov 23, 2009 3:46 pm

Hi Ray,

Yes this requirement is satisfied.

Thanks

CGI_SUREN · Post by **CGI_SUREN** » Mon Nov 23, 2009 3:48 pm

Hi Ernie,

The job is simple with no transformations.
We are on 7.5.2

Could you please send me the link for your blog.

Thanks in Advance.

chulett · Post by **chulett** » Mon Nov 23, 2009 4:07 pm

The link is in his sig.

Kryt0n · Post by **Kryt0n** » Mon Nov 23, 2009 4:19 pm

Is there a transaction option? Something to tell DataStage to read one message at a time?

eostic · Post by **eostic** » Mon Nov 23, 2009 8:24 pm

Transaction semantics and messages are two separate things.....quite often they are indeed 1:1, but sometimes they are 1:many, many:1 and even many:many --- it depends on the payload of the message.

Transaction semantics is a whole separate dialog in itself.....as for messages, however, and to your question, DS always reads one message at a time...it can "browse" for messages, and even search for them, but it always reads one message to put on an output link ---- it can, however, chop that message up if in fact it contains multiple "records". This isn't too common, but happens every once in awhile if a message is really a "file" with embedded CRLF's or LF's. Then, for example, DataStage will read one message, but spit out "n" 'rows' on the output link. There is a property on the Stage that specifies this...

Ernie

Kryt0n · Post by **Kryt0n** » Mon Nov 23, 2009 8:54 pm

I was more meaning is there a setting that will control whether DataStage processes a message as it receives it or whether it waits for X number of messages or an EOW (to try explain why DataStage is refusing to read the messages unless a timeout is specified).

Looking through the options... "Message quantity" and "Record count" have promise to this... although how it handles different values of this I cannot check.

Of course, make sure no message composition options are set either otherwise it may be looking for a missing message...