Problem in reading response from MQ

pradkumar · Post by **pradkumar** » Mon Sep 14, 2009 10:08 am

Hi All ,

Before posting this i have searched the forum and couldnt find any post related to the below error.

I have a job that reads from a dataset, creates an xml message using the XML output stage and then places the xml messgages on to a MQ using the MQ Connector.And i have a requirement to read the responses back from MQ and splits the transactions based on the result code SUCCESS/FATAL.I have a job with MQ Connector --->T/F--->Sequential file.The options iam using in the MQ stage are

Message Padding --> No ; Message Truncation -->Yes ;Treat EOL as row terminator-->Yes;Error Queue -->No;Filter Messages-->No;

Iam reading the MQ stage as longvarchar datatype with length 30000. so when i run the job with EOL as row terminator --YES ;iam able to read and load the responses into a sequential file as an formatted ouput.but when i try to read the responses using the option EOL as row terminatior-->NO it loads the data into file but the messages looks like

RFH

RFH

and so on.
In the transformer i have logic to grab the result code using the index functions and based on the result code iam splitting the records.Could anyone help me where iam going wrong and what other options i have to use to read the entire transaction in a single line and splits the records.why iam getting only the two fields RFH( may be header) and null values.

I would really appreciate your help.

Thanks in Advance

eostic · Post by **eostic** » Mon Sep 14, 2009 9:09 pm

Hi pradumar...

What is your ultimate goal? Is there a reason you can't use EOL terminator = Yes?

It's not clear to me what error scenario you are trying to detect.

There are many many properties in the new MQ Connector....there are many unique combinations that probably aren't used often together.... if you can explain more, some of us might be able to try a few other things and see what our experience offers, and/or see how it behaves on our own platforms.

Ernie

JRodriguez · Post by **JRodriguez** » Tue Sep 15, 2009 7:12 am

Well . ..

There are two properties in MQ Connector stage that you would like to set to read the entire content of the queue and purge messages already processed

Message quantity = -1 ( read all messages in the queue)
Message read mode = delete ( After reading the message MQ connector purge them from the queue)

Do your response queue have an specific format? Do you know the lay out? How about taking a look at what's coming from MQ without any transformation? Source the response queue and dump the content into a sequential file without any transformation. that will allow you to see the output and play with the MQ Connector's properties

Regarding the datatype I rather prefer to read the MQ queue as char, which will make the output fix and easier to split because its predetermine column sizes, so the job will look like:

MQ Connector ---> Column Import ---> TRF( If you need any) ---> Target

pradkumar · Post by **pradkumar** » Tue Sep 15, 2009 12:36 pm

Hi,
Thanks for the replys.

I have created a job just to dump the responses from MQ into a sequential file. MQ-->sequential file. when i put the option EOL as Yes i can see all the responses in the seq file (in xml formatted output) ex:
RFH
<TCRMService xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="tCRMResponse.xsd">
<ResponseControl>
<ResultCode>FATAL</ResultCode>
<ServiceTime>18064</ServiceTime> --- so on

when i change the EOL to No -- the only messgaes i can see in the file are

RFH
RFH
RFH -- so on..

How do i read one response transaction as single message.so that i can create next job to read the file which has responses in a single messgages and splitt the responses based on result code.I have tried many options in the MQ connector stage but could get success in reading the whole transaction as one messgae.

I would really arreciate if you could tell the right way in reading the response 1) MQSTAGE-->T/F-->SEQUENTIAL FILE OR 2) mq-->DUMP INTO SEQUENTIAL FILE--> NEXT JOB TO READ FROM THE SEQUENTIAL FILE -->T/F -->SUCCESS/FAILURE.
Thanks in Advance

eostic · Post by **eostic** » Tue Sep 15, 2009 3:30 pm

As JRodriguez, noted, read the payload (xml in your case) as char and then process downstream..... in this case, w/XMLInput, or in other cases, if it were (for example) csv, with Column Import.

In the case of XML, I can't see why you would EVER want "Treat EOL as row terminator" = YES. This property was put in place for those situations (rare, but they occur) when a single messge actually contains a whole (or part of) a larger ASCII file. In those cases, you would get "n" rows from a single MQ Message.

...but that is notwhat you have here. Leave your XML message as a "whole" message. Pass it in its entirety to downstream stages for further processing. Leave that property as "No".

Ernie

eostic · Post by **eostic** » Tue Sep 15, 2009 3:32 pm

Further to that answer, the Job ought to look like:

MQConnector --- XMLOutput --(extract your return code and other such things) ---- other parts of the job, branching as necessary for FATAL, etc..

Ernie

pradkumar · Post by **pradkumar** » Tue Sep 15, 2009 10:42 pm

Thanks for your valuable suggestions. I will work out with the options you have suggested.

pradkumar · Post by **pradkumar** » Wed Oct 07, 2009 4:35 pm

Hi,

I have tried different options in the MQ Connector stage to pull the xml transaction as a single message by putting the End of the row terminator to No but unsuccesfull.So i have designed the job to pull the responses from MQ and dumps into a sequential file by putting the End of the line termniator to Yes and the dataypes to longvarchar(30000). When i see the file in the unix file system using vi i see there are null characters. So iam using translate command in the after job sub-routine to remove the nulls from the file and creates a new file. And i have created another job which reads the sequential file and parses into success/fatal. The jobs are running fine till now. But iam facing a new problem in the first job which reads from MQ and dumps into a sequential file.if i get more responses from the MQ the Raw file which is creating on the unix system(before removing the nulls) is exceeding the size limit and the job is aborting with the error outfile full. It seems the file is exceeding 2 gb space. i tried to remove the nulls while creating the raw file in the transformer by using trim(function,char(0).'A') by changing the datatypes to varchar (not using the translate command)but that didnt work.
Could you please suggest some ideas how to solve this issue and prevent the job abort with the error output file full.

How to remove the nulls while extracting from MQ and loading into sequential file.(Note : iam using end of the line terminator as Yes).

Thanks in Advance

eostic · Post by **eostic** » Wed Oct 07, 2009 10:55 pm

Something is still strange here.... Can you confirm two questions?

a) Is the message oroginally created and written to a queue by an MQ application, or by a Java JMS application?

b) Is there ONE xml document per MQ message, or many? (ie...is this a whole "file" of XML messages, or one single xml document in each mesage)?

Ernie

pradkumar · Post by **pradkumar** » Mon Oct 12, 2009 3:46 pm

Hi Ernie.

a) Its the MDM Java application

b) One XML document per MQ message

Please looking forward for your valuable suggestions.

Thanks in Advance

pradkumar · Post by **pradkumar** » Tue Oct 13, 2009 10:23 am

Hi Could any one suggest how to remove the nulls in my case..i have searched the forum but couldn't get successfull while extracting from MQ Connector stage..

eostic · Post by **eostic** » Tue Oct 13, 2009 1:21 pm

Are you getting a clean xml document now? You noted above that you have "one" xml document per message...but the symptoms that started this thread sound otherwise, or that maybe the xml documents are (potentially) huge and split across multiple messages, perhaps with things like correlation ids, etc.

Nulls shouldn't be a problem either, and the Connector can assist with that if you are using varchar.......where are you seeing nulls that are an issue?

Also, do you have access to MQ itself? Are you able to query the queue and see how many messages are in it? If there are 10 messages, then a quick job to read those messages should get 10 rows.....

Ernie