Reading WebSphere MQ messages with DataStage

suryadev · Post by **suryadev** » Mon Aug 12, 2013 4:12 pm

Hello,

Can anyone suggest me how to read IBM MQ msgs from datastage?

As of now a sample file was given to see if the data appeared can be transformed in datastage. This sample file when opened by Note++ is opened in ASCII which is NUL,DC1...etc but when read the same file from Datastage now the data looks like square boxes , @,y with .. on top something like that......

Please suggest how to make the data look like normal data...I mean settings in sequential file...

Thanks very much!

ray.wurlod · Post by **ray.wurlod** » Mon Aug 12, 2013 4:51 pm

Why are you using Sequential File stage? Or, to put it another way, why aren't you using the MQ Connector?

suryadev · Post by **suryadev** » Tue Aug 13, 2013 9:20 am

Thanks very much!

Will use the websphere MQ connector , it is in real time stages.
Does it need to be deployed through ISD or can be run just as a batch job?

This is just a one time requirement so for sample file used sequential file. As given below how can I change the data from this sample file be changed to normal view from the square boxes , @,y with .. on top symbols.....once it can be transformed I will use the MQ connector and let the user know...

eostic · Post by **eostic** » Tue Aug 13, 2013 1:19 pm

Do a lot of searching and reading of this forum and of the formal documentation...MQ is not something you enter into lightly....lots of things to consider and potentially things to install and configure......on the DataStage side and on the mq side, depending on your application variables for performance, security, ensured delivery, etc.

Your viewing issue could be lots of things. Start by not trying to view it in DataStage ----- confirm that the data is what it should be, binary or otherwise.......if it's truly wrong, let us know.

Ernie

suryadev · Post by **suryadev** » Tue Aug 13, 2013 2:14 pm

This is just a one time transformation, so its better I will ask them to send a file which has data in it.

Coming to the sample file which is a QAW file, with out viewing it in Data Stage I opened it in Notepad++ and it looked like this

thresholdxp?@NUL NUL NUL NUL CANw

which I think is ASCII format
Now I saved the above line in Notepad and now when I open it looks like this

thresholdxp?@ w

Which I think is Unicode format

Now as this is file , I used sequential file as source
Please suggest me what should be given at the NLS settings in stage properties of sequential file so that the data is changed to regular format , I mean with out any special characters

Thank you!

eostic · Post by **eostic** » Tue Aug 13, 2013 2:26 pm

It's not necessarily that simple. Those "special" characters might be critical binary values. You need to speak with whoever authored that file and understand what is truly stored in each of its bytes.

Ernie

suryadev · Post by **suryadev** » Tue Aug 13, 2013 3:59 pm

So if we change the format will that change the real data?

I did ask them and they said change the format and transform the data.

Please let me know how to read this format from DS so that it looks like normal UTF-8.

Thanks again

ray.wurlod · Post by **ray.wurlod** » Tue Aug 13, 2013 4:36 pm

Read the file using a hex dump utility, such as the od -xc command, to learn what the characters are. I suspect that you have one or two binary representations of numbers in there. As Ernie noted, you need to speak with whoever authored that file and understand what is truly stored in each of its bytes.

suryadev · Post by **suryadev** » Wed Aug 14, 2013 8:10 pm

Thanks!

Do I need to give this command in the before job routine of the job along with the path or where do I need to give?

I did give hexdump filename.txt in the directory of the file location and I see the data in all numbers....

ray.wurlod · Post by **ray.wurlod** » Thu Aug 15, 2013 4:17 am

Now find somebody who can explain to you what those numbers mean. For example, the pair 41 (in hexadecimal, equates to 65 in decimal) represents the ASCII code for the character 'A'. Some non-printing characters have codes in the range 00 through 1F.

eostic · Post by **eostic** » Wed Aug 21, 2013 8:41 pm

It's possible that it is MQ header information. If you are receiving a file and not actually reading from MQ, see if the application that is reading MQ and writing the file can just capture the "message content" and leave out the header, so that you don't have to deal with it. I can't remember if it is definitely "fixed" but that would be something to look for. Perhaps, if the header is always (for example) the first 160 bytes, you can just describe them as a big 160 byte column and then ignore them downstream. See if these bytes are consistent for every single record that you are reading.

Of course, if you "need" something in that header (correlation id's, message ID's, etc.), then you'll have to learn more about the binary values, what they mean, and how to read them.

Ernie

folksam · Post by **folksam** » Fri Sep 27, 2013 3:38 am

If i got it right, a similar situation at our site was solved by adding a few WSMQ columns(WSMQ.PUTTIME) to the payload column.
See MQ connector stage -> columns tab ->"data elements" .

These new columns took all unnecessary MQ property fileds out of payload making it free from the seemingly strange characters. In our case we were left with just xlm text in the payload column.

Not sure if it is correct to interpret this as a solution. Hope experts on this forum can shed some light on this. I myself need some clarity on this