MQ Output Connector Padding output with nulls

BradMiller · Post by **BradMiller** » Tue Apr 21, 2009 2:08 pm

I have a job that reads data from a DataSet, creates an XML message using the XML Output stage and then places the XML message onto an MQ Queue using the MQ Connector. The output XML message could be quite large because I am aggregating the message by one of the columns. I set the output column out of the XML Output stage to be varchar(100000) to allow for our largest messages. Everything works, however we discovered that our disk space on the MQ Server was filling up faster than expected when loading these messages onto the queue. One of our MQ administrators checked and has told me that all the messages that I am putting onto the queue are 100,000 bytes long. All messages are padded to the max size of the varchar with nulls.

I checked the message prior to loading it onto the queue by writing it to a sequential file and there are no nulls. The message ends after the last XML tag.

Has anybody else seen XML messages being padded with nulls? Is there a setting I am missing somewhere to keep the output message at its actual length? Is this possibly an issue with MQ and not DataStage?

Thanks!

eostic · Post by **eostic** » Tue Apr 21, 2009 4:03 pm

Hi Brad...

I'll have to take another look at the Connector (the older MQStage has provisions for this, so it doesn't pad -- the Connector may also....look very closely at the myriad of properties)....but it also may be EE itself. I had a similar issue one time with a Web Service, and corrected it by setting APT_STRING_PADCHAR to a single blank (just = and then the space bar, with no quotes or anything else). Then the downstream stages that compress blanks were able to correctly, but the long string of nulls were problematic.

...of course, try the MQStage also, if you are able (there are some restrictions there, based on which MQ API you are using and your choices at install time).

Ernie

BradMiller · Post by **BradMiller** » Tue Apr 21, 2009 6:34 pm

Thanks for your reply Ernie,

I added the APT_STRING_PADCHAR and set it to a default of blank. Initial testing is looking good. We have sent through one message and it wasn't padded with nulls. We will be doing futher testing tomorrow at which time I will update this post with the results.

BradMiller · Post by **BradMiller** » Wed Apr 22, 2009 2:30 pm

Ernie,

Testing proved that adding the APT_STRING_PADCHAR environmental variable and setting it to a space fixed our problem with nulls being appended to the end of our xml messages.

Thank you for your assistance...

Kryt0n · Post by **Kryt0n** » Wed Apr 22, 2009 5:43 pm

Does it not mean you are changing the nulls with spaces and still have 100000 byte long messages? Thought one of your issues is that your filespace is filling up quicker than you would like (or are you trimming as well?)

With this resolving the issue it would imply the column was declared as char somewhere along the line (we never have problems with varchars being padded unless we managed to declare the column as a char prior to being a varchar)

eostic · Post by **eostic** » Wed Apr 22, 2009 10:45 pm

The "hope" (looking forward to Brad confirming) is that you may be correct --- it may end up padding the column downstream, but then kick in default behavior in the Connector to be smart about targeting and trim everything before actually writing out the message. Only a len() prior to the MQ Connector will tell, along with a review of the actual messages as they sit inside the Q (to see if they are random or shorter lengths, as they should be).

Ernie

BradMiller · Post by **BradMiller** » Thu Apr 23, 2009 10:55 am

The message was never set to char. It was always a varchar. I am not sure why we were getting nulls added to the end of every message, but as soon as I set the Apt_String_Padchar variable to space, it fixed the problem. I am assuming that the MQ Connector will strip padded spaces but not nulls. How they get padded onto the message to begin with is a mystery. I replaced the MQ connector with a sequential file and there were no nulls. So something was happening in the MQ connector.

Thanks again.