Reading Large XML from MQ and passing it to MQ Input Stage

[joffijoy]

Hi,

I have a requirement to read a very large XML file from MQ and pass it in to XML INPUT and then to flat file.

MQ------->XML INPUT ---> Transform ------> flat file.



The file I have now is 26.5 MB in size and has 26113609 characters. I could read the same XML as input from a sequntial file (Using XML file path option of XML input) and write to flat file.

I think the issue is in defining the field size of the XML file. When I define the XML as VARCHAR/LONGVARCHAR 99999999 it give me error "APT_CombinedOperatorController,0: Fatal Error: Fatal: Length of column and Display of column cannot be both 0." If I give it as VARCHAR 999999 it does not gives me eror and terminates successfully, but will not give me result since part of the XML will be truncated and hence may not be parse properly.

1. Could any one please suggect how do I handle this kind of large XMLs from MQ?. I have read that there are restriction imposed by DataStage when the XML is larger than 100 MB.


Sincerely,
Joffi

[eostic]

Hi Joffi...

A couple of things potentially going on here.... first, MQ imposes a limit of 100M on message size, so anything bigger than that is going to need segmentation or other techniques on the MQ side, even before we think about DS.....

....26M though, should be fine, but you'll need to increase the buffer size variable for EE... I don't have access to DS at the moment, but I believe its APT_DEFAULT_TRANSFER_BLOCKSIZE....it defaults to something small like 128k

I've read some very large XML from MQ this way...

Ernie