MQ and Server Shared container

dodda · Post by **dodda** » Mon Aug 13, 2012 2:19 pm

I have a job design like

MQ--Transformer--Server Shared Container(with Folder Stage in it).
I am reading around 4 messages from MQ at a time and creating each XML for each message and storing in a FOlder stage. each xml is of around 1.2 MB size. I increade the APT_DEFAULT_TRASPORT_BLOCK_SIZE to higher number as well.
I am getting the below error when i am reading the file from MQ and storing it in a FOlder on the server.

SCLoadXML,0: row is to big to fit in shared memory buffer

Can some one help me on this

ArndW · Post by **ArndW** » Tue Aug 14, 2012 1:03 am

It has been 2 or more years since I did a DataStage with MQ project, but seem to recall having hit the same problem as you are seeing. What OS are you working on? We were on AIX and I believe we did something AIX-specific. There was also an interrelationship between the default and max block sizes in the DSParams as well.

dsoupaty · Post by **dsoupaty** » Tue Aug 14, 2012 12:27 pm

Hi
I am using AIX 6.1. I also played aroung with
APT_BUFFER_MAXIMUM_MEMORY
APT_BUFFER_DISK_WRITE_INCREMENTAPT_BUFFER_FREE_RUN

environemntal vairables as well. All the APT variables i used i changed them at the job level.
but no use.

Can you tell me the relation ship you are talking about.

Thanks

ArunaDas_Maharana · Post by **ArunaDas_Maharana** » Tue Aug 14, 2012 5:05 pm

what is transformer used for? you are reading XML from MQ and creating xml in transformer?

dsoupaty · Post by **dsoupaty** » Tue Aug 14, 2012 8:15 pm

I am using transformer for concatinating messageputime data element and message date data element from mq and generating a disint file name so that each message goes to the single distinct file in the folder stage.

eostic · Post by **eostic** » Tue Aug 14, 2012 8:17 pm

Skip the EE Job altogether. Real time --- do you "really" need the deep parallelism that you can accomplish with EE? Sometimes you do...but often Server will be just fine in a real time paradigm -- maybe even better. Unless this is quality Stage, or you can really justify the need for massive parallelism, drop EE as the host and use a Server Job all the way...the barrier to shipping large rows across the Container "boundary" go away --- and Server is better at handling huge, totally undefined varchar strings anyway.....

Ernie

dsoupaty · Post by **dsoupaty** » Wed Aug 15, 2012 8:17 am

Thanks ernie. Will try that and let you know

ArunaDas_Maharana · Post by **ArunaDas_Maharana** » Wed Aug 15, 2012 11:54 am

what is the datatype you were storing xml and passing through transformer --> folder stage through shared container?