Hello All:
I have designed a job that reads messages form the MQ Message queue and then seperates the header and data. The data which is delimited is read and parsed into a variable. That variable in which the data is parsed is broken into number of columns using row splitter then dumped into transformer since the source columns are lesser than the target columns the extra columns are set to @NULL and then all is dumped into the sequential (target) file.
Not using destructive read. Set to read.
The setting for MQ Pluggin is set as follows:
WAIT TIME = 0 (specifies no wait time to read the message)
MESSAGE LIMIT = 244,014
END OF DATA MESSAGE = 0
It reads the data in 58 minutes. I have checked my design and have no unnessary processes or files that could increase the read/write time.
But is there any way of increasing the performace and read the data in lesser time. Is there a benchmark that tells me its a good read and write time (58 Minutes)
Is there a better way...? Would appreciate a response at the earliest convenience.
Thanks in advance.
MJ
MQ Series Pluggin
Moderators: chulett, rschirm, roy
-
throbinson
- Charter Member
- Posts: 299
- Joined: Wed Nov 13, 2002 5:38 pm
- Location: USA
1. Do a MQbrowse from a DOS prompt to a flat file. Record the time.
2. Do a ETL job from MQ straight to a flat file. Record the time.
The differences in these two times and the time of your job will tell you how long it takes to read from MQ, to read and write from MQ in DS and how long to process rows in your job design.
If you job design is slowing things down then perhaps multi-instances reading from the same queue would help. You can constrain the writes on MOD @INROWNUM at the very least to build multiple output files that can then be cat'ed together.
If the MQ plug-in is the bottelneck then I have no comment.
2. Do a ETL job from MQ straight to a flat file. Record the time.
The differences in these two times and the time of your job will tell you how long it takes to read from MQ, to read and write from MQ in DS and how long to process rows in your job design.
If you job design is slowing things down then perhaps multi-instances reading from the same queue would help. You can constrain the writes on MOD @INROWNUM at the very least to build multiple output files that can then be cat'ed together.
If the MQ plug-in is the bottelneck then I have no comment.
Hi throbinson:
Thanks for your reply.
However, my job encounters no bottlenecks or any such thing. Its working fine.
What I would like to know that in MQ stage the setting
Wait time = 0 (not to wait just read)
is the best or
Wait time = -1 (specifies indefinite wait period (Default)) is a better option
to improve read time performance. Or is there any other setting much more efficient.
I ran the job with Wait time = 0 and Wait time = -1 and did not find any difference in time. Its just that if there are better way then I would like to know before I call this job done.
Thanks.
MJ
Thanks for your reply.
However, my job encounters no bottlenecks or any such thing. Its working fine.
What I would like to know that in MQ stage the setting
Wait time = 0 (not to wait just read)
is the best or
Wait time = -1 (specifies indefinite wait period (Default)) is a better option
to improve read time performance. Or is there any other setting much more efficient.
I ran the job with Wait time = 0 and Wait time = -1 and did not find any difference in time. Its just that if there are better way then I would like to know before I call this job done.
Thanks.
MJ