Hi,
I am using an MQ Connector stage(Datastage 8.5) for extracting the XML messages from 4 Queue Managers (Websphere Message Broker Queues).
I am facing an issue with the number of messages extracted by Datastage from the queues.For example, if the queue has some 5000 Xmls (There are 4 Queue Managers which points the same queue), Datastage does not extract all the messages.(checked both in Monitor and Performance Stats). Also there is one more process to ensure all messages are extracted from the queues.It clearly shows that Datastage doesnot extract all the messages from the queue.
There is some count mismatch between the WMB and ETL.
I could not find any information in director regarding the message drop.
The message read mode is set as Keep and I delete the messages from the queue , once i am completing the message processing.
During deletion, all the messages in the queue are getting deleted correctly.
I checked in the queue, whether all the messages are in committed mode.All the messages are in committed mode only.
So, where am i missing the messages.
Is there any option , that i need to enable in MQ Connector to stop this ?
Could someone help me fixing this issue.
Message getting dropped while extracting using MQ Connector
Moderators: chulett, rschirm, roy
-
Nithyasankari
- Participant
- Posts: 3
- Joined: Sat Jul 26, 2014 3:57 am
- Location: India
Message getting dropped while extracting using MQ Connector
Datastage Developer.
Some thoughts...
...do you have multiple Jobs reading from all the different queue managers?
...are there other readers (again, multiple jobs, or other apps) reading also from the same queues?
These are just guesses, but when you use "keep", you are placing responsibility for duplicate reading and managing the queues in your own hands. Consider placing that in MQ's hands. Use "delete" in some tests instead of "keep" and DS won't "browse" the queues, it will take messages and own them directly, thus preventing anyone else from reading "that" particular message. If that clears things up for you, then you can start looking at one of the more advanced capabilities of the Connector, which is to do a "delete under transaction" and move the message into a work queue (so that there is zero chance of dropping it entirely).
DON'T just change to delete under transaction.....go slow and learn how the behavior works before implementing it....and learn it with a SINGLE reader before you move on to multiple readers. This can be a very complex environment, but work-able.
Your issue may be something different entirely, but I've seen lots of situations with conflicts over MQ multiple readers.
Ernie
...do you have multiple Jobs reading from all the different queue managers?
...are there other readers (again, multiple jobs, or other apps) reading also from the same queues?
These are just guesses, but when you use "keep", you are placing responsibility for duplicate reading and managing the queues in your own hands. Consider placing that in MQ's hands. Use "delete" in some tests instead of "keep" and DS won't "browse" the queues, it will take messages and own them directly, thus preventing anyone else from reading "that" particular message. If that clears things up for you, then you can start looking at one of the more advanced capabilities of the Connector, which is to do a "delete under transaction" and move the message into a work queue (so that there is zero chance of dropping it entirely).
DON'T just change to delete under transaction.....go slow and learn how the behavior works before implementing it....and learn it with a SINGLE reader before you move on to multiple readers. This can be a very complex environment, but work-able.
Your issue may be something different entirely, but I've seen lots of situations with conflicts over MQ multiple readers.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
-
Nithyasankari
- Participant
- Posts: 3
- Joined: Sat Jul 26, 2014 3:57 am
- Location: India
-
Nithyasankari
- Participant
- Posts: 3
- Joined: Sat Jul 26, 2014 3:57 am
- Location: India
Can you please explain little more about your design..
You mentioned you are running the job in Keep mode and deleting the messages after you job read the messages.
Are all the processes putting the messages in MQ are stopped when you are running your DataStage?
If not, you might see more messages than your job processed as the process which puts messages might be running.
Where exactly you see the messages are dropped? in MQ connector or any other XML input/Xfm..?
Other thing you can do is, find out exactly what message is not processed by DataStage and see if it is any different than others.. Since you have mentioned that there are different QManagers (may be different processes) putting the messages, you never know there may be some difference in the way they are putting the message..
You mentioned you are running the job in Keep mode and deleting the messages after you job read the messages.
Are all the processes putting the messages in MQ are stopped when you are running your DataStage?
If not, you might see more messages than your job processed as the process which puts messages might be running.
Where exactly you see the messages are dropped? in MQ connector or any other XML input/Xfm..?
Other thing you can do is, find out exactly what message is not processed by DataStage and see if it is any different than others.. Since you have mentioned that there are different QManagers (may be different processes) putting the messages, you never know there may be some difference in the way they are putting the message..