Hi,
I have requirement to design a datastage job to load POS transaction data in datawarehouse near real-time. Source system will send a message and my job should be able to read the message (xml) and load in respective tables.
Once data loaded in DW successfully then message can be deleted from the queue, in case of load failure message should be available to re-read.
At this point I am not sure what type to message application will be used.
My questions are:
1. Can I use WebSphere_MQ_Connector to read message when
messaging application is not IBM WebSphere MQ (MQ Series).
2. Do we need to install anything message related (Client etc.) in
datastage server.
3. Any suggetions and tips from your experience for similar
requirement.
I have been reading Dsxchange posting and other Pdfs, I found all are pointing to IBM WebSphere MQ (MQ Series) related.
I really appreciate your time and valuable inputs.
Thanks.
Near Real-Time
Moderators: chulett, rschirm, roy
If the queue technology is not WebSphere MQ, then you will need to use JMS via your own class with JavaPack. ...not difficult, but you will need to have some java expertise, especially that knows the ins and outs of JMS because of your interest in transaction logic....
With the MQ Connector, you can commit the data to a work-queue as soon as you read it -- thereby ensuring that if the Job dies down stream, or you have multiple jobs, or you wait until the next day to load to your target rdbms, you won't "lose" the message..... this same logic could be written into your JMS code also, depending on what messaging system you have and what they provide for queue-to-queue transaction handling (often in the topic of "XA" transactions).
With the MQ Connector and the DTS (Distributed Transaction Stage), you could also do this directly if your source is the right release of MQ and is configured for use with Oracle and or DB2 and MQ's support for XA with these rdbms'....... that's a lot of "if's" and pre-reqs, but the infrastructure exists if the pieces are all correct.
Still...the JMS scenario is valid too --- and there are lots of threads here about it....you can always just park the messages into a flat file somewhere so that you don't lose them. The transactional piece here is the difficult challenge. The truly "integrated" solutions work, but are highly reliant on releases and correct configuration.....reconciliation scenarios work well too (where you "browse" for messages, load them into your target, and then go back later and clean up messages that you can confirm made it successfully into the target), and are often simpler to code and conceptualize.
Ernie
With the MQ Connector, you can commit the data to a work-queue as soon as you read it -- thereby ensuring that if the Job dies down stream, or you have multiple jobs, or you wait until the next day to load to your target rdbms, you won't "lose" the message..... this same logic could be written into your JMS code also, depending on what messaging system you have and what they provide for queue-to-queue transaction handling (often in the topic of "XA" transactions).
With the MQ Connector and the DTS (Distributed Transaction Stage), you could also do this directly if your source is the right release of MQ and is configured for use with Oracle and or DB2 and MQ's support for XA with these rdbms'....... that's a lot of "if's" and pre-reqs, but the infrastructure exists if the pieces are all correct.
Still...the JMS scenario is valid too --- and there are lots of threads here about it....you can always just park the messages into a flat file somewhere so that you don't lose them. The transactional piece here is the difficult challenge. The truly "integrated" solutions work, but are highly reliant on releases and correct configuration.....reconciliation scenarios work well too (where you "browse" for messages, load them into your target, and then go back later and clean up messages that you can confirm made it successfully into the target), and are often simpler to code and conceptualize.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>