MQ Stage - Batch or Realtime

gsherry1 · Post by **gsherry1** » Tue Jul 04, 2006 10:49 am

Hello Forum,

I don't have so much experience with Real time ETL products. My understanding is that MQ Series is usually used in real time processing to consume messages produced a registered source.

So my questions are:

1. Does the MQ Stage perform any real time functionality. Or does it just read messages in batch when I explicitly schedule my job to run?

2. Is the MQ stage suppose/has to be run with RTI/Soa package?

3. If I do use MQ stage within the Soa package, do I get any benefit?

Your input is appreciated.

Greg

lstsaur · Post by **lstsaur** » Tue Jul 04, 2006 6:46 pm

Greg,
To answere your question #2 and #3:
A MQ Server is required to be installed on the same server that DataStage resides in order for MQ Stage to work. I don't think MQ Stage must be run with RTI/SOA. I have had a terrible experience with MQ Server because of all kinds of versions incompatible problems with either the JRE or the DataStage server. I think you will also get some problems with 7.5x2 EE. I was told by Ascential using lower version of MQ Server with DS 7.5 server edition as a workaround or wait for the Hawk release(late this year?).

If you are going into a lot of SOA stuff, I would suggest you to wait for the Hawk release. One benefit you gain right here is that you don't have to spending too much time to put MQ and/or RTI/SOA to work on your 7.5x2 EE server. I was told by Ascential that Hawk comes with a built-in MQ server and uses a WebSphere App Server (J2EE compliant server) as the mid-tier. With a J2EE compliant server available, then you know you can code/build/deploy the same things such as Web Service, JMS, EJB, XML, etc. that RTI/SOA, MQ Stage, Web Services PACK, Java PACK, etc. are trying to provide. That's another benefit that you will gain.

It's just my 2 cents.

gsherry1 · Post by **gsherry1** » Wed Jul 05, 2006 8:11 am

Thanks for your response lstsaur.

I have a couple other questions:

If RTI/Soa package is not needed, how does a job with the MQ stage run? Does it run in batch, or in real time. In other words, when I run my job, does it read all the messages available from the queue, and then terminate? Or is the job somehow running and listening for new messages on the queue?

- Greg

chulett · Post by **chulett** » Wed Jul 05, 2006 8:27 am

Without RTI/SOA, nothing is 'real time'.

chulett · Post by **chulett** » Wed Jul 05, 2006 10:03 am

Can't answer that as I have no experience with using that stage. It's not even one that's installed here.

We do have the SOA Edition and I don't recall any discussion of MQ Series in the documentation. Someone who has both, or more experience in general with that stage, needs to chime in.

trobinson · Post by **trobinson** » Wed Jul 05, 2006 11:17 am

I have used the MQSeries stage to run 24X 7-ish. We shut it down every Sunday night. You are very close in your last post. There are three ways to stop a DS Server job that takes queue messages as input.
1. By number of messages read
2. After a certain amount of inactivity.
3. With a Stop message.

We started the job from the command line and it would read real-time for 6 days. As messages were received into the queue, the running job would read and process them (Beware Transaction Control!). We would feed the queue with a specific stop message (via another DS job) and the MQ job would stop.

Our Vendor would input into a CICS system, XML messages were emitted and two seconds later the DS job would insert the data into our ODS Oracle database. Real-time.

chulett · Post by **chulett** » Wed Jul 05, 2006 11:26 am

Ah... interesting. Good to know! :D

ray.wurlod · Post by **ray.wurlod** » Thu Jul 06, 2006 2:10 am

Even real time is a continuously running job with a "listener" awaiting some input, unless the job startup time is acceptable (a small proportion of the overall execution time).

gsherry1 · Post by **gsherry1** » Thu Jul 06, 2006 11:13 am

trobinson wrote: We would feed the queue with a specific stop message (via another DS job) and the MQ job would stop.

Thanks.

I didn't think of using another DS job to send the message. I thought I was going to have to do that outside of DataStage.

- Greg