How can I make a job "always on"?

ramsubbiah · Post by **ramsubbiah** » Thu Feb 03, 2011 6:48 am

Hi Everyone,

i have a requirement to develop a job which will fetch the files from source Queue whenever file arrives , so i have created the following job design for testing,

MQ----->TRANSFORMER------->Dataset

in the MQ stage i have given the following values for the parameters

Wait time = -1
message quantity 1
end of data message type = 9999989
Record Count = 0
End Of Wave = After
End Of Data = Yes

The job is successfully fetching up to four files , after that its getting finished, later i observed that my configuration file is having four nodes,
if i change to 2 nodes then its fetching only up to two files after its getting finished.

Note: After running the job i am placing files one by one.

please guide me if any other setting needs to be done for making the job always running mode to fetch the files when ever that arrives!

Thanks in advance.
Ram

eostic · Post by **eostic** » Thu Feb 03, 2011 7:17 am

Not sure if there is an issue? .....sounds like it is working great... ...but of course, for a real time job like this, you should have a single node configuration. Then, if you browse, you will get only one copy of the message, not multiples. There are only very rare cases when a real time job needs multiple nodes. The single node is also vastly simpler to debug and work with during your initial testing.

Also, during testing, I like to use a wait of 10 seconds, and a message count of 10..... in case I make any mistakes. Once I have the logic tested and working, move to -1 and a message type stop mechanism like you have now.

Ernie

JRodriguez · Post by **JRodriguez** » Thu Feb 03, 2011 8:27 am

Hello Ramsubbiah,

Follow Ernie suggestions ... and please try increasing the number of messages that you are expecting to unlimited

Wait time = -1
message quantity -1
end of data message type = 9999989
Record Count = 0
End Of Wave = After
End Of Data = Yes

Regards

ramsubbiah · Post by **ramsubbiah** » Thu Feb 03, 2011 11:22 am

Hi Ernie/julio,

Thanks for your information, i will re test the job based on your findings and get back to you shortly,

Thanks,
Ram

ray.wurlod · Post by **ray.wurlod** » Thu Feb 03, 2011 3:13 pm

Is this an Information Analyzer question?

ramsubbiah · Post by **ramsubbiah** » Thu Feb 03, 2011 10:20 pm

eostic wrote:Not sure if there is an issue? .....sounds like it is working great... ...but of course, for a real time job like this, you should have a single node configuration. Then, if you browse, you will get only one copy of the message, not multiples. There are only very rare cases when a real time job needs multiple nodes. The single node is also vastly simpler to debug and work with during your initial testing.

Also, during testing, I like to use a wait of 10 seconds, and a message count of 10..... in case I make any mistakes. Once I have the logic tested and working, move to -1 and a message type stop mechanism like you have now.

Ernie

Hi Ernie,

No luck, if i am running the job with single node configuration its fetching the first file which is arrives in to queue, after that the job is getting finished,

later i have given wait of 120 seconds and message count is 2 in this case the job is waiting for two files to arrive once they arrive its fetching both two files and the job is getting finished status,

but my requirement is whenever the file arrives it should fetch the file and load in to target and again it should wait for next file, the job should not get finished status, it should be always running,

Additional information:
i am using datastage 8.0.1 version

please correct me if i am in a wrong way.

ramsubbiah · Post by **ramsubbiah** » Thu Feb 03, 2011 10:22 pm

ray.wurlod wrote:Is this an Information Analyzer question?
...

Hi Ray,

sorry wrongly i have created this topic under this forum, please tell me how to move this topic to datastage parallel forum

Thanks,
Ram

ramsubbiah · Post by **ramsubbiah** » Thu Feb 03, 2011 10:26 pm

JRodriguez wrote:Hello Ramsubbiah,

Follow Ernie suggestions ... and please try increasing the number of messages that you are expecting to unlimited

Wait time = -1
message quantity -1
end of data message type = 9999989
Record Count = 0
End Of Wave = After
End Of Data = Yes

Regards

Hi Julio,

if i change the message quantity to -1 , then the job is not fetching any files, though i keep file in the queue , once i post a file in to queue suddenly its disappearing , in the job also its not fetching, am wondering where the file is gone!
Thanks,
Ram

suman27 · Post by **suman27** » Fri Feb 04, 2011 5:00 pm

Hi ramsubbiah,

We had a similar requirement. If you tried to run the mq stage on multinode config file it will duplicate messages times the number of nodes. So we developed a saparate job which read from MQ and runs on single config file. And the second job will process data with multinode config file.

You can disign a routene/script which will trigger this job in loop continually checking the status of the job.

Regards,
Suman.