DS Stability over Longer Run Times

banactp · Post by **banactp** » Thu Mar 03, 2005 1:46 pm

Hello,

Has anyone had any experience running DSEE/PX jobs or sequences over longer periods of time (several days)?

If so, have you had any bad experiences with the jobs hanging, memory overflows, etc.?

In my specific case, I'm not intending to do anything complicated during most of the run - but I do want to "sleep", mid-sequence, for up to 72 hours before resuming processing. Can anyone please comment on whether or not that would be a good idea?

TIA,

tpb
DataStage newbie

ray.wurlod · Post by **ray.wurlod** » Thu Mar 03, 2005 2:23 pm

There's no reason in theory that you shouldn't do what you want, but you're taking a risk that the system might be stopped during your long sleep. Or that an administrator may clean up apparently idle processes.

I'd advise researching a different approach, perhaps finish gracefully now and schedule another job to start 72 hours later. Scheduled jobs are queued, and survive system shutdowns and the like.

T42 · Post by **T42** » Fri Mar 04, 2005 11:30 am

DataStage (especially with EE) are VERY solid with their memory usage. Memory leaks are VERY rare out in the field, and jobs could practically run for years if you let it.

However, do pay attention to what Ray said. Do you REALLY want to let DataStage, a pure ETL box, to handle scheduling duties? Why not invest into a good third party scheduler and let that tool do the job? Ascential do recommend that if you want an enterprise scheduler.

banactp · Post by **banactp** » Fri Mar 04, 2005 11:42 am

Oh,

We have a good 3rd party scheduler. It's just that my group doesn't "own" it and we have to go through someone else to *do anything* with it.

Thanks,

tpb
DataStage newbie

dsxuserrio · Post by **dsxuserrio** » Fri Mar 04, 2005 4:37 pm

Another option is to use your 3rd party scheduler to schedule the job regularly. Daily or hourly. To control from your end you can have a file . IF the file is present the job will not start. Once you remove the file, the job will run regularly. I am assuming you are starting you sequencer using a unix shell script.

This way you never have to ask the scheduling team to stop or start your jobs.

banactp · Post by **banactp** » Mon Mar 07, 2005 12:28 pm

Another option is to use your 3rd party scheduler to schedule the job regularly. Daily or hourly. To control from your end you can have a file . IF the file is present the job will not start. Once you remove the file, the job will run regularly. I am assuming you are starting you sequencer using a unix shell script.

This way you never have to ask the scheduling team to stop or start your jobs.

Nice try. But then I'd have to go through the file control team!

After further review, I've decided I'm just going to have to create 3 job sequences that can be run by the 3rd-party scheduler; one for M-Th that runs entirely to completion, one for Friday nights that runs part way, and another for Sunday PM that finishes up where Friday night left off.

Oh well, just 200% more things to keep track of.

tpb
DataStage newbie