Start Loop - infinite loop

[wjfitzgerald]

Hi,

I am new to datastage so hopefully this might be a very simple fix.

I am looking to create a sequence control loop that will execute a number of jobs continuously. The break of the loop will be controlled from a parameter in an ini file.

I can break the loop based on the parameter.

My problem is that the startloop is finite, limited to the number of iterations specified or number of entries on the list.

Perhaps startloop is not the right stage to use?
If anyone one has any suggestions, I would really appreciate it.

Regards,
John Fitz

[chulett]

Welcome. :D

Define 'continuous' and 'infinite' in your case. The loop counter can be a rather large number (9 9s from what I recall) - are you saying that you would need to loop more than that number of times in a continous run?

Perhaps an idea of what kind of problem you are trying to solve is in order? I've built jobs that run every X over the course of the day, but I wouldn't run anything 'continuously' day over day unless it was an RTI/SOA job. Mine take a break at night and are relaunched the next morning, and poll for interrupt files in case they need to be gracefully shutdown. Curious what you have in mind.

[wjfitzgerald]

Craig,

Thanks for coming back to me, and the welcome.

The idea here is we are running a 24/7 monitoring process. to simplify the model let me describe the jobs as follows:
1. extract job - creating a simple flat file - extract details of change since last extract
2. processing job(s) - file created in the last job is fed as input to a processing job which creates an XML message and places it on a websphere queue.

The control job needs to achieve the following:
1. read a parameter flat file (contains details of MQ settings, and parameter which tells the job to finish)
2. start first job
3. on completion start second job
4. interrogate the "finish" indicator, if time to finish end the control job
5. otherwise return to 1 above

The only time that the job will be stopped would be to allow for code releases, or system maintenance of some sort. other than that the process needs to run continuously. My understanding of the STARTLOOP stage is that eventually (even setting the terminating point to all nines, and the increment value to .0001) the loop will end.

I appreciate I could build in an email message or something to inform Operations to restart but this intervention seems unnecessary.

Just purely from a theoretical point of view it seems strange to me that the loops break conditions can only be the counter or the list contents. surely a program set break indicator is not beyond the bounds or possibility?

Anyway, as I said I am new to this, so maybe I am just using the wrong stage.

I really appreciate you taking the time to have a look at this.

Regards,
John Fitz

[ArndW]

The loop mechanism doesn't cater for infinite loops. But I've written 'never ending' loops using the counted method and computed that this job would run for several years taking into account the typical loop execution times; so for all intents and purposes I felt confidant that the loop would never complete.
If you wish, you could add one more level of checking - write a job sequence that is triggered hourly; this checks to see if the other sequence is still running and if it is not running it re-starts it.

[wjfitzgerald]

thanks again, there is definitely something in idea of a second job restarting the first.

Thanks,

John Fitz

[wjfitzgerald]

thanks again, there is definitely something in idea of a second job restarting the first.

Thanks,

John Fitz

[chulett]

No, not the wrong stage per se, it's just that it was built with a fixed end in mind. What you'd really want to build is an RTI job, but (I assume) with that not being a possibility...

What you could do is hand code something similar to the Loop stages in a batch job but without the hard end point. However, not something undertaken lightly without some prior experience in that area or something to copy as a starting point. Technically, you could build your Sequence job and (from the Job Control tab) snag the generated source and attempt to modify it to take the loop counter limit out of the picture.

In either case, anything can be a 'break' condition in a Loop, the counter and list of values just define the maximum iteration limit. You could always link to a Terminator inside the loop based on conditions, or link outside the loop past the End Loop stage to break early.

Arnd given you some ideas on restarting it automatically. I still prefer the 'stop once a day and restart' design, even if the stop is only for a minute or two, it gives the logs a chance to clear on a regular basis. And something like that can be easily setup with an Enterprise scheduler or even cron. I've built time based exit logic into loops that I know will never hit their max loop count but rather exit once a parameterized time boundary is crossed.

Have fun! :D

[wjfitzgerald]

probably a silly question, but is there any way to identify the variable that is holding the stage counter and just reset that as part of the loop??

[chulett]

StartLoopStageName.$Counter

Now, as to whether you can reset it or not inside the loop? [shrug] Never tried but go ahead and give it a shot.

[ray.wurlod]

Unless you take the code generated by the job sequence as a prototype and load that into a server job, there is no activity that can alter the value assigned to the StartLoop.$Counter activity variable other than the Start Loop activity to which it belongs.

[wjfitzgerald]

ray,

that saves me a bit of time anyway, thanks?
any thoughts on an alternative approach?

thanks
john fitz

[chulett]

Don't build an 'infinite' loop?