DSXchange

caterin_huang wrote:

chulett wrote:Turn off the Compiler options you've turned on, that gives you explicit control over things. Then use the appropriate triggers to keep aborts from stopping things (Unconditional, for example) and perhaps a Terminator when it actually is time to shut things down.

As noted, the problem will be restarts - you will need to explicitly control where to join the job stream depending on where it needs to restart.

Hmm, do you mean I should uncheck the "Automatically handle activities that fail" box ?
Ive tried it, but when a job is aborted, the sequence is reported as finish succesfully. and when I klik start at the sequence after resetting the aborted job,I can't run the job from certain checkpoint. Because It will rerun from the beginning again.

Thanks,
Caterin

can datastage run a finished sequence from certain checkpoint only? I see that it is only possible for datastage to run from the checkpoint if the sequence is aborted. is it true?

Thanks,
Caterin

That is true. But, with explicit triggers and record-keeping, you can design sequences do to whatever you like.

ray.wurlod wrote:That is true. But, with explicit triggers and record-keeping, you can design sequences do to whatever you like. ...

hmm,if a single job activity is aborted at a sequence, and I want to make a job sequence finish with aborted status after all other independent jobs finished, then I must use the help of terminator stage right?
But terminator stage also stop all jobs at the time when the terminator is triggered whether all other independent job is still running and must be finished. So how to design the sequence as my objective before? Thanks

Don't use a Terminator, for the reasons you've stated. Record-keeping is the clue here - the sequence must record the success or otherwise of each activity and make decisions on restart about which activities to skip. You have to build the logic.

caterin_huang wrote:So how to design the sequence as my objective before?

As already noted. Your requirements preclude the use of the automatic checkpointing, hence the suggestion to disable those compiler options. You need to control the job flow using triggers, for example if you don't want an aborted job to stop the flow, use an Unconditional trigger. If you want to conditionally abort the Sequence itself based on a previous job's status once everything finishes, do so after the last of the Job Activity stages with a Terminator or perhaps by calling (from memory) the UtilityAbortToLog function. You'll need to write something to do those checks, however, and use the result to decide if things need to be 'terminated' or not.

chulett wrote:

caterin_huang wrote:So how to design the sequence as my objective before?

As already noted. Your requirements preclude the use of the automatic checkpointing, hence the suggestion to disable those compiler options. You need to control the job flow using triggers, for example if you don't want an aborted job to stop the flow, use an Unconditional trigger. If you want to conditionally abort the Sequence itself based on a previous job's status once everything finishes, do so after the last of the Job Activity stages with a Terminator or perhaps by calling (from memory) the UtilityAbortToLog function. You'll need to write something to do those checks, however, and use the result to decide if things need to be 'terminated' or not.

hmm, it means that at the end of the job sequence, I must check each job status whether to make the job sequence finished with aborted status. is it right?

Thanks,
Caterin

Ultimately, yes, but you will be gathering this information while the sequence is running and making decisions then about which parts of the sequence the process may need to abandon, as well as keeping records of everything that is happening.

But don't think you still need to Abort the Sequence to trigger the checkpoint restarting you no longer have. Abort it for other reasons, perhaps, but not that one.

chulett wrote:But don't think you still need to Abort the Sequence to trigger the checkpoint restarting you no longer have. Abort it for other reasons, perhaps, but not that one.

hmm, does it mean that there is another way to trigger the checkpoint except by making a sequence finished with abort status?

Please understand - you cannot make the built-in automatic check-pointing work the way you 'need' it to work, so you should put that from your mind. For the behaviour you say you want, you'll need to (in essence) build your own check-pointing system, one which may (or more likely may not) require the Sequence job to abort to function properly. It's all up to you.

If you can forego the 'automatic' restart part of the requirement, just getting things to run in the fashion you're saying should be pretty straight-forward and can be handled via the proper triggers, it seems to me.

chulett wrote:Please understand - you cannot make the built-in automatic check-pointing work the way you 'need' it to work, so you should put that from your mind. For the behaviour you say you want, you'll need to (in essence) build your own check-pointing system, one which may (or more likely may not) require the Sequence job to abort to function properly. It's all up to you.

If you can forego the 'automatic' restart part of the requirement, just getting things to run in the fashion you're saying should be pretty straight-forward and can be handled via the proper triggers, it seems to me.

hmm,how to make my own check-pointing system? Could you please give me an example? Thanks

Best example would be in any existing checkpoint enabled job. Check the Job Control tab under Job Properties, the fully generated code will be there including all of the checkpointing logic.

Basically, you are leaving "bread crumbs" to show where you've been and what the result of each step was. Could be in a sequential file, hashed file, whatever you are comfortable with. Then check it when you restart and then perhaps use Sequencers set to 'Any' as entry points into the stream just before each restart point. That or write your own routine version of the Job Activity stage that checks your bread crumbs to know if it should run or not and then also handles the crumbs post run. Me, I'd go the latter route as it would be cleaner, not to mention easier to setup and maintain. IMHO.

chulett wrote:Best example would be in any existing checkpoint enabled job. Check the Job Control tab under Job Properties, the fully generated code will be there including all of the checkpointing logic.

Basically, you are leaving "bread crumbs" to show where you've been and what the result of each step was. Could be in a sequential file, hashed file, whatever you are comfortable with. Then check it when you restart and then perhaps use Sequencers set to 'Any' as entry points into the stream just before each restart point. That or write your own routine version of the Job Activity stage that checks your bread crumbs to know if it should run or not and then also handles the crumbs post run. Me, I'd go the latter route as it would be cleaner, not to mention easier to setup and maintain. IMHO.

hmm, I see... seems that there is no other choice except this solution. thanks very much for your help and clue.. nice to know you,and sorry what I did last time

No worries and good luck. Should be... interesting.