Using an array as a job handler

[fordxhome]

Hi Ken,
May I get a copy as well. Thanks.

[fjgirante]

Hi kcbland,
I would like to ask you, if you could share with me your job sequencer.
I'm very curious about it.

Thank you in advance
FG

Absolutely, you can put job handles into an array.

In fact, the job control that myself and associates have developed and freely distributes does everything that Guillermo has so far asked about. Namely:

1. Users a predecessor/successor array of jobs to control job execution. Think of Microsoft Project with tasks as jobs. In fact, my associates and myself have built a Project .mpp file with macros and templates as the graphical metaphor for our job control sequencer. Jobs belong to groups, with rolled up tasks, predecessor relationships, etc.
2. Jobs have individual attributes such as maximum runtime before a warning notifcation needs to be sent, maximum number of warning messages before a warning notification needs to be sent, maximum runtime and warnings before the job is automatically killed.
3. Jobs have an instantiation parameter, meaning when that job is run the job control should actually run N number of those jobs, passing parameters PartitionCount set to N and PartitionNumber set from 1 to N for the N jobs. This means at runtime the jobs dynamically expand to N instances for maximized throughput in Instantiate-Empowered jobs.
4. Jobs have automatic parameter resolution at a job level. If you supply a runtime parameter file, plus any custom parameters at startup, all jobs are resolved against that file+custom list and set at runtime.
5. Job control has an auto-compile option to compile the job stream at runtime to insure all jobs are compiled and ready to run, as well as cleanup a previous run's instantiated job sets.
6. Job control automatically resets aborted job.
7. Job control publishes log metadata, as well as a spreadsheet of job run, their completion status, start time, and end time.
8. Job control publishes link statistic information in a spreadsheet form for each job executed.
9. Job control has a hook in it to load all process metadata created during runtime, so that an external audit repository is kept up-to-date during job control execution. Really cool if you run an intranet and want your process metadata in Oracle/SQL-Server, etc, without a MetaStage import/cleanup/publish/export lag.
10. Job control has a resurrect capability, so that it can be restarted in a resurrection mode and reads it's own log file of jobs executed and retries failed jobs and continues through the predecessor/successor tree.
11. Job control uses categorization of jobs into subject areas and source system supplied data so that you can "prune" the jobstream at runtime with parameters that dictate which subject areas or source system related jobs you want run.
12. Job control has a milestone feature where a synchronization step can be put into the stream to provide a predecessor/successor stall point. That allows you to "jump" to a point in the job stream, skipping over predecessor jobs all the way to the inception point, and also end at a point in the job stream. Think of many parallel streams connection and diverging, with the ability to prune branches and roots by either source system, subject area, or jump-to/stop-at capabilities.
13. Customizable notification based on job finish statuses, jobs running long, blah blah blah.
14. WORK-AROUNDS for bugs in DataStage, such as running out of disk space and jobs not dying. We check job logs during/after execution to make sure some of these bugs are treated as fatal.
15. Runtime link statistic monitorring. You can set rules to watch jobs as they execute and make sure link values are within tolerance. Imagine a reference lookup that you require 90% or better hits. You can define a simple rule in a spreadsheet to require a 90% ratio between two link values after N number of rows have processed else the job is automatically killed or just a warning is sent out. You could even say warn at 90% and kill at 75%, for example.
16. Load throttling, where you set the maximum number of jobs to be running at any given time in your jobstream. Kind of handy for CPU-challenged systems.
17. External file polling mechanism, to introduce PAUSE, RESUME, STOP, KILL, STATUS controls for third-parties to interact once a job stream has initiated. Pause means don't start any new jobs, resume means continue starting new jobs, stop means stop starting any new, wait for jobs already running to finish and then stop, kill of course means kill off any jobs currently running and then stop, and status says dump a current status file out of what jobs have run, those that are waiting to be run and their predecessors. It's pretty cool to STOP a job stream, then resurrect it, because it picks up right where it left off.
18. Written by consultants who love DataStage and have used it in many environments since the earliest releases. We know what is really expected out there and find the one-size-fits-all as-long-as-its-black Sequencer not to our liking.
...
...
99. Free. It was developed in a cooperative mood by consultants and clients over the last few years. All we ask is that you improve it, don't sell, give us a copy back with your enhancements if you think anyone else out there might benefit from it.

If anyone is interested drop me an email. I have a release 6.0 compatible version available with documentation. At worst, it may be helpful to simply look at the logic written to see how we used inherent API's.

Kenneth Bland

[bhagavath]

Hi Ken,

Can i get one copy.

Thanks,
Kumar

[poodari2]

Hi Ken,

Can I get a copy or a link for downloading the code and the documentation.

Thanks
Srinivas

[Miguel.A.Rodriguez]

Hello Keneth. I'm new to DataStage, but I'm an ardent student of that product. Is the Job Control documentation & code described in your email still available? When you can, please show me how I can receive a copy to Miguel.A.Rodriguez@AdvantexMail.Net.

Your efforts are deeply appreciated.

Try to have a pleasant day.

Absolutely, you can put job handles into an array.

In fact, the job control that myself and associates have developed and freely distributes does everything that Guillermo has so far asked about. Namely:

1. Users a predecessor/successor array of jobs to control job execution. Think of Microsoft Project with tasks as jobs. In fact, my associates and myself have built a Project .mpp file with macros and templates as the graphical metaphor for our job control sequencer. Jobs belong to groups, with rolled up tasks, predecessor relationships, etc.
2. Jobs have individual attributes such as maximum runtime before a warning notifcation needs to be sent, maximum number of warning messages before a warning notification needs to be sent, maximum runtime and warnings before the job is automatically killed.
3. Jobs have an instantiation parameter, meaning when that job is run the job control should actually run N number of those jobs, passing parameters PartitionCount set to N and PartitionNumber set from 1 to N for the N jobs. This means at runtime the jobs dynamically expand to N instances for maximized throughput in Instantiate-Empowered jobs.
4. Jobs have automatic parameter resolution at a job level. If you supply a runtime parameter file, plus any custom parameters at startup, all jobs are resolved against that file+custom list and set at runtime.
5. Job control has an auto-compile option to compile the job stream at runtime to insure all jobs are compiled and ready to run, as well as cleanup a previous run's instantiated job sets.
6. Job control automatically resets aborted job.
7. Job control publishes log metadata, as well as a spreadsheet of job run, their completion status, start time, and end time.
8. Job control publishes link statistic information in a spreadsheet form for each job executed.
9. Job control has a hook in it to load all process metadata created during runtime, so that an external audit repository is kept up-to-date during job control execution. Really cool if you run an intranet and want your process metadata in Oracle/SQL-Server, etc, without a MetaStage import/cleanup/publish/export lag.
10. Job control has a resurrect capability, so that it can be restarted in a resurrection mode and reads it's own log file of jobs executed and retries failed jobs and continues through the predecessor/successor tree.
11. Job control uses categorization of jobs into subject areas and source system supplied data so that you can "prune" the jobstream at runtime with parameters that dictate which subject areas or source system related jobs you want run.
12. Job control has a milestone feature where a synchronization step can be put into the stream to provide a predecessor/successor stall point. That allows you to "jump" to a point in the job stream, skipping over predecessor jobs all the way to the inception point, and also end at a point in the job stream. Think of many parallel streams connection and diverging, with the ability to prune branches and roots by either source system, subject area, or jump-to/stop-at capabilities.
13. Customizable notification based on job finish statuses, jobs running long, blah blah blah.
14. WORK-AROUNDS for bugs in DataStage, such as running out of disk space and jobs not dying. We check job logs during/after execution to make sure some of these bugs are treated as fatal.
15. Runtime link statistic monitorring. You can set rules to watch jobs as they execute and make sure link values are within tolerance. Imagine a reference lookup that you require 90% or better hits. You can define a simple rule in a spreadsheet to require a 90% ratio between two link values after N number of rows have processed else the job is automatically killed or just a warning is sent out. You could even say warn at 90% and kill at 75%, for example.
16. Load throttling, where you set the maximum number of jobs to be running at any given time in your jobstream. Kind of handy for CPU-challenged systems.
17. External file polling mechanism, to introduce PAUSE, RESUME, STOP, KILL, STATUS controls for third-parties to interact once a job stream has initiated. Pause means don't start any new jobs, resume means continue starting new jobs, stop means stop starting any new, wait for jobs already running to finish and then stop, kill of course means kill off any jobs currently running and then stop, and status says dump a current status file out of what jobs have run, those that are waiting to be run and their predecessors. It's pretty cool to STOP a job stream, then resurrect it, because it picks up right where it left off.
18. Written by consultants who love DataStage and have used it in many environments since the earliest releases. We know what is really expected out there and find the one-size-fits-all as-long-as-its-black Sequencer not to our liking.
...
...
99. Free. It was developed in a cooperative mood by consultants and clients over the last few years. All we ask is that you improve it, don't sell, give us a copy back with your enhancements if you think anyone else out there might benefit from it.

If anyone is interested drop me an email. I have a release 6.0 compatible version available with documentation. At worst, it may be helpful to simply look at the logic written to see how we used inherent API's.

Kenneth Bland

[richdhan]

Kenneth,

Pls let me know once you make it available for download.

Thanks
Rich

[Miguel.A.Rodriguez]

Hello, do you know how I can receive copies of the Job Control documentation and code which Ken had described? I'd deeply appreciate your assistance.

Thank you.

Hi Ken,
May I get a copy as well. Thanks.

[kcbland]

Please be patient, my website is down right now and is being recovered. I'll make everything available shortly.

[divine]

Please be patient, my website is down right now and is being recovered. I'll make everything available shortly.

Hi Ken,

I need a copy as well.

Please let us know once your website will be up and available to access.

[chulett]

Umm... now. The 'down' message was from 3 years ago.

[kcbland]

Just double-checked, as of right now it's up and running.

[fyaqq]

Please send me a copy. Thanks

[kcbland]

No thanks, but you can go get it yourself.

[kduke]

We found that the array is not accurate. It seems that DataStage will not allow you to attach to a lot of jobs. It appears to disconnect. If the status becomes negative then we attach again. It runs really smooth now.

[ray.wurlod]

A dimensioned array or a dynamic array? And what is "a lot" here?