job Parameters for Sequencer Jobs

poorna_76 · Post by **poorna_76** » Mon Oct 11, 2004 12:32 pm

Is there any way i can make my Sequencer job to read the JobParameters from a Parameter File?

I have 7 jobs to be put in JobSequencer.

All these jobs read the parameters from a Parameter File.

Instead of manually setting the Parameter values in JobSequencer, can i make the Sequncer job to read from a Parameter File?

Thanks in Advance.

kcbland · Post by **kcbland** » Mon Oct 11, 2004 12:36 pm

No, the Sequencer lacks any functionality to do this.

tonystark622 · Post by **tonystark622** » Mon Oct 11, 2004 1:49 pm

Kenneth,

I think that this answers Poorna_76's question, but since you said that the Sequencer can't do this, I'm not sure if I understood his question correctly. Can you review the scenario I describe below and let me know if I'm off base? Thanks!

Poorna_76

We do this with most of our job sequencers.

We have a parameter file formatted as Name=Value. I wrote a routine that takes a file name and a parameter name and returns the value of the parameter from the file.

We use a Routine Activity stage to execute this routine and retrieve the value of the parameter from the file... One Routine Activity stage for each Job Parameter to retrieve. I normally name the Routine Activity stage the same as the Job Parameter that I'm retrieving. So, if I have a Job Parameter called DB_NAME, I'll have a Routine Activity named DB_NAME.

In the Job Activity stage, where the Job Parameters are set, put the Routine Activity stage name followed by .$ReturnValue. So, in the Job Activity stage for the DB_NAME Job Parameter "Value Expression", I put DB_NAME.$ReturnValue . At runtime, the Routine Activity fetches the Job Parameter value from the file and feeds it to the Job Activity.

I hope this makes sense. If not, please ask more questions and I'll do my best to explain.

tony

kcbland · Post by **kcbland** » Mon Oct 11, 2004 1:57 pm

The Sequencer has no capability to feed parameters from a file into a job via the Job Activity stage functionality.

However, if you write a function (which Tony has) which reads a file, parses the contents, and returns the values into fixed return codes, you can send those results into the job parameter values in the Job Activity stage.

BUT, there is no functionality in the Sequencer to do this out of the box. To quote Ray:

"You must roll your own."

kcbland · Post by **kcbland** » Mon Oct 11, 2004 1:59 pm

Also consider the scalability of a RoutineActivity stage per job parameter. If you have 20+ parameters in a job (paths, credentials, dates, etc), then this becomes an nasty job.

tonystark622 · Post by **tonystark622** » Mon Oct 11, 2004 2:15 pm

Also consider the scalability of a RoutineActivity stage per job parameter. If you have 20+ parameters in a job (paths, credentials, dates, etc), then this becomes an nasty job.

Yes it does, no argument from me. I wish we could stuff all these in a container, but there is no container stage in Sequencer jobs

Thanks for reviewing my email, Kenneth.

Tony

kduke · Post by **kduke** » Mon Oct 11, 2004 8:53 pm

Ken

I friend of mine Peter Lai invented this routine which you can call as routine activity and from that point on it will change the value of one parameter. I have heavily modified it. Peter's original version would read all the parameters in a sequential file and set any or all of them. I just wanted to set them one at a time. I first have a job which will update a specific hash file named MaxHashKeyHash. The key to this hash file is the parameter name. I wanted to select the max key from a hash file which has the original source keys and then I only need to get keys greater than this from my dimension to build my crossref hash file. This saves reading millions of rows and allows us to sort of treat a hash file like it is persistant but rebuild it if necessary.

Anyway here is the code. Thanks Peter you are the best. This should eliminate the need for batch jobs.

Code: Select all

* ------------------------------------------------------------
* KgdSetLastKeyParam(ParamName, KeyName, Debug)
* Decription: Set parameters from a file
* Written by: Kim Duke
* Copied From: Peter Lai
* Notes:
* Routine set one parameter value from a hash file.
* ------------------------------------------------------------
$INCLUDE DSINCLUDE JOBCONTROL.H
$INCLUDE DSINCLUDE DSD_STAGE.H
$INCLUDE DSINCLUDE DSD.H
$INCLUDE DSINCLUDE DSD_RTSTATUS.H

*----------------------*
* Initialize Variables *
*----------------------*

      Ans = 0                            ; * 0=Successful(default), 1=Error occured
      ProgName = "Function: KgdSetLastKeyParam"

*------*
* Main *
*------*
      HashFileName = "MaxHashKeyHash"
      Open HashFileName TO HashFile Else
         Ans = 1
         Call DSLogWarn('HashFile "':HashFileName:'" does not exist.',ProgName)
         goto ProgramEnd
      End
* -----------------------------------------------------------------
* Read the named record from the file.
* -----------------------------------------------------------------
      Readv ParamValue From HashFile, KeyName, 1 Else
         Ans = 1
         Call DSLogWarn('"':HashFileName:'", "':KeyName:'" does not exist.',ProgName)
         goto ProgramEnd
         * ParamValue = 0
      End

      ParamList = ParamName : @FM : ParamValue
      If Debug then Call DSLogInfo('Parameter Name: ':ParamName,ProgName)
      If Debug then Call DSLogInfo('Parameter Value: ':ParamValue,ProgName)
      
      JobName = Field(STAGECOM.NAME,'.',1,2)
      if Debug then Call DSLogInfo('STAGECOM.NAME=':STAGECOM.NAME,JobName:" ":ProgName)

      StatusFileName = FileInfo(DSRTCOM.RTSTATUS.FVAR,1)
      StatusId = JobName:'.':STAGECOM.WAVE.NUM
      if Debug then Call DSLogInfo('StatusFileName=':StatusFileName,JobName:" ":ProgName)

      * Get all job parameter names and values

      JobParamList = STAGECOM.JOB.CONFIG<CONTAINER.PARAM.NAMES>
      if Debug then Call DSLogInfo('Job Parameter: ':JobParamList,JobName:" ":ProgName)

      IF JobParamList <> "" THEN
         LOCATE ParamName IN JobParamList<1> SETTING JobParamPos THEN
            JobParamName = JobParamList<1,JobParamPos>
            JobParamValue = ParamValue
            STAGECOM.JOB.STATUS<JOB.PARAM.VALUES,JobParamPos> = JobParamValue
            Call DSLogInfo('Parameter: ':JobParamName:' = ':JobParamValue,JobName:" ":ProgName)
         END ELSE
            Ans = 1
            Call DSLogWarn('Parameter "':ParamName:'" does not exist in parameters.',ProgName)
            goto ProgramEnd
         END
      END

ProgramEnd:
      print Ans

poorna_76 · Post by **poorna_76** » Tue Oct 12, 2004 7:58 am

Kenneth,
To be more clear about my question:
I have 7 jobs to be executed through Sequncer.
Each job takes more than 15 parameters.
Right now i am creating a seperate JobControl job,
which will set the parameters and run my jobs from there.
DSAttach..
DSRunJob..
DSWait....

I thought there might be some thing similar, in SequencerJobs too.

Now i know Sequencer lacks this functionality.

Thanks for all.

tonystark622 wrote:Kenneth,

I think that this answers Poorna_76's question, but since you said that the Sequencer can't do this, I'm not sure if I understood his question correctly. Can you review the scenario I describe below and let me know if I'm off base? Thanks!

Poorna_76

We do this with most of our job sequencers.

We have a parameter file formatted as Name=Value. I wrote a routine that takes a file name and a parameter name and returns the value of the parameter from the file.

We use a Routine Activity stage to execute this routine and retrieve the value of the parameter from the file... One Routine Activity stage for each Job Parameter to retrieve. I normally name the Routine Activity stage the same as the Job Parameter that I'm retrieving. So, if I have a Job Parameter called DB_NAME, I'll have a Routine Activity named DB_NAME.

In the Job Activity stage, where the Job Parameters are set, put the Routine Activity stage name followed by .$ReturnValue. So, in the Job Activity stage for the DB_NAME Job Parameter "Value Expression", I put DB_NAME.$ReturnValue . At runtime, the Routine Activity fetches the Job Parameter value from the file and feeds it to the Job Activity.

I hope this makes sense. If not, please ask more questions and I'll do my best to explain.

tony

chulett · Post by **chulett** » Tue Oct 12, 2004 8:42 am

Depending on the nature of what you are doing, they are trying to address situations like this. If you have 7.5 there are new Sequencer stages to control "looping" within a Sequencer job and from what I've read the LoopStart stage has some ability to pull information from a delimited source.

I expect functionality in this area to be augmented in future versions.

kduke · Post by **kduke** » Tue Oct 12, 2004 8:53 am

You do not need the batch job when you use the routine posted above. Your first job could read from a sequence job and write to this hash file. Next is the above routine activity. You could call it 15 times or modify it to send it 15 paramaters or hard code the parameter names. Next have all your jobs wait on this one job and one routine activity. Then start all your jobs.

Peter has one routine that does step one and step two.

Code: Select all


ParamSeqToHash -> LoadParamFromHash -> Wait1 -> StartAllMyJobsHere

This works try it. It is very cool. This will change the values of the parameters in the current sequence.

peternolan9 · Post by **peternolan9** » Tue Oct 12, 2004 9:05 am

Hi All,
well, I have never liked the job scheduler language and the poor dependency management.....

So, I just went the 'easy' way for my own testing and wrote a command scheduler and a DataStage job submission utility for most of my testing....the job submission utility allows the presentation of parameters from any ODBC data source.

These utilities can drive higher utilisation of the machine by intelligently managing dependencies between groups of jobs without the need to updating any code on a restart.....

I don't use it in production yet, but if anyone wants a really good scheduler on win2000 that can also schedule DS jobs feel free to get it from my web page (source code and all) or contact me.....if they get used enough and are reliable enough there is no reason why they could not be used in production environments...

ray.wurlod · Post by **ray.wurlod** » Tue Oct 12, 2004 3:17 pm

I wrote one of these that's been in production for nearly two years. It uses BCI functions to access an ODBC database (Oracle 7 as it happens). The tricky part was an arbitrarily deep level of nesting in the dependencies.

martin · Post by **martin** » Thu Oct 21, 2004 1:09 pm

Instead of running ur 7 jobs through sequencer,
I would recommend u to run ur jobs from JobControl
Where you Can Pass Parameters to all Jobs By Reading parameter File.
Thats What We are doing.

tonystark622 · Post by **tonystark622** » Thu Oct 21, 2004 2:12 pm

Martin,

It's possible to do the same thing from a Sequencer Job.

Tony

martin · Post by **martin** » Thu Oct 21, 2004 3:24 pm

Tony

The Way U Define In JobControl( Flexibility In Using Basic Language)
U Can Write Better Job Sequence..