Page 1 of 2
job Parameters for Sequencer Jobs
Posted: Mon Oct 11, 2004 12:32 pm
by poorna_76
Is there any way i can make my Sequencer job to read the JobParameters from a Parameter File?
I have 7 jobs to be put in JobSequencer.
All these jobs read the parameters from a Parameter File.
Instead of manually setting the Parameter values in JobSequencer, can i make the Sequncer job to read from a Parameter File?
Thanks in Advance.
Posted: Mon Oct 11, 2004 12:36 pm
by kcbland
No, the Sequencer lacks any functionality to do this.
Posted: Mon Oct 11, 2004 1:49 pm
by tonystark622
Kenneth,
I think that this answers Poorna_76's question, but since you said that the Sequencer can't do this, I'm not sure if I understood his question correctly. Can you review the scenario I describe below and let me know if I'm off base? Thanks!
Poorna_76
We do this with most of our job sequencers.
We have a parameter file formatted as Name=Value. I wrote a routine that takes a file name and a parameter name and returns the value of the parameter from the file.
We use a Routine Activity stage to execute this routine and retrieve the value of the parameter from the file... One Routine Activity stage for each Job Parameter to retrieve. I normally name the Routine Activity stage the same as the Job Parameter that I'm retrieving. So, if I have a Job Parameter called DB_NAME, I'll have a Routine Activity named DB_NAME.
In the Job Activity stage, where the Job Parameters are set, put the Routine Activity stage name followed by .$ReturnValue. So, in the Job Activity stage for the DB_NAME Job Parameter "Value Expression", I put DB_NAME.$ReturnValue . At runtime, the Routine Activity fetches the Job Parameter value from the file and feeds it to the Job Activity.
I hope this makes sense. If not, please ask more questions and I'll do my best to explain.
tony
Posted: Mon Oct 11, 2004 1:57 pm
by kcbland
The Sequencer has no capability to feed parameters from a file into a job via the Job Activity stage functionality.
However, if you write a function (which Tony has) which reads a file, parses the contents, and returns the values into fixed return codes, you can send those results into the job parameter values in the Job Activity stage.
BUT, there is no functionality in the Sequencer to do this out of the box. To quote Ray:
"You must roll your own."
Posted: Mon Oct 11, 2004 1:59 pm
by kcbland
Also consider the scalability of a RoutineActivity stage per job parameter. If you have 20+ parameters in a job (paths, credentials, dates, etc), then this becomes an nasty job.
Posted: Mon Oct 11, 2004 2:15 pm
by tonystark622
Also consider the scalability of a RoutineActivity stage per job parameter. If you have 20+ parameters in a job (paths, credentials, dates, etc), then this becomes an nasty job.
Yes it does, no argument from me. I wish we could stuff all these in a container, but there is no container stage in Sequencer jobs
Thanks for reviewing my email, Kenneth.
Tony
Posted: Mon Oct 11, 2004 8:53 pm
by kduke
Ken
I friend of mine Peter Lai invented this routine which you can call as routine activity and from that point on it will change the value of one parameter. I have heavily modified it. Peter's original version would read all the parameters in a sequential file and set any or all of them. I just wanted to set them one at a time. I first have a job which will update a specific hash file named MaxHashKeyHash. The key to this hash file is the parameter name. I wanted to select the max key from a hash file which has the original source keys and then I only need to get keys greater than this from my dimension to build my crossref hash file. This saves reading millions of rows and allows us to sort of treat a hash file like it is persistant but rebuild it if necessary.
Anyway here is the code. Thanks Peter you are the best. This should eliminate the need for batch jobs.
Code: Select all
* ------------------------------------------------------------
* KgdSetLastKeyParam(ParamName, KeyName, Debug)
* Decription: Set parameters from a file
* Written by: Kim Duke
* Copied From: Peter Lai
* Notes:
* Routine set one parameter value from a hash file.
* ------------------------------------------------------------
$INCLUDE DSINCLUDE JOBCONTROL.H
$INCLUDE DSINCLUDE DSD_STAGE.H
$INCLUDE DSINCLUDE DSD.H
$INCLUDE DSINCLUDE DSD_RTSTATUS.H
*----------------------*
* Initialize Variables *
*----------------------*
Ans = 0 ; * 0=Successful(default), 1=Error occured
ProgName = "Function: KgdSetLastKeyParam"
*------*
* Main *
*------*
HashFileName = "MaxHashKeyHash"
Open HashFileName TO HashFile Else
Ans = 1
Call DSLogWarn('HashFile "':HashFileName:'" does not exist.',ProgName)
goto ProgramEnd
End
* -----------------------------------------------------------------
* Read the named record from the file.
* -----------------------------------------------------------------
Readv ParamValue From HashFile, KeyName, 1 Else
Ans = 1
Call DSLogWarn('"':HashFileName:'", "':KeyName:'" does not exist.',ProgName)
goto ProgramEnd
* ParamValue = 0
End
ParamList = ParamName : @FM : ParamValue
If Debug then Call DSLogInfo('Parameter Name: ':ParamName,ProgName)
If Debug then Call DSLogInfo('Parameter Value: ':ParamValue,ProgName)
JobName = Field(STAGECOM.NAME,'.',1,2)
if Debug then Call DSLogInfo('STAGECOM.NAME=':STAGECOM.NAME,JobName:" ":ProgName)
StatusFileName = FileInfo(DSRTCOM.RTSTATUS.FVAR,1)
StatusId = JobName:'.':STAGECOM.WAVE.NUM
if Debug then Call DSLogInfo('StatusFileName=':StatusFileName,JobName:" ":ProgName)
* Get all job parameter names and values
JobParamList = STAGECOM.JOB.CONFIG<CONTAINER.PARAM.NAMES>
if Debug then Call DSLogInfo('Job Parameter: ':JobParamList,JobName:" ":ProgName)
IF JobParamList <> "" THEN
LOCATE ParamName IN JobParamList<1> SETTING JobParamPos THEN
JobParamName = JobParamList<1,JobParamPos>
JobParamValue = ParamValue
STAGECOM.JOB.STATUS<JOB.PARAM.VALUES,JobParamPos> = JobParamValue
Call DSLogInfo('Parameter: ':JobParamName:' = ':JobParamValue,JobName:" ":ProgName)
END ELSE
Ans = 1
Call DSLogWarn('Parameter "':ParamName:'" does not exist in parameters.',ProgName)
goto ProgramEnd
END
END
ProgramEnd:
print Ans
job Parameters for Sequencer Jobs
Posted: Tue Oct 12, 2004 7:58 am
by poorna_76
Kenneth,
To be more clear about my question:
I have 7 jobs to be executed through Sequncer.
Each job takes more than 15 parameters.
Right now i am creating a seperate JobControl job,
which will set the parameters and run my jobs from there.
DSAttach..
DSRunJob..
DSWait....
I thought there might be some thing similar, in SequencerJobs too.
Now i know Sequencer lacks this functionality.
Thanks for all.
tonystark622 wrote:Kenneth,
I think that this answers Poorna_76's question, but since you said that the Sequencer can't do this, I'm not sure if I understood his question correctly. Can you review the scenario I describe below and let me know if I'm off base? Thanks!
Poorna_76
We do this with most of our job sequencers.
We have a parameter file formatted as Name=Value. I wrote a routine that takes a file name and a parameter name and returns the value of the parameter from the file.
We use a Routine Activity stage to execute this routine and retrieve the value of the parameter from the file... One Routine Activity stage for each Job Parameter to retrieve. I normally name the Routine Activity stage the same as the Job Parameter that I'm retrieving. So, if I have a Job Parameter called DB_NAME, I'll have a Routine Activity named DB_NAME.
In the Job Activity stage, where the Job Parameters are set, put the Routine Activity stage name followed by .$ReturnValue. So, in the Job Activity stage for the DB_NAME Job Parameter "Value Expression", I put DB_NAME.$ReturnValue . At runtime, the Routine Activity fetches the Job Parameter value from the file and feeds it to the Job Activity.
I hope this makes sense. If not, please ask more questions and I'll do my best to explain.
tony
Posted: Tue Oct 12, 2004 8:42 am
by chulett
Depending on the nature of what you are doing, they are trying to address situations like this. If you have 7.5 there are new Sequencer stages to control "looping" within a Sequencer job and from what I've read the LoopStart stage has some ability to pull information from a delimited source.
I expect functionality in this area to be augmented in future versions.
Posted: Tue Oct 12, 2004 8:53 am
by kduke
You do not need the batch job when you use the routine posted above. Your first job could read from a sequence job and write to this hash file. Next is the above routine activity. You could call it 15 times or modify it to send it 15 paramaters or hard code the parameter names. Next have all your jobs wait on this one job and one routine activity. Then start all your jobs.
Peter has one routine that does step one and step two.
Code: Select all
ParamSeqToHash -> LoadParamFromHash -> Wait1 -> StartAllMyJobsHere
This works try it. It is very cool. This will change the values of the parameters in the current sequence.
Posted: Tue Oct 12, 2004 9:05 am
by peternolan9
Hi All,
well, I have never liked the job scheduler language and the poor dependency management.....
So, I just went the 'easy' way for my own testing and wrote a command scheduler and a DataStage job submission utility for most of my testing....the job submission utility allows the presentation of parameters from any ODBC data source.
These utilities can drive higher utilisation of the machine by intelligently managing dependencies between groups of jobs without the need to updating any code on a restart.....
I don't use it in production yet, but if anyone wants a really good scheduler on win2000 that can also schedule DS jobs feel free to get it from my web page (source code and all) or contact me.....if they get used enough and are reliable enough there is no reason why they could not be used in production environments...
Posted: Tue Oct 12, 2004 3:17 pm
by ray.wurlod
I wrote one of these that's been in production for nearly two years. It uses BCI functions to access an ODBC database (Oracle 7 as it happens). The tricky part was an arbitrarily deep level of nesting in the dependencies.
Posted: Thu Oct 21, 2004 1:09 pm
by martin
Instead of running ur 7 jobs through sequencer,
I would recommend u to run ur jobs from JobControl
Where you Can Pass Parameters to all Jobs By Reading parameter File.
Thats What We are doing.
Posted: Thu Oct 21, 2004 2:12 pm
by tonystark622
Martin,
It's possible to do the same thing from a Sequencer Job.
Tony
Posted: Thu Oct 21, 2004 3:24 pm
by martin
Tony
The Way U Define In JobControl( Flexibility In Using Basic Language)
U Can Write Better Job Sequence..