Set Job Parameters in Sequence without a batch job

[kduke]

Tony asked this in a private message:

Kim,

When you have a minute I would appreciate a hand. If this private message isn't the proper way of asking this question, please let me know or just disregard it. I don't want to take up too much of your time.

I'm trying to understand how to implement the routine that you posted for setting a Job Param from a hash file. Could you please explain the process again in just a bit more detail. I think where I'm getting lost is 1) how does the routine know which job(s) to set params for and 2) you put a ---Wait--- step in the Job Sequence you described. I'm not sure what that means.

As far as 1) goes, I don't understand how it knows what jobs are in JobActivity stages downstream from the RoutineActivity where the routine is executed.

Thanks again,
Tony

I wanted to answer it here because I think others are confused. The trick Peter figured out is this. You have access to the actual paramater values. Most of us use the variable with the same name as the parameter to get its value but to set it you cannot do:

ParamName = "Param Value"

It does not work in job control. In routine activity you can change this value indirectly or actually directly based on how the run time code actually works. The parameter names are stored in

STAGECOM.JOB.CONFIG<CONTAINER.PARAM.NAMES>

This list is multivalued. The parameter values are stored in:

STAGECOM.JOB.STATUS<JOB.PARAM.VALUES>

So if you locate the parameter name in STAGECOM.JOB.CONFIG<CONTAINER.PARAM.NAMES> and then set the value in:

STAGECOM.JOB.STATUS<JOB.PARAM.VALUES,JobParamPos> = JobParamValue

Then from this point on in the job sequence the parameter value is exactly what you just set it to. So I need to make sure all my jobs run after this routine activity.

What I did in KgdSetLastKeyParam was read my parameter value from a hash file called MaxHashKeyHash. The parameter name is the first argument to this routine. The second argument is the hash file id. Using this I can read the parameter value from MaxHashKeyHash and set ParamName to the value just read by locating it in STAGECOM.JOB.CONFIG<CONTAINER.PARAM.NAMES>. Based on what position this parameter name is in this multivalued field then I change STAGECOM.JOB.STATUS<JOB.PARAM.VALUES,JobParamPos>.

The reason this routine is named KgdSetLastKeyParam is because I wanted to load new keys from my dimensions into my crossref hash file. This hash file has source keys and looks up the surrogate key to see if it is new or not. So if I already have a million records in this hash file then why load them all. So first I get the max(surrogate_key) from this hash file. I use a UV stage. I works just like the job that reads max key from our dimension.

select max(#SurrogateKey#) from #HashFile#

Now that I have the last key in my crossref hash file then I need to feed it in to job that builds my crossref hash file. So my select for building this hash file is now:

select * from DimTable where Key > #MaxKeyValue#

I can set this to zero if I want to totally rebuild this hash file.

This is very complex but should see large gains by not reading millions of rows. We all try to do incremental loads from source to target. Why not do incremental loads from target to crossref hash files.

Cool huh?

[kduke]

This routine's code is in this post viewtopic.php?t=89419.

[kduke]

Tony

Sorry, I hope you don't mind me responding this way but I think I did a poor job of explaining this routine. I hope this is a better explaination.

See you guys at ASCL world.

[kduke]

If anyone wants Peter Lai's original routines to read a group of parameters from a sequential file then ask. I will ask Peter if it is ok to post it. He will be at ASCL world so ask him if you see him. He is a much nicer man than I am so treat him good.

Go by and say hello to Dennis James. He owns this web site and will have a both at the conference. Thanks Dennis for such an awesome web site.

[martin]

would you please Make Peter Lai's original routines available to the forum

Thanks
martin

[kduke]

Okay:

Code: Select all

* -----------------------------------------------------------------
* UICISetParamsFromFile(ParamFile,FromParams,ToParams,Debug)
* Decription: Set parameters in a running sequence.
* Written by: Peter Lai
* Notes:
* -----------------------------------------------------------------
$INCLUDE DSINCLUDE JOBCONTROL.H
$INCLUDE DSINCLUDE DSD_STAGE.H
$INCLUDE DSINCLUDE DSD.H
$INCLUDE DSINCLUDE DSD_RTSTATUS.H


*------------------*
* Define Functions *
*------------------*

      DEFFUN GetParameterArray(A) Calling 'DSU.GetParameterArray'


*----------------------*
* Initialize Variables *
*----------------------*

      Ans = 0                            ; * 0=Successful(default), 1=Error occured
      ProgName = "Function: UICISetParamsFromFile"
      convert ',;|' to @FM:@FM:@FM in FromParams   ; * convert ,;| delimited to field mark
      convert ',;|' to @FM:@FM:@FM in ToParams     ; * convert ,;| delimited to field mark

      if trim(ToParams) = "" then
         Call DSLogWarn('ToParams parameter list required.',JobName:" ":ProgName)
         Ans = 1                         ; * Set error flag
         goto ProgramEnd
      END

      If Debug then Call DSLogInfo('From Parameter Names: ':FromParams,ProgName)
      If Debug then Call DSLogInfo('To Parameter Names: ':ToParams,ProgName)


*------*
* Main *
*------*

      * Get all default parameter names and values from input file
      ParamList = GetParameterArray(ParamFile)

      If Debug then Call DSLogInfo('Parameter Names: ':ParamList<1>,ProgName)
      If Debug then Call DSLogInfo('Parameter Values: ':ParamList<2>,ProgName)

      IF ParamList <> "" THEN

         JobName = Field(STAGECOM.NAME,'.',1,2)
         if Debug then Call DSLogInfo('STAGECOM.NAME=':STAGECOM.NAME,JobName:" ":ProgName)

         StatusFileName = FileInfo(DSRTCOM.RTSTATUS.FVAR,1)
         StatusId = JobName:'.':STAGECOM.WAVE.NUM
         if Debug then Call DSLogInfo('StatusFileName=':StatusFileName,JobName:" ":ProgName)

         * Get all job parameter names and values

         JobParamList = STAGECOM.JOB.CONFIG<CONTAINER.PARAM.NAMES>
         if Debug then Call DSLogInfo('Job Parameter: ':JobParamList,JobName:" ":ProgName)

         IF JobParamList <> "" THEN

            ToParamCount = Dcount(ToParams,@FM)
            For ToParamPos = 1 to ToParamCount

               FrParamName = FromParams<ToParamPos>
               ToParamName = ToParams<ToParamPos>

               * if empty string set FromParam name to ToParam name as default value
               if FrParamName = "" then FrParamName = ToParamName

               * Validate parameter if it exists the job parameters
               LOCATE ToParamName IN JobParamList<1> SETTING JobParamPos THEN

                  LOCATE FrParamName IN ParamList<1> SETTING ParamPos THEN
                     ParamValue = ParamList<2,ParamPos>
                     STAGECOM.JOB.STATUS<JOB.PARAM.VALUES,JobParamPos> = ParamValue
                     Call DSLogInfo('Set FromParameter: ':FrParamName:' ToParameter: ':ToParamName:' = ':ParamValue,JobName:" ":ProgName)
                  END ELSE
                     Ans = 1
                     Call DSLogWarn('Parameter "':ToParamName:'" does not exist in parameter file ':ParamFile:'.',ProgName)
                     goto ProgramEnd
                  END

               END ELSE
                  Ans = 1
                  Call DSLogWarn('Parameter "':ToParamName:'" does not exist in job parameter.',ProgName)
                  goto ProgramEnd
               END

            Next JobParamPos

         END
      END ELSE
         Call DSLogWarn('Missing parameter list from input file ':ParamFile,JobName:" ":ProgName)
         Ans = 1                         ; * Set error flag
      END

ProgramEnd:

[kduke]

Code: Select all

* ------------------------------------------------------------
* UICISetJobParamsFromFile(ParamFile, SkipParams, Debug)
* Decription: Set parameters from a file
* Written by: Kim Duke
* Notes:
* ------------------------------------------------------------
$INCLUDE DSINCLUDE JOBCONTROL.H
$INCLUDE DSINCLUDE DSD_STAGE.H
$INCLUDE DSINCLUDE DSD.H
$INCLUDE DSINCLUDE DSD_RTSTATUS.H


*------------------*
* Define Functions *
*------------------*

      DEFFUN GetParameterArray(A) Calling 'DSU.GetParameterArray'


*----------------------*
* Initialize Variables *
*----------------------*

      Ans = 0                            ; * 0=Successful(default), 1=Error occured
      ProgName = "Function: UICISetJobParamsFromFile"
      convert ',;|' to @FM:@FM:@FM in SkipParams   ; * convert ,;| delimited to field mark

*------*
* Main *
*------*

      * Get all default parameter names and values from input file
      ParamList = GetParameterArray(ParamFile)
      If Debug then Call DSLogInfo('Parameter Names: ':ParamList<1>,ProgName)
      If Debug then Call DSLogInfo('Parameter Values: ':ParamList<2>,ProgName)
      If Debug then Call DSLogInfo('Skip Parameters: ':SkipParams,ProgName)

      IF ParamList <> "" THEN

         JobName = Field(STAGECOM.NAME,'.',1,2)
         if Debug then Call DSLogInfo('STAGECOM.NAME=':STAGECOM.NAME,JobName:" ":ProgName)

         StatusFileName = FileInfo(DSRTCOM.RTSTATUS.FVAR,1)
         StatusId = JobName:'.':STAGECOM.WAVE.NUM
         if Debug then Call DSLogInfo('StatusFileName=':StatusFileName,JobName:" ":ProgName)

         * Get all job parameter names and values

         JobParamList = STAGECOM.JOB.CONFIG<CONTAINER.PARAM.NAMES>
         if Debug then Call DSLogInfo('Job Parameter: ':JobParamList,JobName:" ":ProgName)

         IF JobParamList <> "" THEN

            JobParamCount = Dcount(JobParamList<1>,@VM)
            For JobParamPos = 1 to JobParamCount

               JobParamName = JobParamList<1,JobParamPos>

               * check for skip parameter
               LOCATE JobParamName IN SkipParams SETTING TempPos THEN continue

               LOCATE JobParamName IN ParamList<1> SETTING ParamPos THEN
                  ParamValue = ParamList<2,ParamPos>
                  STAGECOM.JOB.STATUS<JOB.PARAM.VALUES,JobParamPos> = ParamValue
                  Call DSLogInfo('Parameter: ':JobParamName:' = ':ParamValue,JobName:" ":ProgName)
               END ELSE
                  Ans = 1
                  Call DSLogWarn('Parameter "':JobParamName:'" does not exist in parameter file ':ParamFile:'.',ProgName)
                  goto ProgramEnd
               END

            Next JobParamPos

         END
      END ELSE
         Call DSLogWarn('Missing parameter list from input file ':ParamFile,JobName:" ":ProgName)
         Ans = 1                         ; * Set error flag
      END

ProgramEnd:

[kduke]

Code: Select all

* ------------------------------------------------------------
* GetParameterArray(Arg1)
* Decription: Get parameters
* Written by: 
* Notes:
* Bag of Tricks Version 2.3.0  Release Date 2001-10-01
* Arg1 = Path and Name of Parameter File
* 
* Result =  ( <1> = Parameter names, <2> = Parameter values)
* ------------------------------------------------------------
      DEFFUN FileFound(A) Calling 'DSU.FileFound'

      cBlank = ''
      cName = 1
      cValue = 2

      vParamFile = Arg1
      aParam = cBlank
      vParamCnt = 0
      vCurRoutineName = 'Routine: GetParameterArray'
      vFailed = @FALSE
      Done = @FALSE

      IF vParamFile AND FileFound(vParamFile) Then
         OPENSEQ vParamFile TO hParamFile Then
            Loop
               READSEQ vLineRaw FROM hParamFile
               ON ERROR
                  Call DSLogWarn('Error from ':vParamFile:'; Status = ':STATUS(),vCurRoutineName)
                  CLOSE hParamFile
                  vFailed = @TRUE
                  Done = @TRUE
               End Then
                  vLine = TRIM(vLineRaw)
                  vFirstChar = LEFT(vLine,1)
                  vRemark = LEFT(vLine,4)
                  IF NOT(vFirstChar = cBlank OR vFirstChar = '#' OR vFirstChar = '*' OR vFirstChar = '"' OR vFirstChar = "'" OR vFirstChar = ';' OR vFirstChar = ':' OR vFirstChar = '[' OR vRemark = 'REM ') THEN
                     vParamCnt += 1      ; * Add to any parameter array passed as an argument
                     aParam<1,vParamCnt> = TRIM(FIELD(vLine,'=',cName))
                     aParam<2,vParamCnt> = FIELD(vLine,'=',cValue)
                  END
               END ELSE
                  Done = @TRUE
               END
            Until Done Do Repeat
            CLOSE hParamFile

         End Else
            Call DSLogWarn('Error from ':vParamFile:'; Status = ':STATUS(),vCurRoutineName)
            vFailed = @TRUE
         End
      End Else
         vFailed = @TRUE
      End

      Call DSLogInfo("Values loaded from file: ":vParamFile:@AM:aParam, vCurRoutineName)
      If vFailed Then
         Ans = "ERROR"
      End Else
         Ans = aParam
      End

[kduke]

Code: Select all

* ------------------------------------------------------------
* SetJobParameters(Arg1, Arg2, Arg3)
* Decription: Set parameters
* Written by: 
* Notes:
* Bag of Tricks Version 2.3.0  Release Date 2001-10-01
* Arg1 = Parameter List ( <1> = parameter names, <2> = parameter values)
* Arg2 = Job Handle that parameters will be set for.  
* (Must be attached to the Job/Batch)
* Arg3 = Parameter names to be skipped. (Comma seperated)
* ------------------------------------------------------------
      DEFFUN FileFound(A) Calling 'DSU.FileFound'

      cBlank = ''
      cName = 1
      cValue = 2

      vParamFile = Arg1
      aParam = cBlank
      vParamCnt = 0
      vCurRoutineName = 'Routine: GetParameterArray'
      vFailed = @FALSE
      Done = @FALSE

      IF vParamFile AND FileFound(vParamFile) Then
         OPENSEQ vParamFile TO hParamFile Then
            Loop
               READSEQ vLineRaw FROM hParamFile
               ON ERROR
                  Call DSLogWarn('Error from ':vParamFile:'; Status = ':STATUS(),vCurRoutineName)
                  CLOSE hParamFile
                  vFailed = @TRUE
                  Done = @TRUE
               End Then
                  vLine = TRIM(vLineRaw)
                  vFirstChar = LEFT(vLine,1)
                  vRemark = LEFT(vLine,4)
                  IF NOT(vFirstChar = cBlank OR vFirstChar = '#' OR vFirstChar = '*' OR vFirstChar = '"' OR vFirstChar = "'" OR vFirstChar = ';' OR vFirstChar = ':' OR vFirstChar = '[' OR vRemark = 'REM ') THEN
                     vParamCnt += 1      ; * Add to any parameter array passed as an argument
                     aParam<1,vParamCnt> = TRIM(FIELD(vLine,'=',cName))
                     aParam<2,vParamCnt> = FIELD(vLine,'=',cValue)
                  END
               END ELSE
                  Done = @TRUE
               END
            Until Done Do Repeat
            CLOSE hParamFile

         End Else
            Call DSLogWarn('Error from ':vParamFile:'; Status = ':STATUS(),vCurRoutineName)
            vFailed = @TRUE
         End
      End Else
         vFailed = @TRUE
      End

      Call DSLogInfo("Values loaded from file: ":vParamFile:@AM:aParam, vCurRoutineName)
      If vFailed Then
         Ans = "ERROR"
      End Else
         Ans = aParam
      End