How to take Values from Parameter File?

[nvkravi]

Hi All,
Im new to DataStage. Here I have got one problem with Dates. Here I have 2 dates MAXDATE and MINDATE. Here Im running 10 Jobs. So, How to take MAXDATE and MINDATE values from Parameter File? Is this possible with DataStage Job Parameters? R we have to assign as DEFAULT values? R have to use any Built-In Functions? Pls help me out in this? It is easy in Informatica but Here I dont know how to use. I have read the Documentation also. But I could not get any result out this.
Right now Im using DataStage 5.2 Version in Windows 2000 O/S.
Can anybody give your openions and solutions to do this? Thanx your help!!!

Regards
Ravi

[tonystark622]

Ravi,

It is relatively easy to read job parameters from a file and run a job, setting the appropriate job parameters with values from a file. This can be accomplished with a Job Sequence job or Batch Job (I can't remember if v5.2 has job sequence jobs or not).

For Job Sequence jobs, I wrote a small routine to read a config file and return a specific job parameter value. I put this in a routine stage in a job sequence job, then set the job parameter value to the return value of this routine in the job activity stage.

For batch jobs you will want to look up DSSetParam and possibly batch jobs in the help files to get you going.

This is a fairly common question on this forum. You may want to check the archives. You should find code samples to do this in the archives.

What are the advantages to running a job from a parameter file, as opposed to the default values of the job parameters that you define in the job? Since this question appears fairly frequently, I suspect that there must be good reasons for doing this, I just wondered what they were.

Hope this helps,
Tony

[dickfong]

We have had tried so in our project. The reason why we wanted a parameter file is because it is very difficult to manage the consistence of parameter name and default value in DataStage projects, especially when there are several projects with several versions on several machines. Of course, that may be another story if tools like parameter manager is used.

Regards,
Dick Fong

[kcbland]

Here is a set of functions written by Stephen Wong of ISSP (Stephen@issp-inc.com) that has been passed around for years that will set all parameters in a job to correspondingly named values in a file. You will need to auto-format these functions when pasting into DS Manager. The next posting will show how to use these two functions

This function, GetParameterArray, will be passed an parameter file in the format:

parameter_name1=value1
parameter_name2=value2
parameter_name3=value3

This function reads the file into an internal array, throwing away lines with leading comment characters.

DEFFUN FileFound(A) Calling 'DSU.FileFound'

FunctionName = "GetParameterArray"
Call DSLogInfo("Loading parameters from file: ":Arg1, FunctionName)
cBlank = ''
cName = 1
cValue = 2

vParamFile = Arg1
aParam = cBlank
vParamCnt = 0
vCurRoutineName = 'GetParameterArray'
vFailed = @FALSE
Done = @FALSE

IF vParamFile AND FileFound(vParamFile) Then
OPENSEQ vParamFile TO hParamFile Then
Loop
READSEQ vLineRaw FROM hParamFile
ON ERROR
Call DSLogWarn('Error from ':vParamFile:'; Status = ':STATUS(),vCurRoutineName)
CLOSE hParamFile
vFailed = @TRUE
Done = @TRUE
End Then
vLine = TRIM(vLineRaw)
vFirstChar = LEFT(vLine,1)
vRemark = LEFT(vLine,4)
IF NOT(vFirstChar = cBlank OR vFirstChar = '#' OR vFirstChar = '*' OR vFirstChar = '"' OR vFirstChar = "'" OR vFirstChar = ';' OR vFirstChar = ':' OR vFirstChar = '[' OR vRemark = 'REM ') THEN
vParamCnt += 1 ; * Add to any parameter array passed as an argument
aParam = TRIM(FIELD(vLine,'=',cName))
aParam = FIELD(vLine,'=',cValue)
END
END ELSE
Done = @TRUE
END
Until Done Do Repeat
CLOSE hParamFile

End Else
Call DSLogWarn('Error from ':vParamFile:'; Status = ':STATUS(),vCurRoutineName)
vFailed = @TRUE
End
End Else
vFailed = @TRUE
End

Call DSLogInfo("Values loaded: ":aParam, FunctionName)
If vFailed Then
Ans = "ERROR"
End Else
Ans = aParam
End


Create a second function, SetJobParameters, with three arguments:

Arg1 = Parameter List ( = parameter names, = parameter values)
Arg2 = Job Handle that parameters will be set for. (Must be attached to the Job/Batch)
Arg3 = Parameter names to be skipped. (Comma separated)

This function is used when actually starting a job up. It takes a job handle and queries the job for its list of parameters. It then matches that list to the internal array of parameter file values and sets the job to them:

$INCLUDE DSINCLUDE JOBCONTROL.H

*------------------*
* Define Functions *
*------------------*

DEFFUN Exists(A) Calling 'DSU.Exists'

*----------------------*
* Initialize Variables *
*----------------------*

Ans = 0 ; * 0=Successful(default), 1=Error occured
ParamList = Arg1
Job = Arg2
ProgName = "Function: SetJobParameters"
SkipParams = Arg3

CONVERT "|" TO @AM IN SkipParams

*------*
* Main *
*------*

IF Exists(ParamList) THEN

IF NOT(Job) THEN
Call DSLogFatal('Could not attach to job (':JobName:')',ProgName)
END

JobParamList = DSGetJobInfo(Job,DSJ.PARAMLIST)

IF Exists(JobParamList) THEN

CONVERT ',' TO @AM IN JobParamList
ParamCnt = DCOUNT(JobParamList,@AM)

FOR P = 1 TO ParamCnt
ParamName = JobParamList
If Exists(SkipParams) Then
LOCATE ParamName IN SkipParams SETTING SkipPos THEN CONTINUE
End

LOCATE ParamName IN ParamList SETTING Pos THEN
ParamValue = ParamList
* Call DSLogInfo('Parameter: ':ParamName:' = ':ParamValue,ProgName)
ErrCode = DSSetParam(Job,ParamName,ParamValue)

IF ErrCode THEN
Ans = 1
BEGIN CASE
CASE ErrCode = DSJE.BADHANDLE
ErrMsg = 'Invalid Job Handle for ':JobName
CASE ErrCode = DSJE.BADSTATE
ErrMsg = 'Job (':JobName:') is not in the right state (compiled, not running)'
CASE ErrCode = DSJE.BADPARAM
ErrMsg = 'ParamName (':ParamName:') is not a known parameter of the job'
CASE ErrCode = DSJE.BADVALUE
ErrMsg = 'ParamValue (':ParamValue:') is not appropriate for ParamName (':ParamName:')'
CASE 1
ErrMsg = 'Problem setting ParamName (':ParamName:'); Value = ':ParamValue
END CASE
Call DSLogWarn('Error [':ErrCode:'], ':ErrMsg,ProgName)
END
* END ELSE
* Ans = 1
* Call DSLogWarn('Parameter(':ParamName:') not passed in parameter list.',ProgName)
END
NEXT P
END
END ELSE
Call DSLogWarn('Missing parameter list ',ProgName)
Ans = 1 ; * Set error flag
END


Kenneth Bland

[kcbland]

In your controlling batch job or job control routine/function, declare the two functions:

DEFFUN GetParameterArray(A) Calling "DSU.GetParameterArray"
DEFFUN SetJobParameters(A, B, C) Calling "DSU.SetJobParameters"


You now can call the GetParameterArray function and read your parameter file:

ParamList = GetParameterArray(ParameterFile) ; * get parameters from the .ini file
If ParamList = "ERROR" Then
Call DSLogFatal("Error occured getting parameter array from file - ":ParameterFile, FunctionName)
End

You can now add parameters not in parameter file to the array ParamList:

ParamList = "ProcessDate":@VM:"BatchNumber"
ParamList = ProcessDate:@VM:BatchNumber

You now have an array of parameters and values, gained from both a file and also from variables you may have derived. Now, as you proceed to start, stop, and monitor jobs, this array will be available to you to use when setting jobs. Wherever you attach to a job and start it, use the following code:

JobName = "whatever your job name is"
hJob = DSAttachJob(JobName, DSJ.ERRFATAL)
ErrCode = SetJobParameters(ParamList, hJob, SkipParameters)
If ErrCode Then
Call DSLogFatal("Error setting parameters for Job - ":JobName, "JobControl")
End
ErrCode = DSRunJob(hJob, DSJ.RUNNORMAL)


I've posted this from job control code that has been passed around for over 4 years, so many hands have had a part it the logic. No matter what, you should get the idea how to go about doing what you need. Here's the two supporting functions that these functions need:

This is function FileFound, which takes an array of fully qualified filenames and verifies that they exist. It works well for verifying a single file:

FileList = Arg1
FileFound = @TRUE
FileCnt = DCOUNT(FileList,@AM)

FOR idx = 1 TO FileCnt UNTIL NOT(FileFound)

OPENSEQ FileList TO hFile ON ERROR STOP "Cannot open file (":FileList:")" THEN
CLOSESEQ hFile
END ELSE
FileFound = @FALSE ;* file not found
EXIT
END
NEXT idx

Ans = FileFound

Here's the Exists function, a handy function for returning a TRUE/FALSE answer to whether a passed variable has value (Not NULL, blank, or a bunch of spaces):

DataIn = TRIM(Arg1)

If IsNull(DataIn) or DataIn = "" Then
Ans = @FALSE
End Else
Ans = @TRUE
End

Good luck!





Kenneth Bland

[nvkravi]

Hi,

Thanq very much to all.I'll try this.

Regards
Ravi

[ray.wurlod]

Earlier this year in Mumbai I implemented a similar function, though it retrieved the parameter values from an Oracle table rather than from a text file, and they had to take into account separate instances of jobs that ran in multi-instance mode.
I don't intend to post that code here, since it was rather specific to site, but rather just to suggest that you can be as flexible as you want to be with this kind of task.

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

[EJRoufs]

Ravi,

It is relatively easy to read job parameters from a file and run a job, setting the appropriate job parameters with values from a file. This can be accomplished with a Job Sequence job or Batch Job (I can't remember if v5.2 has job sequence jobs or not).

For Job Sequence jobs, I wrote a small routine to read a config file and return a specific job parameter value. I put this in a routine stage in a job sequence job, then set the job parameter value to the return value of this routine in the job activity stage.

Hope this helps,
Tony

Tony,

I'm very interested in using your "small routine" you created. I'm writing routines in DataStage for the first time. I'm guessing i should be taking an advanced DataStage class to get me started, but i haven't done so yet (and can't at this time). Could you post your routine out here, or point me in the right direction? I did find where i go within DataStage to create, save, and compile the routine. I've just never done it before. ;>

Thanks!

[tonystark622]

Here's the Routine. I'm sure it could be optimized some, but I haven't had the need or time todate. If you do optimize it, I would apprecite it if you would post the changes back, so that I can improve my code as well.

Code: Select all

FUNCTION GetParameterFromFile(ConfigFileName, ParamName)
********************************************************************************
*
* Name:      GetParameterFromFile
* 
* Arguments: ConfigFileName   - The path and filename of the config file
*            ParamName        - The parameter within the file to retrieve
*
* Expects file with the contents in the form: ParamName=ParamValue
*
* Open file passed in 'ConfigFileName'. Read until finding the line
* where ParamName equals the 'ParamName' argument value passed in.
* Return the ParamValue found.
*
* If the 'ParamName' value passed in was not found in the file
* return @Null
********************************************************************************
Equate TransformName To "GetParameterFromFile"


   * Trim input args and assign to local vars
   ConfigFile = Trim(ConfigFileName, ' ', 'B')
   ParameterName = Trim(ParamName, ' ', 'B')
      

   ParamFound = @False

   * Try to open the file...
   OpenSeq ConfigFile To ConfigFileVar
   On Error
      Msg = 'Error opening parameters file, ' : SQuote(ConfigFile): ', status = ' : Status()
      Call DSLogFatal(Msg, TransformName )
      Abort
   End
   Else
      Msg = 'Cannot open parameters file, ' : SQuote(ConfigFile): ', status = ' : Status()
      Call DSLogFatal(Msg, TransformName )
      Abort
   End

   * We have successfully opened the file
   * Now read the parameter list from the file.
   If FileInfo(ConfigFileVar, 0)
   Then
      Loop
      While ReadSeq Line From ConfigFileVar
         ParName = Trim(Field(Line,"=",1,1))
         ParamValue = Trim(Field(Line,"=",2,1))

         If UpCase(ParName) = UpCase(ParameterName) Then
            ParamFound = @True
            Exit
         End
         
      Repeat
      CloseSeq ConfigFileVar
   End

   If ParamFound Then
      Ans = ParamValue
   End Else
      Ans = @Null
      Msg = 'Cannot find parameter ' : SQuote(ParamName): ', status = ' : Status()
      Call DSLogFatal(Msg, TransformName )

   End
RETURN(Ans)

If you call this routine from a Routine Activity in a Job Sequencer job, it will return the value of the parameter. You put this RoutineStageName.$ReturnValue as the Job Parameter Value on a Job Activity stage later in the job.

The file that you're reading should be in the format:
ParameterName1=ParameterValue2
ParameterName2=ParameterValue2

Good Luck,
Tony

[EJRoufs]

Here's the Routine. I'm sure it could be optimized some, but I haven't had the need or time todate. If you do optimize it, I would apprecite it if you would post the changes back, so that I can improve my code as well.............
.
.
.
Good Luck,
Tony

Thanks, Tony! I'll give it a shot. I'm definitely not at the expertise "level" yet to be optimizing this, but if i get to that level, i'll definitely give you anything i come up with. :>

[sumitgulati]

Hi Ravi,

Earlier this year we had exactly the same requirement. We were replacing Informatica mapping by a DataStage job and below is the routine we used to read the parameter file.

Code: Select all

ParamFile = ParameterFileName

Var = 0



OpenSeq ParamFile To FileVar Else

Call DSLogFatal("Cannot open ":ParamFile: "- Aborting Job", "ParameterRoutine")

Ans = @Null

GoTo ErrorExit

End

Loop

ReadSeq FileLine From FileVar

Then

If Var = 1 Or TrimF(TrimB(Field(EReplace(FileLine, '[',""),']',1))) = TrimF(TrimB(JobName))

Then

Var = 1

If Var = 1 And TrimF(TrimB(Field(FileLine,"=",1))) = TrimF(TrimB(ParameterName))

Then

Ans = TrimF(TrimB(Field(FileLine,"=",2)))

GoTo MatchFound

End Else

Ans = @Null

End

End Else

Ans = @Null

End

End Else

Exit

End

Repeat

MatchFound:

CloseSeq FileVar

ErrorExit:

ParameterFileName, ParameterName, JobName are the input parameters to the routine. ParameterFileName should be given with the entire path.

Hope this works.

Regards,
Sumit