Job Looping
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 74
- Joined: Tue Sep 30, 2003 4:25 am
- Location: Malaysia
Job Looping
Hallo all,
I would need your suggestion on my situation here.'
I have a hash file which contains 2 columns, code and description.
I have another job, Job A which reads from a sequential file n write to different file depending on the code value in the hash file.
Which means, if i have 5 records in the hash file, I would like to loop Job A 5 times and write to 5 different files. I can use job parameter to set the naming of different target sequential files.
But I am not sure how to loop the Job A..
Would appreciate suggestion from you.
Thank you
I would need your suggestion on my situation here.'
I have a hash file which contains 2 columns, code and description.
I have another job, Job A which reads from a sequential file n write to different file depending on the code value in the hash file.
Which means, if i have 5 records in the hash file, I would like to loop Job A 5 times and write to 5 different files. I can use job parameter to set the naming of different target sequential files.
But I am not sure how to loop the Job A..
Would appreciate suggestion from you.
Thank you
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Don't use a loop at all.
Use one job in which the Transformer stage has five outputs, writing to the five different files. Use constraints based on the return value from the lookup to determine which rows are sent along which links.
This way you only need to process your source data once. This will yield the shortest completion time for your processing.
Use one job in which the Transformer stage has five outputs, writing to the five different files. Use constraints based on the return value from the lookup to determine which rows are sent along which links.
This way you only need to process your source data once. This will yield the shortest completion time for your processing.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 74
- Joined: Tue Sep 30, 2003 4:25 am
- Location: Malaysia
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If the maximum number of paths you will have is finite and manageably small, create that many output links. It doesn't matter if no rows are passed along any particular output link.
In your plan, you want to process the same data N times, where N is the number of values in the hashed file. This is necessarily inefficient. You should only process data once.
The main problem with an arbitrary number of outputs is that you have to have some method for generating that number of output file names. If the number is not manageably small, then you probably are better off creating a routine that can keep track of which of these files are open, opening more if required, and actually writing to the files from within the routine. An after-job routine should be used to close all the open files.
Some necessary error handling has been omitted for the sake of clarity.
(Yes, it is permissible to have more than one WHILE statement in a loop.)
In your plan, you want to process the same data N times, where N is the number of values in the hashed file. This is necessarily inefficient. You should only process data once.
The main problem with an arbitrary number of outputs is that you have to have some method for generating that number of output file names. If the number is not manageably small, then you probably are better off creating a routine that can keep track of which of these files are open, opening more if required, and actually writing to the files from within the routine. An after-job routine should be used to close all the open files.
Code: Select all
FUNCTION WriteNFiles(Line, HashResult)
* Writes Line to file FileHashResult
$INCLUDE UNIVERSE.INCLUDE FILEINFO.H
* Change this constant to be able to handle more files, do not exceed NFILE.
Equate MAXFILES To 100
* File variables are in COMMON so the files are not closed on return.
COMMON /WriteNFiles/FileVar(MAXFILES), FileInUse, FileNames, LineCount(MAXFILES)
If FileInUse = 0
Then
FileInUse = "" ; * dynamic array
FileNames = "" ; * dynamic array
End
* Determine whether file is open already, open it if not.
FileName = "File" : HashResult
Locate FileName In FileNames By "AL" Setting WhereFound
Else
Ins FileName Before FileNames<WhereFound>
OpenSeq FileName To FileVar(WhereFound)
Then
FileInUse<WhereFound> = 1
WeofSeq FileVar(WhereFound) ; * truncate file
End
End
* If the file is open, then write the line to the file.
If FileInfo(FileVar(WhereFound))
Then
WriteSeq Line To FileVar(WhereFound)
Then
LineCount(WhereFound) += 1
End
End
Ans = 0
RETURN(Ans)
Code: Select all
SUBROUTINE CloseNFiles(InputArg, ErrorCode)
* Close the files opened by WriteNFiles
ErrorCode = 0 ; * set to a non-zero value to stop job execution
COMMON /WriteNFiles/FileVar(MAXFILES), FileInUse, FileNames, LineCount(MAXFILES)
SetRem 0 On FileInUse
WhereFound = 0
Loop
Remove Flag From FileInUse Setting MoreFlags
While Flag
WhereFound += 1
CloseSeq FileVar(WhereFound)
While MoreFlags
Repeat
RETURN
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 74
- Joined: Tue Sep 30, 2003 4:25 am
- Location: Malaysia
Hallo Ray,
There are about 30 output files that I would need to create each time. Therefore, it is not feasible to create multiple output links.
I appreciate your solution to this problem. However, as I am not familiar with using BASIC all these while, I don't really understand your solution.
Firstly, how would I call this function , WriteNFiles in the job?
Secondly, wat does this input 'Line' and 'HashResult' in 'WriteNFiles' function comes from? Should HashResult be a hash file ... an array of hash result? How do I get it?
Would it be easier if i append all the results in a file instead of creating multiple files?
There are about 30 output files that I would need to create each time. Therefore, it is not feasible to create multiple output links.
I appreciate your solution to this problem. However, as I am not familiar with using BASIC all these while, I don't really understand your solution.
Firstly, how would I call this function , WriteNFiles in the job?
Secondly, wat does this input 'Line' and 'HashResult' in 'WriteNFiles' function comes from? Should HashResult be a hash file ... an array of hash result? How do I get it?
Would it be easier if i append all the results in a file instead of creating multiple files?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
First off, 30 does fall into the "small and manageable" category. I'd be happy to have a Transformer stage with 30 outputs, given that only one of them is triggered for any input row.
WriteNFiles would be called in an output column derivation expression. Argument Line is the line to be written to the file (which you can construct in the same expression) and HashResult is the result of the hashed file lookup, which is an Input Column in the derivation.
If you're appending everything in one file then you're not fulfilling your original design requirement. That's not to say it's an invalid solution, but how then do you split the single output file into 30 files? Are you familiar with awk or sed (either could do it)?
WriteNFiles would be called in an output column derivation expression. Argument Line is the line to be written to the file (which you can construct in the same expression) and HashResult is the result of the hashed file lookup, which is an Input Column in the derivation.
If you're appending everything in one file then you're not fulfilling your original design requirement. That's not to say it's an invalid solution, but how then do you split the single output file into 30 files? Are you familiar with awk or sed (either could do it)?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 74
- Joined: Tue Sep 30, 2003 4:25 am
- Location: Malaysia
Hallo Ray,
Thank you for your suggestion. I decided to use unix scripting (sed) to append 30 sets of records into a common file and update the category for each set of records at the same time.
Thank you for your suggestion. I decided to use unix scripting (sed) to append 30 sets of records into a common file and update the category for each set of records at the same time.
Code: Select all
#!/bin/ksh
szPath=$1
szOriginalFile=$2
szClientVarTemplate=$3
szTargetFile=$4
echo "szPath = $szPath"
echo "szOriginalFile = $szOriginalFile"
echo "szTargetFile = $szTargetFile"
if test ! -s $szPath$szTargetFile
then
echo "$szPath$szTargetFile is empty or does not exist"
echo "No deletion of file is done"
else
echo "$szPath$szTargetFile already exists.Deletion of file is done"
rm $szPath$szTargetFile
fi
for fieldfile1 in `cat $szPath$szClientVarTemplate`
do
echo $fieldfile1
sed "s/EXMT/$fieldfile1/g" $szPath$szOriginalFile >> $szPath$szTargetFile
done