Hi
How can I create target files on the fly in datastage MVS job?
Example:
Source - Raw data (key Customer ID)
lookup - Customer table (key Customer ID)
target - dynamic (dynamically create fixed width files depending on the number of customers matched in lookup)
Your help in this case will appreciated. Thanks.
Regards
Rupesh
Creating Files on the fly
Moderators: chulett, rschirm, roy
Please clarify your requirement?
1) Are you asking if you match, for example, 5 customers with your lookup, you want to create 5 output files?
or
2) Are you asking if you match 5 customers with your lookup, you want to create 1 output file with an array of 5 customers?
Mainframe jobs have very simple and limited transformation capabilities. There really is nothing "dynamic" in the sense that you seem to be looking for.
Mike
1) Are you asking if you match, for example, 5 customers with your lookup, you want to create 5 output files?
or
2) Are you asking if you match 5 customers with your lookup, you want to create 1 output file with an array of 5 customers?
Mainframe jobs have very simple and limited transformation capabilities. There really is nothing "dynamic" in the sense that you seem to be looking for.
Mike
Well there is nothing dynamic about the number of output files...
You would have to design the job with the maximum number of output files that would be needed and then use output constraints to route individual rows to the proper output file.
Of course this is only feasible with a reasonable number of possible output files.
Mike
You would have to design the job with the maximum number of output files that would be needed and then use output constraints to route individual rows to the proper output file.
Of course this is only feasible with a reasonable number of possible output files.
Mike
Mike
Well, there are limitations of the tool as there are less options on transformation. This is what I am doing to resolve the issue. Hopefully it should work.
At the end of the day, the datastage job is a cobol program. For it to run requires jcl and compile is compile jcl.
I am developing a datastage job to generate 3 output files. This files are dynamically generated text files (cobol and jcl) depending on the number of customers or customer count I retrieve from customer table.
This is what the jobs are
job1: write looked up and source data in one output file.
job2:
source - count from customer table
business rule stage - to loop the cobol section that many number of times.
outout - fixed width flat file
I would then upload this output files to the related libraries and then run them as any other cobol program.
This cobol program will read file generated in job 1 and create different files for different customers.
I dont know if I am explaining this correctly, but if you need more explaination let me know.
Thanks.
Regards
Rupesh
Well, there are limitations of the tool as there are less options on transformation. This is what I am doing to resolve the issue. Hopefully it should work.
At the end of the day, the datastage job is a cobol program. For it to run requires jcl and compile is compile jcl.
I am developing a datastage job to generate 3 output files. This files are dynamically generated text files (cobol and jcl) depending on the number of customers or customer count I retrieve from customer table.
This is what the jobs are
job1: write looked up and source data in one output file.
job2:
source - count from customer table
business rule stage - to loop the cobol section that many number of times.
outout - fixed width flat file
I would then upload this output files to the related libraries and then run them as any other cobol program.
This cobol program will read file generated in job 1 and create different files for different customers.
I dont know if I am explaining this correctly, but if you need more explaination let me know.
Thanks.
Regards
Rupesh
I'm either really confused about what you're trying to accomplish, or you're confused about working with mainframe jobs. No, the generated code has nothing to do with your data. It is strictly based on job design and gets produced when you hit the "Generate Code" button.This files are dynamically generated text files (cobol and jcl) depending on the number of customers or customer count I retrieve from customer table.
No matter what active stage you use (and no matter how many times you loop in a business rule stage), the number of output files is governed by the number of fixed width flat file stages sitting on the canvas.source - count from customer table
business rule stage - to loop the cobol section that many number of times.
outout - fixed width flat file
Mike
Mike
I think I did not explain it correctly. Following is the problem and its solution.
Problem:
.................... Customer table
..............................|
..............................|
..............................V
Raw data -----> lookup -----> Files (Multiple)
Source is raw data with one of the column as customer id. The records from source are looked up against the Customer table using customer id key. Output is fixed width files, one for each customer.
Requirement: separate file for each customer
Complications:
In MVS, one cannot, on the fly, decide how many files will be generated at the output. It is fixed in the job.
Solution:
1. In this case, one has to find the maximum number of files that may be created during the run and get those many file stages on the DS designer canvas.
Drawback: Empty files created if the customers are less. Customers are lost if their count exceeds the number of files.
2. Get the entire data into one file and write an External routine to do the splitting of the file.
Drawback: No expertise in Cobol/PL1 to write External Routine
3. Get the entire data into one file and have a Cobol program that splits the file into number of customers stored in the file.
Step1: create a ds job with one input file and one output file with a transformer doing nothing but straight mapping.
Step2: Generate code. This will give test.cob, test.run and test.cmp.
Step3: Modify the job to include another output file. Generate code to get test1.cob, test1.run and test1.cmp.
Step4: Compare the two generated codes, test.cob and test1.cob. Look for Section of the code that accommodates the new file introduced. In all, there are 6 small sections that get amended with the new introduced file.
Step5: If I get the count of customers in customer table, I can repeat this 6 section that many times in a text file. To do this I create another ds job that gets a count from customer table followed by a business rule stage. In Business rule stage there are Constants for Cobol section that do not change and Variables for Cobol section that will change depending on the number of customers. The output is written to a fixed width file that has only one column and is of Varchar datatype. The generated text is a Cobol program that has exact number of files as that of the customer.
Step6: Similar is followed for run JCL. Compile JCL does not change.
Step7: upload this two output files on mainframe and run then as any other generated code from ds job.
Step8: it does work.
Hope, I explained it thoroughly now. Thanks.
Regards
Rupesh
I think I did not explain it correctly. Following is the problem and its solution.
Problem:
.................... Customer table
..............................|
..............................|
..............................V
Raw data -----> lookup -----> Files (Multiple)
Source is raw data with one of the column as customer id. The records from source are looked up against the Customer table using customer id key. Output is fixed width files, one for each customer.
Requirement: separate file for each customer
Complications:
In MVS, one cannot, on the fly, decide how many files will be generated at the output. It is fixed in the job.
Solution:
1. In this case, one has to find the maximum number of files that may be created during the run and get those many file stages on the DS designer canvas.
Drawback: Empty files created if the customers are less. Customers are lost if their count exceeds the number of files.
2. Get the entire data into one file and write an External routine to do the splitting of the file.
Drawback: No expertise in Cobol/PL1 to write External Routine
3. Get the entire data into one file and have a Cobol program that splits the file into number of customers stored in the file.
Step1: create a ds job with one input file and one output file with a transformer doing nothing but straight mapping.
Step2: Generate code. This will give test.cob, test.run and test.cmp.
Step3: Modify the job to include another output file. Generate code to get test1.cob, test1.run and test1.cmp.
Step4: Compare the two generated codes, test.cob and test1.cob. Look for Section of the code that accommodates the new file introduced. In all, there are 6 small sections that get amended with the new introduced file.
Step5: If I get the count of customers in customer table, I can repeat this 6 section that many times in a text file. To do this I create another ds job that gets a count from customer table followed by a business rule stage. In Business rule stage there are Constants for Cobol section that do not change and Variables for Cobol section that will change depending on the number of customers. The output is written to a fixed width file that has only one column and is of Varchar datatype. The generated text is a Cobol program that has exact number of files as that of the customer.
Step6: Similar is followed for run JCL. Compile JCL does not change.
Step7: upload this two output files on mainframe and run then as any other generated code from ds job.
Step8: it does work.
Hope, I explained it thoroughly now. Thanks.
Regards
Rupesh
I totally agree with you Mike.
I have been working on DS MVS for last 3 years and I would not recommend anyone to change the generated code (unless you want to pass a parameter from run JCL).
This was one exceptional case which required this sort of altering. (I would not mind as far as it does what its meant to do)
Anyways, thanks for your help. Much appreciated.
Regards
Rupesh
I have been working on DS MVS for last 3 years and I would not recommend anyone to change the generated code (unless you want to pass a parameter from run JCL).
This was one exceptional case which required this sort of altering. (I would not mind as far as it does what its meant to do)
Anyways, thanks for your help. Much appreciated.
Regards
Rupesh