How do I use multiple inputs to output to a single CSV? Do I use the Row Merge stage or just the Merge stage, something else? I've been experimenting and I can't get anything to work.
For manageability I have split a dozen database queries that I want to union into a single CSV file.
Multiple Inputs to one CSV
Moderators: chulett, rschirm, roy
Re: Multiple Inputs to one CSV
You can use funnel stage if all the input files records have to be stacked up in a csv file.(if metadata are same) If its a column wise addition you can use join stage or merge stage or look up stage
Thanks & Regards!
BJ
BJ
I'm on the server edition, I don't have the funnel stage available to me. I think the Merge Stage might be the right track but I'm confused that it doesn't have any input links, only output?
http://pic.dhe.ibm.com/infocenter/iisin ... tages.html
http://pic.dhe.ibm.com/infocenter/iisin ... tages.html
Hi,
If I understood correctly,
you have severl source files that need to eventually be written to a single sequential files with the CSV format.
So there are several solutions available, depending on the similarity or lack of, between your source files.
Writing to a sequential files simultaniously from more then 1 process/link at the same time should not work or will not work correctly -even if you managed to produce such a job.
Scenario 1 - option 1:
* All sopurce files have the same definition
- use filter command to TYPE/cat all the files to a single input streem
TYPE - for windows server ; cat for unix/linux server.
Scenario 2 - option 2:
* Source files have different definition (table definition - beyond column names)
- Read all files seperatly in the same job, use transformer to
get them to the same output file format needed for the CSV target file
- use link collector stage to collect the data using the round robin method
and output the collected data to the target CSV file.
You can also use option 2 in the Scenario 1 example as well.
P.S.
It wouldn't surprise me if there are other solutions available,
theses are 2 from the top of my head.
IHTH (I Hope This Helps),
If I understood correctly,
you have severl source files that need to eventually be written to a single sequential files with the CSV format.
So there are several solutions available, depending on the similarity or lack of, between your source files.
Writing to a sequential files simultaniously from more then 1 process/link at the same time should not work or will not work correctly -even if you managed to produce such a job.
Scenario 1 - option 1:
* All sopurce files have the same definition
- use filter command to TYPE/cat all the files to a single input streem
TYPE - for windows server ; cat for unix/linux server.
Scenario 2 - option 2:
* Source files have different definition (table definition - beyond column names)
- Read all files seperatly in the same job, use transformer to
get them to the same output file format needed for the CSV target file
- use link collector stage to collect the data using the round robin method
and output the collected data to the target CSV file.
You can also use option 2 in the Scenario 1 example as well.
P.S.
It wouldn't surprise me if there are other solutions available,
theses are 2 from the top of my head.
IHTH (I Hope This Helps),
Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org