Multiple files based on number
Moderators: chulett, rschirm, roy
Multiple files based on number
Hello
I have a requirement where lets say i have a flat file with 200 records and after doing some transformations i need to produce a file each for every 50 records. So i will have to produce 4 files with 50 records each. If the input file has 60 records i need to produce 2 files with 1 file 50 records and the second with 10 records. Is there a way that this can be done through datstage.
Thanks
I have a requirement where lets say i have a flat file with 200 records and after doing some transformations i need to produce a file each for every 50 records. So i will have to produce 4 files with 50 records each. If the input file has 60 records i need to produce 2 files with 1 file 50 records and the second with 10 records. Is there a way that this can be done through datstage.
Thanks
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Yes. Create sufficient output links to handle the worst case scenario, and filter on row number (ideally row number from original source). Run in sequential mode.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Re: Multiple files based on number
Hi,
If predicting the number of branches you might need is tough, you can try this:
Create a Wrapped stage (Custom stage)
You can call a Unix script that will partition the data and write
to multiple files.
This is just an idea.
I will also try this for learning purpose and post the result.
If predicting the number of branches you might need is tough, you can try this:
Create a Wrapped stage (Custom stage)
You can call a Unix script that will partition the data and write
to multiple files.
This is just an idea.
![Idea :idea:](./images/smilies/icon_idea.gif)
I will also try this for learning purpose and post the result.
Hi,dodda wrote:Hi
thanks for your response. Yes the input file might have any number of records but i need to produce a file for every 50 records.I have never created custom stages before. Is there a way other than creating custom stages.
Thanks
Once you have the unix script for the purpose, implementing is not difficult.
or instead of a stage, you can land the data in an intermediary sequential file and call the script once the job is complete.
If you go that route, the UNIX command "split" can be used to chunk up the full file into smaller files and then you may need to loop through the results and rename the files unless you can live with the naming convention the command uses.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers