splitting files

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
kavuri
Premium Member
Premium Member
Posts: 161
Joined: Mon Apr 16, 2007 2:56 pm

splitting files

Post by kavuri »

Hi,
here I am having millions of records in DB2. I need to transform them and pass into text files. I want to split the flat files such that each contains 50,000 records. Is there any way to do. If so how can we name the files?

Thanks in advance.

Kavuri
iDomz
Participant
Posts: 81
Joined: Wed Jul 25, 2007 5:25 am
Location: London

Post by iDomz »

you can use unix split command.

split -l 10000 -a 3 filename 'splitfile.'

will split filename into multiple files of 10000 lines each, each named
splitfile.aaa, splitfile.aab etc..

There could be a ds way of doing this, but I am not upto it yet.

Cheers,
D
bcarlson
Premium Member
Premium Member
Posts: 772
Joined: Fri Oct 01, 2004 3:06 pm
Location: Minnesota

Post by bcarlson »

The file stages can export to a fileset. A big benefit is that the files are created concurrently so exporting 1 million records into 10 files should be a fraction of the time of exporting 1 million records into 1 file.

Check the online documentation for how to do this with the file stages. I think both the sequential file stage and complex file stages can do this. But check the docs to be sure.

Brad.
Post Reply