Spiltting a huge file into multiple small files

avats · Post by **avats** » Thu May 05, 2011 9:40 pm

Hi,

We have a requirement where we need to split the data into multiple small XML files with around 1000 rows. We can do this based on the rowcount but we are not sure which stage to be used to split the rows in multiple files.

Earlier we were writing all the data to a single file but the job is failing due to space issue on the server, so we need to split the data.

Please help.

Thanks,
Avats

chulett · Post by **chulett** » Thu May 05, 2011 9:52 pm

Space is space, no matter if it's all in one file or many... perhaps you meant a file size issue? Regardless, what you want to take advantage of is the Trigger Column in the XML Output stage.

Every time the value in the column changes, the output file is closed and a new one is opened. It doesn't need to be a column you pass to the output field so you could simply count the output rows and change the value in the trigger column every 1000 rows.

vishal_rastogi · Post by **vishal_rastogi** » Fri May 06, 2011 12:04 am

you can use the unix script to split the file using the sed command and later load the spiltted file to target

ray.wurlod · Post by **ray.wurlod** » Fri May 06, 2011 2:14 am

I'm with Craig. 15GiB of data in one file is still going to occupy 15GiB in multiple files.

udayk_2007 · Post by **udayk_2007** » Fri May 06, 2011 5:26 am

unix split command can be used for this purpose.

Regards
Ulhas

MarkB · Post by **MarkB** » Fri May 06, 2011 5:35 am

I don't think coming up with ways to split the file will accomplish anything other than complicating the job and getting the same results.

The issue the OP has is one of disk space. You can split the file into hundreds of smaller files and it won't matter one iota if you are still writing them out to the same disk - eventually you are going to run out of space.

What the OP needs is another disk to write to or to make enough space on the disk he is currently writing to in order to run the job.