Processing in input data in batch wise

monaz · Post by **monaz** » Wed Jul 15, 2009 12:24 am

Hi Have a requirement in which i need to proceess the input records based onbatches of 100,000 records to the final output file.. This requirement is because to resolve performance issue..

Please suggest..

As it has 15month's old record so we extracting all the transaction records and limiting in the final output file to process 100,000 records per day

ArndW · Post by **ArndW** » Wed Jul 15, 2009 1:04 am

Do you mean that you wish to process your input in batches of 100,000 or that you wish to parcel out your output to files limited to 100,000 lines each? If you create one output file you could use the UNIX command "split" to effect that.

monaz · Post by **monaz** » Wed Jul 15, 2009 4:38 am

ArndW wrote:Do you mean that you wish to process your input in batches of 100,000 or that you wish to parcle out your output to files limited to 100,000 lines each? If you create one output file you could use th ...

Thanks for the quick response.

i mean i wish to process your input in batches of 100,000. Could you please help me out in achieving this task

ArndW · Post by **ArndW** » Wed Jul 15, 2009 5:27 am

I assume your input file is bigger than 100,000 rows. Isn't doing several runs, each processing 100,000 [different] rows to separate output files the same as doing one run and then splitting the output into chunks of 100,000? The latter would be so much easier and less susceptible to errors.

chulett · Post by **chulett** » Wed Jul 15, 2009 5:56 am

Monaz, please - why are you starting over again on the same subject as your previous post?