How to Write a Job to Hanlde Thousand Files in Several Runs
Posted: Tue Jan 22, 2013 2:03 pm
Hi there,
how can I write a job to handle, like to say, thousand files in a folder. We don't want to process all these files in one batch, but want to each run to limit to process 10 million records in these files. Suppose the total records of these files exceed 100 millions.
With this job, two we need to solve:
1. Count records in each file
2. Pick files for each run to guarantee that these files contain arround 10 million reecords, and each file has to be processed one and only one time.
Thanks,
how can I write a job to handle, like to say, thousand files in a folder. We don't want to process all these files in one batch, but want to each run to limit to process 10 million records in these files. Suppose the total records of these files exceed 100 millions.
With this job, two we need to solve:
1. Count records in each file
2. Pick files for each run to guarantee that these files contain arround 10 million reecords, and each file has to be processed one and only one time.
Thanks,