Split Data into different files
Posted: Fri Mar 20, 2009 9:28 am
Hi All
I am trying to achieve a new logic to produce flat files. The current design of the job extracts the data from Oracle Database around 150 Million records and then loads into a sequential file for the business to use for reporting. I tried to load the complete 150 Million into the Sequential file but I get the following error.
INVOICE_XRF_ACTL_ORD_Load,0: Output file full
I think sequential file has only 2GB of memroy limit. I want to know if there is any better way to split the file per size and not per rows (transformer stage constraint). I want to creat another Flat file in the output directory as soon as the current output file reaches the memory limit. Example the job runs and produces approx 10 million records in file1 as soon as it reaches memory limit another sequential file2 should be created. Similarly then file3, file4, file5 and so on are created until the complete data is saturated in the job.
Please Advice
Thanks
I am trying to achieve a new logic to produce flat files. The current design of the job extracts the data from Oracle Database around 150 Million records and then loads into a sequential file for the business to use for reporting. I tried to load the complete 150 Million into the Sequential file but I get the following error.
INVOICE_XRF_ACTL_ORD_Load,0: Output file full
I think sequential file has only 2GB of memroy limit. I want to know if there is any better way to split the file per size and not per rows (transformer stage constraint). I want to creat another Flat file in the output directory as soon as the current output file reaches the memory limit. Example the job runs and produces approx 10 million records in file1 as soon as it reaches memory limit another sequential file2 should be created. Similarly then file3, file4, file5 and so on are created until the complete data is saturated in the job.
Please Advice
Thanks