Sequentioal File

dudam · Post by **dudam** » Fri Mar 30, 2012 6:09 am

Hi to all

I want to load 10 million records in to Sequential File ...
how many ways we load and which way can take very less time
(which options are available for above scenario in seq file)

Plz help me any one

Thanks in Advance

:D

jwiles · Post by **jwiles** » Fri Mar 30, 2012 7:38 am

What is your data source?

You could place a column export stage in front of the sequential file stage to export your columns into one varchar column with the proper column delimiters (assuming your seqfile is text only), then write the one column with the sequential file stage using the proper record delimiter.

Regards,

qt_ky · Post by **qt_ky** » Fri Mar 30, 2012 11:30 am

10 million records in a sequential file may take 4 seconds. How much less time do you need?

jwiles · Post by **jwiles** » Fri Mar 30, 2012 11:53 am

And of course the other question to ask is: What have you tried?

Sequential File stage alone may provide the performance needed to meet your business requirements/SLAs. The only benefit of my earlier suggestion is if your file is quite large in terms of volume (number of bytes)--it could parallelize the conversion of your data into the output format, whereas Sequential File stage would do so only sequentially.

Regards,

Mike · Post by **Mike** » Fri Mar 30, 2012 1:14 pm

Your sequential write speed is primarily going to be governed by your disk hardware... if you're not keeping the output buffer full, then you need to look at upstream processing... otherwise it's a matter of upgrading to hardware with a higher data transfer rate.

Mike

ray.wurlod · Post by **ray.wurlod** » Fri Mar 30, 2012 2:57 pm

Sequential File stage actually uses STREAMS I/O, so should not be much of a bottleneck in the export operator. As noted, the problem may be in your disk system itself.

qt_ky · Post by **qt_ky** » Sat Mar 31, 2012 6:22 am

OK, then, try out your job, and if it doesn't finish in 2 seconds, let us know.