how to beyond the filelimit in Unix using sequence file?

[donhoff]

Hi, I developed a parallel job which extracted a huge amount of data and wrote them into a sequence file. But the file was so huge that it was beyond the unix file limit(2G).

So I want to write data in a list of files, such as filename_01,filename_02,.... The number of files depends upon the data.

Could anyone tell me how to realize this function in DataStage EE?
Thanks!

[Ragunathan Gunasekaran]

You can use fileset stage instead of sequential file stage . The fileset stage has the option of overcoming the file size limit.

[donhoff]

Yes, I tried to use fileset. But Fileset produced only 6 files for I have 6 partitions. It could not produce more files and so the job failed. All 6 files reached the 2G limit but there are still data not written to file.

Is there any other solution? For example , some way that can produce 2G data files according to the actual data volume?

Thanks!

[Raghumreddy]

Hi
Contact your Unix admin on the file size . they can increase the limit on the file system that you are using.

HTH
Raghu M

Hi, I developed a parallel job which extracted a huge amount of data and wrote them into a sequence file. But the file was so huge that it was beyond the unix file limit(2G).

So I want to write data in a list of files, such as filename_01,filename_02,.... The number of files depends upon the data.

Could anyone tell me how to realize this function in DataStage EE?
Thanks!

[donhoff]

Hi
Contact your Unix admin on the file size . they can increase the limit on the file system that you are using.

HTH
Raghu M

Hi, I developed a parallel job which extracted a huge amount of data and wrote them into a sequence file. But the file was so huge that it was beyond the unix file limit(2G).

So I want to write data in a list of files, such as filename_01,filename_02,.... The number of files depends upon the data.

Could anyone tell me how to realize this function in DataStage EE?
Thanks!

OK, Thanks! This is a good suggestion. Do you know what is the max file size for a HP9000 Unix Server and a IBM AIX5.3 p570 Server?
Thanks!

[Raghumreddy]

Default max size is 2G . your admin can make it what ever you want on a partition

or they can create a file on multiple nodes that you can use for placing huge volume of data

HTH
Raghu M

[donhoff]

Default max size is 2G . your admin can make it what ever you want on a partition

or they can create a file on multiple nodes that you can use for placing huge volume of data

HTH
Raghu M

Hi, you said my admin can make it what ever I want on a partition. Does that mean the actual max file size can only be limited by the disk space?

Someone told me that a HP-Unix or AIX can only have a max file size of 128G. Even you have many Ts of disk space, you could not produce a file beyond 128G. Is that true? This is very important for me.

Because for my estimating, the single file I extracted from DB is at least 300G

[Raghavendra]

I have seen a file of 210GB on AIX server and so I belive a file size of max 120GB might not be true.

[donhoff]

OK,Thanks very much

[chulett]

Ask your SysAdmin what the limits are.