DSXchange

Hi, I developed a parallel job which extracted a huge amount of data and wrote them into a sequence file. But the file was so huge that it was beyond the unix file limit(2G).

So I want to write data in a list of files, such as filename_01,filename_02,.... The number of files depends upon the data.

Could anyone tell me how to realize this function in DataStage EE?
Thanks!

You can use fileset stage instead of sequential file stage . The fileset stage has the option of overcoming the file size limit.

Yes, I tried to use fileset. But Fileset produced only 6 files for I have 6 partitions. It could not produce more files and so the job failed. All 6 files reached the 2G limit but there are still data not written to file.

Is there any other solution? For example , some way that can produce 2G data files according to the actual data volume?

Thanks!

Hi
Contact your Unix admin on the file size . they can increase the limit on the file system that you are using.

HTH
Raghu M

donhoff wrote:Hi, I developed a parallel job which extracted a huge amount of data and wrote them into a sequence file. But the file was so huge that it was beyond the unix file limit(2G).

So I want to write data in a list of files, such as filename_01,filename_02,.... The number of files depends upon the data.

Could anyone tell me how to realize this function in DataStage EE?
Thanks!

Raghumreddy wrote:Hi
Contact your Unix admin on the file size . they can increase the limit on the file system that you are using.

HTH
Raghu M

donhoff wrote:Hi, I developed a parallel job which extracted a huge amount of data and wrote them into a sequence file. But the file was so huge that it was beyond the unix file limit(2G).

So I want to write data in a list of files, such as filename_01,filename_02,.... The number of files depends upon the data.

Could anyone tell me how to realize this function in DataStage EE?
Thanks!

OK, Thanks! This is a good suggestion. Do you know what is the max file size for a HP9000 Unix Server and a IBM AIX5.3 p570 Server?
Thanks!

Default max size is 2G . your admin can make it what ever you want on a partition

or they can create a file on multiple nodes that you can use for placing huge volume of data

HTH
Raghu M

Raghumreddy wrote:Default max size is 2G . your admin can make it what ever you want on a partition

or they can create a file on multiple nodes that you can use for placing huge volume of data

HTH
Raghu M

Hi, you said my admin can make it what ever I want on a partition. Does that mean the actual max file size can only be limited by the disk space?

Someone told me that a HP-Unix or AIX can only have a max file size of 128G. Even you have many Ts of disk space, you could not produce a file beyond 128G. Is that true? This is very important for me.

Because for my estimating, the single file I extracted from DB is at least 300G

I have seen a file of 210GB on AIX server and so I belive a file size of max 120GB might not be true.

OK,Thanks very much

Ask your SysAdmin what the limits are.