Dataset Limitations

coehcl · Post by **coehcl** » Mon Nov 27, 2006 8:53 am

Hi,

In our process,First job will load approx. 4-5 GB of data in a dataset.And in the second job, i have to use the above data set as source.
Will Dataset handle that much data or any limitations to that ?

Thanks in Advance,
Kishore

Nageshsunkoji · Post by **Nageshsunkoji** » Mon Nov 27, 2006 9:18 am

Hi,

The amount of data that can be stored in each destination data file is limited by the characteristics of the file system and the amount of free disk space available.

In your case, there is no problem until and unless, if there is an issue with space. You can access that Dataset as a source in other job also.

ArndW · Post by **ArndW** » Mon Nov 27, 2006 9:27 am

I'm pretty sure that PX datasets and filesets avoid any 2Gb limit your filesystem might have by adding partfiles; but I'm not absolutely certain. If you can create files larger than 2Gb in your OS then there is no effective limit other than disk space.

ray.wurlod · Post by **ray.wurlod** » Mon Nov 27, 2006 2:27 pm

Even if your operating system limits file sizes to 2GB there is no effective limit. Data Sets' data will be stored in multiple files per partition. So, if you had 10000 files per partition and 1000 partitions, that's 2.0E9 GB of data. And you're allowed more than that; I only chose those numbers for illustrative purposes.