Page 1 of 1
Amount of data a dataset can handle
Posted: Thu Aug 26, 2004 10:50 pm
by lakshmipriya
We have a data set which has the following columns and data type
The structure of the data set is
Dset1
C1 char(40)
C2 smallint
C3 char(6)
C4 decimal(9,2)
C5 integer
C6 decimal(9,2)
C7 integer
C8 decimal(9,2)
C9 integer
This data set will have 150 million rows so how much will be the size of the Dataset.
Will the datastage can handle the size of the data set?
Posted: Thu Aug 26, 2004 11:51 pm
by richdhan
Hi Lakshmi,
Pls look into this post and check if that helps
viewtopic.php?t=88791
According to the post they have a dataset which can hold 4.5GB of data.
Lakshmi wrote:This data set will have 150 million rows so how much will be the size of the Dataset.
I think Dataset Management Tool available in Datastage Manager can give you the details you want.
If I were to do this I will load the dataset with different volumes and check how much size the Dataset is taking.
Rgds
--Rich
Posted: Fri Aug 27, 2004 1:16 am
by leo_t_nice
Hi
I guess the maximum size of the dataset will be 150 million * the row width, plus a small amount for the 'file pointer', but the size of each of the actual files making up the dataset will depend on the number of nodes you are running.
We have datasets in excess of 20 Gb, but we run with 8 nodes so the size of each file is around 2.5 Gb.
Hope this helps
Posted: Fri Aug 27, 2004 8:21 am
by mandyli
Hi Lakshmi,
I hope you got enough answer from rich and leo_t_nice reply. Size of the dataset files will depends on the number of nodes and no of CPU you are running and also check your datastage server memory size.
Thanks