Number of datasets which we can use in a single project

AmeyJoshi14 · Post by **AmeyJoshi14** » Mon Dec 07, 2009 1:20 am

Hi,

Actually there is a issue occured in our project...We want to store the cleanse data into intermedate stage..same as staging table ...initially we were extracting the data from the source and directly loading it into the target tables...but now we have to store the data into some intermediate stage( that is datasets) ....ideally the best option would be to store it into staging tables...but since there is lack of time....we decide to store the data into datasets....So is there any limit to use datasets, how many datasets which we can use in a particular project...is there any way to find out the free size in a particular project ??
The data volume is pretty huge and number of datasets which we are going to create will also be very huge...

If we use may datasets....it won't effect performance of other proejects on same datastage servers...

I don't know if its really very foolish questions...but gurus ...please shade some light on the same....

Thanks in Advance!!

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Mon Dec 07, 2009 3:22 am

All your questions are relative.

There is no particular limit in the number of datasets. But the more you have, the difficult to maintain.

You can always increase your machine's capacity to handle the data volume.

chulett · Post by **chulett** » Mon Dec 07, 2009 7:18 am

This has nothing to do with individual projects and everything to do with the amount of disk space you have allocated for datasets in your config files. There's no limit other than your hardware.

AmeyJoshi14 · Post by **AmeyJoshi14** » Mon Dec 07, 2009 8:41 am

chulett wrote:This has nothing to do with individual projects and everything to do with the amount of disk space you have allocated for datasets in your config files. There's no limit other than your hardware. ...

Thanks for the reply!!
So if the number of datasets are increased, then it will hamper the job performance..

..ok that's why now one of our job which is using lookup stage is getting aborted because of the space issue.( as we created around 127 datasets)

priyadarshikunal · Post by **priyadarshikunal** » Mon Dec 07, 2009 9:42 am

Its not about the number of datasets but the volume of data in datasets and the disk space allocated for datasets.

chulett · Post by **chulett** » Mon Dec 07, 2009 10:40 am

AmeyJoshi14 wrote:So if the number of datasets are increased, then it will hamper the job performance..

Not really sure how you read that into my reply.