Dealing with huge data

[datastagedw]

Hi All,

We have a requirement of running a query on a huge DB table (100 million records or more) to get the required information. We do not have any filtering criteria to reduce the amount of data to be brought in to DS. Also we need to join this huge data with little less(less than a million records) data from a different database table.

Is it advisable to run this join in DS using DB connector and join stages. We are running on a 4 node SMTP server. let me know if more details are required

[shawn.k]

You should be fine doing this join in DS. I suggest you to use correct partitioning and sort data if possible in DB before it gets to DS. Test your job with less data and see how it works before running it against full set.

[chulett]

Also we need to join this huge data with little less(less than a million records) data from a different database table.

A different table... in the same database?

[DSguru2B]

Is that join going to reduce the data???
If yes, then you can load the data from the "other" database into your current database that houses your source. Pass a sql join and extract.