Dealing with huge data
Posted: Mon Jul 25, 2011 6:25 am
Hi All,
We have a requirement of running a query on a huge DB table (100 million records or more) to get the required information. We do not have any filtering criteria to reduce the amount of data to be brought in to DS. Also we need to join this huge data with little less(less than a million records) data from a different database table.
Is it advisable to run this join in DS using DB connector and join stages. We are running on a 4 node SMTP server. let me know if more details are required
We have a requirement of running a query on a huge DB table (100 million records or more) to get the required information. We do not have any filtering criteria to reduce the amount of data to be brought in to DS. Also we need to join this huge data with little less(less than a million records) data from a different database table.
Is it advisable to run this join in DS using DB connector and join stages. We are running on a 4 node SMTP server. let me know if more details are required