Hi All,
We have a requirement of running a query on a huge DB table (100 million records or more) to get the required information. We do not have any filtering criteria to reduce the amount of data to be brought in to DS. Also we need to join this huge data with little less(less than a million records) data from a different database table.
Is it advisable to run this join in DS using DB connector and join stages. We are running on a 4 node SMTP server. let me know if more details are required
Dealing with huge data
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 53
- Joined: Fri Mar 07, 2008 1:17 am
Dealing with huge data
ETL DEVELOPER
Re: Dealing with huge data
You should be fine doing this join in DS. I suggest you to use correct partitioning and sort data if possible in DB before it gets to DS. Test your job with less data and see how it works before running it against full set.
Shawn K
--------------------------------------------------------
"What is right is not always popular and what is popular is not always right."
— Albert Einstein
--------------------------------------------------------
"What is right is not always popular and what is popular is not always right."
— Albert Einstein
Re: Dealing with huge data
A different table... in the same database?datastagedw wrote:Also we need to join this huge data with little less(less than a million records) data from a different database table.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers