Need help in designing Sparse Lookup Job
Posted: Sun Jan 16, 2011 12:02 pm
Hi All,
I need your help in making a design decision. My requirement is to first query SQL Server and get ID & Code. Use this ID to do a lookup on Sybase table to get other information. Problem here is Sybase table is huge (about 300,000 records) and it's except to grow rapidly in couple of years. On the other hand we know that records returned from sql server will not cross more than 1000 records. I thought to use Sparse lookup for this and my only concern is when ever we use Sparse lookup DataStage will issue a select statement to database for each record and I think this will be very resource intensive. I just want to know is there any other way I can design this job or this is my only option.
Thanks,
Shawn
I need your help in making a design decision. My requirement is to first query SQL Server and get ID & Code. Use this ID to do a lookup on Sybase table to get other information. Problem here is Sybase table is huge (about 300,000 records) and it's except to grow rapidly in couple of years. On the other hand we know that records returned from sql server will not cross more than 1000 records. I thought to use Sparse lookup for this and my only concern is when ever we use Sparse lookup DataStage will issue a select statement to database for each record and I think this will be very resource intensive. I just want to know is there any other way I can design this job or this is my only option.
Thanks,
Shawn