Page 1 of 1

Sparse Lookup

Posted: Thu Sep 20, 2007 11:48 am
by rao_2004
Hi Folks,
I know the sparse lookup purpose but i was curious abt the behaviour of sparse lkp on high volume of data.
I mean the source file got 1 million records and the reference table(oracle) got almost 100 million records.
For each source record their might be 10 to 100 lkp records in reference table.so finally their might be 20 to 30 million lkp records in refernce table.
So My questions is ,can i see 30 million records in Target or 1 million records only.
The reason i'm asking in DataStage if we are using Loolupstage,if we have 1million records in source it process all 1 million records untill if we specify to drop unmatched records.
But here reference records are more than source(its kind of full outer join).
i know how to acheive this by join stage(i feel join stage will cost here), buy using sparse lkp can i perform all the logic ...and wat abt the performace in this situation...
Apprecite your comments...

Thanks,
R

Re: Sparse Lookup

Posted: Thu Sep 20, 2007 3:45 pm
by kwwilliams
You would see 30 million records in the target because the lookup will return 10 to 100 rows per source record. Performance depends on your definition of performance. It may take a while based on your resources to load all of these rows into your target.