Sparse Lookup
Posted: Thu Sep 20, 2007 11:48 am
Hi Folks,
I know the sparse lookup purpose but i was curious abt the behaviour of sparse lkp on high volume of data.
I mean the source file got 1 million records and the reference table(oracle) got almost 100 million records.
For each source record their might be 10 to 100 lkp records in reference table.so finally their might be 20 to 30 million lkp records in refernce table.
So My questions is ,can i see 30 million records in Target or 1 million records only.
The reason i'm asking in DataStage if we are using Loolupstage,if we have 1million records in source it process all 1 million records untill if we specify to drop unmatched records.
But here reference records are more than source(its kind of full outer join).
i know how to acheive this by join stage(i feel join stage will cost here), buy using sparse lkp can i perform all the logic ...and wat abt the performace in this situation...
Apprecite your comments...
Thanks,
R
I know the sparse lookup purpose but i was curious abt the behaviour of sparse lkp on high volume of data.
I mean the source file got 1 million records and the reference table(oracle) got almost 100 million records.
For each source record their might be 10 to 100 lkp records in reference table.so finally their might be 20 to 30 million lkp records in refernce table.
So My questions is ,can i see 30 million records in Target or 1 million records only.
The reason i'm asking in DataStage if we are using Loolupstage,if we have 1million records in source it process all 1 million records untill if we specify to drop unmatched records.
But here reference records are more than source(its kind of full outer join).
i know how to acheive this by join stage(i feel join stage will cost here), buy using sparse lkp can i perform all the logic ...and wat abt the performace in this situation...
Apprecite your comments...
Thanks,
R