Sparse Lookup

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
rao_2004
Charter Member
Charter Member
Posts: 7
Joined: Wed Nov 10, 2004 3:38 pm
Contact:

Sparse Lookup

Post by rao_2004 »

Hi Folks,
I know the sparse lookup purpose but i was curious abt the behaviour of sparse lkp on high volume of data.
I mean the source file got 1 million records and the reference table(oracle) got almost 100 million records.
For each source record their might be 10 to 100 lkp records in reference table.so finally their might be 20 to 30 million lkp records in refernce table.
So My questions is ,can i see 30 million records in Target or 1 million records only.
The reason i'm asking in DataStage if we are using Loolupstage,if we have 1million records in source it process all 1 million records untill if we specify to drop unmatched records.
But here reference records are more than source(its kind of full outer join).
i know how to acheive this by join stage(i feel join stage will cost here), buy using sparse lkp can i perform all the logic ...and wat abt the performace in this situation...
Apprecite your comments...

Thanks,
R
kwwilliams
Participant
Posts: 437
Joined: Fri Oct 21, 2005 10:00 pm

Re: Sparse Lookup

Post by kwwilliams »

You would see 30 million records in the target because the lookup will return 10 to 100 rows per source record. Performance depends on your definition of performance. It may take a while based on your resources to load all of these rows into your target.
Post Reply