Estimate Lookup Size

tostay2003 · Post by **tostay2003** » Tue Dec 21, 2010 8:12 pm

Hi,

Is there any way to estimate the max possible size of data that can be used as secondary link for Lookup Stage.

Thanks

ray.wurlod · Post by **ray.wurlod** » Tue Dec 21, 2010 9:40 pm

Yes. The size of each data type is documented (from memory in Chapter 2 of the Parallel Job Developer's Guide). Add these together for all the columns in the row that you're processing, then multiply by the number of rows in the reference data set.

tostay2003 · Post by **tostay2003** » Wed Dec 22, 2010 1:20 am

ray.wurlod wrote:Yes. The size of each data type is documented (from memory in Chapter 2 of the Parallel Job Developer's Guide). Add these together for all the columns in the row that you're processing, then ...

Hi Ray,

Sorry I didn't post my question properly.

After calculating the size, how can we determine whether Lookup might work without memory issues.

Thanks

ray.wurlod · Post by **ray.wurlod** » Wed Dec 22, 2010 2:25 am

How much memory do you have? How much scratchdisk in the nodes on which the Lookup stage operates? How many other stages are there in the job? How long is a piece of string? (Try it and see.)