DSXchange

I need to do around 200 lookup to get descriptions for every code for every rows. Doing 200 Lookup is ugly design. same is needed for multiple tables.


Input rows are around 10Mill Records. Tried Parallel routine, but its slowing down.


Can some one suggest me what need to be done.

Join?

shukla_krishna wrote:Doing 200 Lookup is ugly design.

I disagree. You do what you need to do.

Tip: spread things out. The design canvas is infinitely wide and deep. Add documentation (annotations) to indicate which lookup is being done where, and supplement that with meaningful stage and link names.

Spend some time monitoring this large job; it is an example where operator combination will be too aggressive; use selective stage-based denial of combinability to manage the processing load better.

Joining to the same table 200 times won't be "pretty" either but as noted, you have to do what you have to do to get the job done. Be it source joins or lookups.