Remove Duplicate Stage

[pgupte]

Hi all, I'm removing duplicates, and sorting both my input fields one of which is Nullable.

eg; colA(NotNull) colB ( Nullable)
10 N
10 Y
10 Null

when I sort on both col A and B and say keep nulls last for col B, and remove duplicates based on A, the output is 10 N and 10 Null when I should be getting only 1 record 10 N. Does the sort or remove duplicate stage doesnt work well with Nullable fields ?? I'm on PX7.1

Thanks

[elavenil]

We use Remove Duplicate stage to eliminate duplicates and what we have done was to transform the null value and submit the result set to RD stage and that would eliminate duplicates in the mentioned key column. So i would suggest you to handle null before submitting the results to RD stage.

HTWH.

Regards
Saravanan

[pgupte]

I handled Nulls before the RD and it still wont give me correct results...

[ray.wurlod]

What do you believe are the "correct results"?
What are you getting? Why do you claim these are wrong?

[pgupte]

Hi Ray sorry, it was a stupid mistake, I was hashing and sorting on all 2 fields when I should be hashing on field A and Sorting on fields A and B..
Thanks

[pgupte]

Saravanan, it had nothing to do with Handling Nulls, thanks for your help.

[ray.wurlod]

Thanks for posting your "solution". Sometimes questions like those are valuable in "taking a step backwards" from your design. Another good thing is to have someone else look at it - it's amazing how the problem you couldn't see is blindingly obvious to them! And if they're professional they won't seek to embarrass you with it.

[pgupte]

Ray what u said is true, I used a sort stage before a RD and it worked and when I used inlink sort(RD) it wouldnt work, So I compared the OSH for both jobs and thats whew it was a silly mistake...