Removing Duplicates using stage variables in parallel job

karthegx · Post by **karthegx** » Wed May 16, 2007 2:28 am

Hi,

I have a source file which contains duplicates in the data. I used this Derivation If Curr <> Prev Then 'Y' Else 'N' in the Stage variable and rejected the duplicate records to the sequential file which i want do it.

I tried the same logic in Parallel job but it's not working.

Why does this doesn,t work in Parllel transformer.

JoshGeorge · Post by **JoshGeorge** » Wed May 16, 2007 2:42 am

Make sure records are sorted and partitioned by key field for your duplicate checking so that matching rows are adjacent to each other and on the same node in the transformer. Also assign stage variables in right order (Stage variables are processed in order)

svPrev = svCurr
svCurr = input.field
svDupeStatus = If svCurr = svPrev Then 'Y' Else 'N'

karthegx · Post by **karthegx** » Wed May 16, 2007 5:11 am

Hi George,

This logic is working in Server job . But when i am trying in the parllel transformer its not working. Why its not working? Can you tell me about that.

Thanks in Advance

thompsonp · Post by **thompsonp** » Wed May 16, 2007 6:13 am

What is it doing that makes you say it is not working?

Try running it with a one node configuration file and see if it works? If it does it is your sorting and / or partitioning that needs looking at as has already been suggested.

DSguru2B · Post by **DSguru2B** » Wed May 16, 2007 8:19 am

Only the fairy god mother can tell why its not working. We can only guess, which is not good. JoshGeorge gave you a nice explanation. thompsonp added to it. We can only do so much sitting on this side of the screen.
A better way to do this is using the sort stage. Search the forum on the 'How to' part.

ray.wurlod · Post by **ray.wurlod** » Wed May 16, 2007 4:14 pm

Data must be partitioned on the comparison field as noted, and must also be sorted on this field so that comparison values are adjacent.