removing duplicates

pradeep_nov18 · Post by **pradeep_nov18** » Fri Mar 21, 2008 6:20 am

suppose if the data is repeated
i wil illustrate with an ex
A b
1 q
1 r
1 k
1 k
1 k
suppose they are 10,000 i don't no wat is ronum of that duplicate location i found that 1,k which is in the center is duplicate since it is denormalised data how to remove that in transformer and also please send me list of frequently used constraint in dtastage jobs in propogating right data to table or file and how to handle nulls remove leading spaces and also if possible send the transformation logic accordling the client requirement which u guys have faced in the projects please do the need ful since i need it very badly even i too some experience i wnt to match my calibre with u guys

kumar_s · Post by **kumar_s** » Fri Mar 21, 2008 8:20 am

You first need to Sort the data based on the key feilds and Either use RemoveDuplicate stage or Aggregator Stage or Sort or Sort coupled with Transformer Stage optionally with Stage Variables.

kumar_s · Post by **kumar_s** » Fri Mar 21, 2008 8:21 am

You first need to Sort the data based on the key feilds and Either use RemoveDuplicate stage or Aggregator Stage or Sort or Sort coupled with Transformer Stage optionally with Stage Variables.

kumar_s · Post by **kumar_s** » Fri Mar 21, 2008 9:38 am

As you asked again,
If you could check in Document, you have whole set of information.
There are 7 Null handling functions available in Transformer.
IsNotNull, IsNull, MakeNull, NullToEmpty, NullToZero, NullToValue, SetNull.
You can use Trim with its different attributes.
If you be more specific, you will get more specific answers.

ray.wurlod · Post by **ray.wurlod** » Fri Mar 21, 2008 5:14 pm

At DSXchange we prefer a more professional standard of written English, the same as you would use when documenting your work. DSXchange is not a mobile telephone. People whose first language is not English have a hard enough time understanding well-written English - they experience added difficulty understanding things like "i don't no wat is ronum" or "u guys". I consider them to be abominations. Finally, there is a Burmese participant on the forum whose name is U - so please use the correct spelling of the second person personal pronoun, which is "you". Take care with spelling of other words too, please.

ray.wurlod · Post by **ray.wurlod** » Fri Mar 21, 2008 5:29 pm

Use a Remove Duplicates stage based on both columns. Ensure that data are partitioned correctly (on both columns).