Warning in when using Remove duplicate and sort stage

[snt_ds]

Hi,

I'm reading a sequential file and passing it to transformer to do hash partition.
data from trnsformer is passed to Sort stage where I'm doing Sorting on natural key in ascending and timestamp in descending to get the latest record at the top then it is passed to remove duplicate stage to retain the first record for that natural key.

I'm getting the below warning :

RDUP_NATKEY_TS: When checking operator: User inserted sort "SORT_NATKEY_TS_Desc" does not fulfill the sort requirements of the downstream operator "RDUP_NATKEY_TS"

can any please let me know how to fix this warning.

Thanks
Suresh

[kris007]

You don't need a transformer stage to do hash partitioning. You can do the hash partitioning with in the sort stage on the input partitioning tab.

[Edwink]

suresh,

after sorting records by keys
i think u are dropping any of the keys or changing datatypes or length any of the two keys in the same stage aor innext stage.


edi

[ray.wurlod]

Are you sorting the same keys, and in the same order, as required by (specified in) the Remove Duplicates stage?

[vmcburney]

I think your descending sort may be the problem. Most sorts by default are ascending. Your sort stage is doing an ascending, descending sort and your remove duplicates stage is probably configured for an ascending, ascending row order. You need to keep them synchronised to avoid the warning.

[snt_ds]

Hi,

I'm sorting the data in ascending order for natural keys and decending order for one of the timestamp column.

same order I'm maintaining in the remove duplicate stage to get the latest record.

Thanks
Suresh

[us1aslam1us]

Read this post.
viewtopic.php?t=95001&highlight=When+ch ... erted+sort

[samsuf2002]

Keys you are using in remove duplicate stage and sort should be same in order and data types . what is your last stage of the job?