DSXchange

Hi,

I am having 2 datasets.

In one of my job i am using funnel to merge the data of the two datasets.

My design is

DS----->Xfm----->
Funnel ---------->Remove Dup --------->Oracle
DS----->Xfm----->

When I try to run the job the funnel is taking a long time to pass the data from XFM to Remove Dup.

I can see the 2 million records before the funnel and I used continuos funnel.

I dont know why it is very slow.

Can anybody help me out.

Thanks,

Probably not the Funnel. Have you dumped the score for this? The Remove Duplicates stage requires sorted data from what I recall, so there may be a tsort operator in there it added that is holding up the show.

You can use Performance Analysis to determine where the "hot spots" are in a job.

Hi,

when a remove duplicate stage is used the it is mandatory that data should be sorted on the key column before the data is passed to the remove duoplicate stage.

When sorting the data will go the spool space i.e temporary sapce and the location is defined in the configuration file.

As it was mentioned by you it has 2 million records you should have enough space on the server for the specified location.

you can use a link sort instead of a sperate sort stage.

The data at funnel stage might be waiting to get sorted befor RDS. It is confusing but DS will work in that way.

As Chulett mentioned do a usage analysis to know it in better way.

Might this solution will help.