Hi,
I am having 2 datasets.
In one of my job i am using funnel to merge the data of the two datasets.
My design is
DS----->Xfm----->
Funnel ---------->Remove Dup --------->Oracle
DS----->Xfm----->
When I try to run the job the funnel is taking a long time to pass the data from XFM to Remove Dup.
I can see the 2 million records before the funnel and I used continuos funnel.
I dont know why it is very slow.
Can anybody help me out.
Thanks,
Funnel Stage
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Hi,
when a remove duplicate stage is used the it is mandatory that data should be sorted on the key column before the data is passed to the remove duoplicate stage.
When sorting the data will go the spool space i.e temporary sapce and the location is defined in the configuration file.
As it was mentioned by you it has 2 million records you should have enough space on the server for the specified location.
you can use a link sort instead of a sperate sort stage.
The data at funnel stage might be waiting to get sorted befor RDS. It is confusing but DS will work in that way.
As Chulett mentioned do a usage analysis to know it in better way.
Might this solution will help.
when a remove duplicate stage is used the it is mandatory that data should be sorted on the key column before the data is passed to the remove duoplicate stage.
When sorting the data will go the spool space i.e temporary sapce and the location is defined in the configuration file.
As it was mentioned by you it has 2 million records you should have enough space on the server for the specified location.
you can use a link sort instead of a sperate sort stage.
The data at funnel stage might be waiting to get sorted befor RDS. It is confusing but DS will work in that way.
As Chulett mentioned do a usage analysis to know it in better way.
Might this solution will help.