Lookup stage performance Versu Merge stage performance
Posted: Wed Jan 12, 2011 5:14 pm
Hi,
I have job which parses an xml file and looks up against a dataset(table dump) and if the keys are existing it will return the key, if not it will generate a new key and writes to a dataset.
My data volumes are really huge, So, once the lookup dataset got close 3.5GB the job was failing due to lack of temp space. So, I thought if I replace lookup stage with Merge it is going help this situation and with performance of the job as well.
But, I don't see any improvement. In fact the job with Lookup is running 30 seconds faster. This is when the volume is a little above 1 million rows.
Are there any specific parameters I need to enable for Merge to perform faster?
Thanks for your help.
I have job which parses an xml file and looks up against a dataset(table dump) and if the keys are existing it will return the key, if not it will generate a new key and writes to a dataset.
My data volumes are really huge, So, once the lookup dataset got close 3.5GB the job was failing due to lack of temp space. So, I thought if I replace lookup stage with Merge it is going help this situation and with performance of the job as well.
But, I don't see any improvement. In fact the job with Lookup is running 30 seconds faster. This is when the volume is a little above 1 million rows.
Are there any specific parameters I need to enable for Merge to perform faster?
Thanks for your help.