Pivot stage too slow
Posted: Mon Apr 20, 2015 8:49 am
My job structure is something like this,
DataSource>Remove Duplicates>Pivot Stage>Transformer>TargetDB
I have 3million records to process. My pivot stage is taking 90mins to do that. I have 8 input columns to Pivot stage out of which 2 need to be pivoted vertically.
In the beginning I grouped by all other 6 columns. Then I reduced it to only 1 column. Still there is no improvment in performance. I used Hash and Roundrobin partitioning. Hash actually decreased the performance where as Roundrobin brought doen the procesing time to 80Mins.
Can you suggest any other steps the improve the performance of Pivot stage?
Thank You.
DataSource>Remove Duplicates>Pivot Stage>Transformer>TargetDB
I have 3million records to process. My pivot stage is taking 90mins to do that. I have 8 input columns to Pivot stage out of which 2 need to be pivoted vertically.
In the beginning I grouped by all other 6 columns. Then I reduced it to only 1 column. Still there is no improvment in performance. I used Hash and Roundrobin partitioning. Hash actually decreased the performance where as Roundrobin brought doen the procesing time to 80Mins.
Can you suggest any other steps the improve the performance of Pivot stage?
Thank You.