DSXchange

Hi All,

We have a job which read records(14488888) from the dataset and sort the records and fileter the outcoming records and to join the records. Until this pahse the job took only few minutes(30000 rec/sec). Again we are sorting the records with the different keys to concatenate the no.of occurences. In this sort stage it almost took 30 hours of time to procude 7000000 as 68rows/sec.

SortUtility:Datastage
StableSort:True
Node:4 nodes

Kindly let me know your thoughts to improve the performance.


Regards,
Satheesh

Are you running on a single physical machine or multiple physical machines? Are you repartitioning the data for the second sort?

What are the sort keys for the two sorts? Can you leverage the "don't sort (previously sorted)" functionality for some of the keys in the second sort?