Hi All,
We have a job which read records(14488888) from the dataset and sort the records and fileter the outcoming records and to join the records. Until this pahse the job took only few minutes(30000 rec/sec). Again we are sorting the records with the different keys to concatenate the no.of occurences. In this sort stage it almost took 30 hours of time to procude 7000000 as 68rows/sec.
SortUtility:Datastage
StableSort:True
Node:4 nodes
Kindly let me know your thoughts to improve the performance.
Regards,
Satheesh
Sort Performance issue
Moderators: chulett, rschirm, roy
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Are you running on a single physical machine or multiple physical machines? Are you repartitioning the data for the second sort?
What are the sort keys for the two sorts? Can you leverage the "don't sort (previously sorted)" functionality for some of the keys in the second sort?
What are the sort keys for the two sorts? Can you leverage the "don't sort (previously sorted)" functionality for some of the keys in the second sort?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.