we use the below 4 nodes configuration for our prod environment and curious to understand multinode configuration for Scratch disk (scratch area) will improve performance during SORT operation.
Despite the fact that you have four nodes, you only have two scratchdisk directories, so there may be some contention for them. Better performance would be had with four scratchdisk directories (on separate file systems unless your disks are part of a SAN). Even better would be eight scratchdisk directories, so you get multiple I/O channels per node.
You did, however, pick up the concept of "round robin" allocation, which will be better than not doing so.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
IS your job taking up most of it's time during a SORT?
==========
Just because you "could" get better performance by writing to a seperate physical drive per compute node, doesn't mean that will improve the speed of your job.
Have you exausted all other avenues of speed tweeks?
Yes. We are facing some slow performance while doing Sort operation in the job.
We are reviewing all avenues to improve the performance. While reviewing the config file, it was identified such config and thought of getting expertise thoughts on this.
What's the volume of data you are regularly sorting (bytes, not number of records) with a single job? If it's in the multi-gigabyte range, I would suggest changing the Restrict Memory Buffer option in the Sort stage to a higher value. The default is 20MB, which results in 10MB scratch work files. Increase by an order of magnitude (from 20, change to 200-256MB) and see if that helps.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
Hi, Could you tell me how to change the Restrict Memory Buffer ?
We run the DS job in osh script , which Environment Variables can change the size of Restrict Memory Buffer ?Thanks!~
Johnny0638 wrote:We run the DS job in osh script , which Environment Variables can change the size of Restrict Memory Buffer ?
The global memory for tsort operators is set by environment variable APT_TSORT_STRESS_BLOCKSIZE. Note that this sets the memory allocated for all tsort operators.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.