buffering parameters
Posted: Thu Sep 21, 2006 7:14 am
Hi,
we're sorting about 50GB of data. According to other posts unix sort should be better then datastage sort at these numbers.
Now I wanted to tune further, I came along the buffering parameters, since we have a lot of IO-wait on the unix box. default these are in MB (3MB max, 1MB disk wirte increment). I thought to multiply by 100 since we have 32 GB of memory.
This gave the error
main_program: Fatal Error: Invalid buffering parameters specified fordataset d_split:daily.v.
The dataset buffering parameters were:
{bufferFreeRun=0, 5, maximumMemoryBufferSize=314572800, diskWriteIncrement=104857600, queueUpperBound=0}
(tried also with 31457280,10485760)
Any ideas about
-why the error, is there some kind of project max I'm going over
-the approach: is my assumption that this will decrease the IO-wait at all valid?
-for this size of data/memory is 300MB a good idea, or should I go much lower(or higher?)
we're sorting about 50GB of data. According to other posts unix sort should be better then datastage sort at these numbers.
Now I wanted to tune further, I came along the buffering parameters, since we have a lot of IO-wait on the unix box. default these are in MB (3MB max, 1MB disk wirte increment). I thought to multiply by 100 since we have 32 GB of memory.
This gave the error
main_program: Fatal Error: Invalid buffering parameters specified fordataset d_split:daily.v.
The dataset buffering parameters were:
{bufferFreeRun=0, 5, maximumMemoryBufferSize=314572800, diskWriteIncrement=104857600, queueUpperBound=0}
(tried also with 31457280,10485760)
Any ideas about
-why the error, is there some kind of project max I'm going over
-the approach: is my assumption that this will decrease the IO-wait at all valid?
-for this size of data/memory is 300MB a good idea, or should I go much lower(or higher?)