Page 1 of 1

buffering parameters

Posted: Thu Sep 21, 2006 7:14 am
by jasper
Hi,
we're sorting about 50GB of data. According to other posts unix sort should be better then datastage sort at these numbers.
Now I wanted to tune further, I came along the buffering parameters, since we have a lot of IO-wait on the unix box. default these are in MB (3MB max, 1MB disk wirte increment). I thought to multiply by 100 since we have 32 GB of memory.
This gave the error

main_program: Fatal Error: Invalid buffering parameters specified fordataset d_split:daily.v.
The dataset buffering parameters were:
{bufferFreeRun=0, 5, maximumMemoryBufferSize=314572800, diskWriteIncrement=104857600, queueUpperBound=0}
(tried also with 31457280,10485760)

Any ideas about
-why the error, is there some kind of project max I'm going over
-the approach: is my assumption that this will decrease the IO-wait at all valid?
-for this size of data/memory is 300MB a good idea, or should I go much lower(or higher?)

Posted: Thu Sep 21, 2006 7:23 am
by jasper
update: It doesn't seem to be the numbers.
The project is set to force buffer(APT_BUFFERING_POLICY) and all stages are left as buffering mode= default.
I understood that changing this to buffering mode=buffer, without changing the params should actually do the same thing. However if I do this I get the same error.
Any idea?

Posted: Thu Sep 21, 2006 3:07 pm
by ray.wurlod
You have marked the job type as server. APT_BUFFERING_POLICY is not applicable to server jobs. What job type is it really?

What value are you giving APT_BUFFERING_POLICY? There is a bug in some releases where the drop down list does not provide legal values (they are not correctly cased). The workaround is to create user-defined environment variables in Administrator, and there to use one of the values documented in the manual.

(I don't have access to parallel jobs this week, so can't provide precise details.)

Posted: Fri Sep 22, 2006 12:23 am
by jasper
correct forum, incorrect indication: It's a parallel job.