Hi,
we're sorting about 50GB of data. According to other posts unix sort should be better then datastage sort at these numbers.
Now I wanted to tune further, I came along the buffering parameters, since we have a lot of IO-wait on the unix box. default these are in MB (3MB max, 1MB disk wirte increment). I thought to multiply by 100 since we have 32 GB of memory.
This gave the error
main_program: Fatal Error: Invalid buffering parameters specified fordataset d_split:daily.v.
The dataset buffering parameters were:
{bufferFreeRun=0, 5, maximumMemoryBufferSize=314572800, diskWriteIncrement=104857600, queueUpperBound=0}
(tried also with 31457280,10485760)
Any ideas about
-why the error, is there some kind of project max I'm going over
-the approach: is my assumption that this will decrease the IO-wait at all valid?
-for this size of data/memory is 300MB a good idea, or should I go much lower(or higher?)
buffering parameters
Moderators: chulett, rschirm, roy
update: It doesn't seem to be the numbers.
The project is set to force buffer(APT_BUFFERING_POLICY) and all stages are left as buffering mode= default.
I understood that changing this to buffering mode=buffer, without changing the params should actually do the same thing. However if I do this I get the same error.
Any idea?
The project is set to force buffer(APT_BUFFERING_POLICY) and all stages are left as buffering mode= default.
I understood that changing this to buffering mode=buffer, without changing the params should actually do the same thing. However if I do this I get the same error.
Any idea?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You have marked the job type as server. APT_BUFFERING_POLICY is not applicable to server jobs. What job type is it really?
What value are you giving APT_BUFFERING_POLICY? There is a bug in some releases where the drop down list does not provide legal values (they are not correctly cased). The workaround is to create user-defined environment variables in Administrator, and there to use one of the values documented in the manual.
(I don't have access to parallel jobs this week, so can't provide precise details.)
What value are you giving APT_BUFFERING_POLICY? There is a bug in some releases where the drop down list does not provide legal values (they are not correctly cased). The workaround is to create user-defined environment variables in Administrator, and there to use one of the values documented in the manual.
(I don't have access to parallel jobs this week, so can't provide precise details.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.