buffering parameters

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
jasper
Participant
Posts: 111
Joined: Mon May 06, 2002 1:25 am
Location: Belgium

buffering parameters

Post by jasper »

Hi,
we're sorting about 50GB of data. According to other posts unix sort should be better then datastage sort at these numbers.
Now I wanted to tune further, I came along the buffering parameters, since we have a lot of IO-wait on the unix box. default these are in MB (3MB max, 1MB disk wirte increment). I thought to multiply by 100 since we have 32 GB of memory.
This gave the error

main_program: Fatal Error: Invalid buffering parameters specified fordataset d_split:daily.v.
The dataset buffering parameters were:
{bufferFreeRun=0, 5, maximumMemoryBufferSize=314572800, diskWriteIncrement=104857600, queueUpperBound=0}
(tried also with 31457280,10485760)

Any ideas about
-why the error, is there some kind of project max I'm going over
-the approach: is my assumption that this will decrease the IO-wait at all valid?
-for this size of data/memory is 300MB a good idea, or should I go much lower(or higher?)
jasper
Participant
Posts: 111
Joined: Mon May 06, 2002 1:25 am
Location: Belgium

Post by jasper »

update: It doesn't seem to be the numbers.
The project is set to force buffer(APT_BUFFERING_POLICY) and all stages are left as buffering mode= default.
I understood that changing this to buffering mode=buffer, without changing the params should actually do the same thing. However if I do this I get the same error.
Any idea?
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You have marked the job type as server. APT_BUFFERING_POLICY is not applicable to server jobs. What job type is it really?

What value are you giving APT_BUFFERING_POLICY? There is a bug in some releases where the drop down list does not provide legal values (they are not correctly cased). The workaround is to create user-defined environment variables in Administrator, and there to use one of the values documented in the manual.

(I don't have access to parallel jobs this week, so can't provide precise details.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
jasper
Participant
Posts: 111
Joined: Mon May 06, 2002 1:25 am
Location: Belgium

Post by jasper »

correct forum, incorrect indication: It's a parallel job.
Post Reply