this is the error i am getting after i ran the job:
APT_CombinedOperatorController(0),0: Failure during execution of operator logic.
APT_CombinedOperatorController(0),0: Fatal Error: Tsort merger aborting: Scratch space full
is this related with the memory of sort stage? actually i am sorting based on two fields and the input number of rows is 8055979.
pls help in solving the error.
i have not used any partitioning methods on this stage ...shuld i use any?
fatal error - sort stage
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You can configure the maximum memory (per partition) that the Sort stage will use (by default it is 30MB). Any more than this spills to the scratch disk nominated in the configuration file.
Partitioning is irrelevant, except that a poorly chosen partitioning algorithm (for example one that forces all rows onto one partition) will exacerbate the problem.
What you must do is some combination of allowing the Sort stage to use more memory and allocating additional scratchdisk resource.
Partitioning is irrelevant, except that a poorly chosen partitioning algorithm (for example one that forces all rows onto one partition) will exacerbate the problem.
What you must do is some combination of allowing the Sort stage to use more memory and allocating additional scratchdisk resource.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
how to avoid this error? is it possible to change the maximum capacity?how?if i use partitioning (i have used HASH partition now) will the same be avoided?ray.wurlod wrote:You can configure the maximum memory (per partition) that the Sort stage will use (by default it is 30MB). Any more than this spills to the scratch disk nominated in the configuration file.
Partit ...
vij wrote:how to avoid this error? is it possible to change the maximum capacity?how?if i use partitioning (i have used HASH partition now) will the same be avoided?ray.wurlod wrote:You can configure the maximum memory (per partition) that the Sort stage will use (by default it is 30MB). Any more than this spills to the scratch disk nominated in the configuration file.
Partit ...
now, i get these fatal errors from sorter - APT_CombinedOperatorController,2: write failed: Output file full, and no more output files
APT_CombinedOperatorController,2: Failure during execution of operator logic.
APT_CombinedOperatorController,2: Fatal Error: Tsort merger aborting: mergeOneRecord() punted
cs}}},2: Failure during execution of operator logic
APT_CombinedOperatorController,2: Fatal Error: Pipe read failed: short read.
Look closely at the error messages.
Your first post highlighted that the Scrtach space was full.
As Ray said the location of the Scratch space is defined in your configuration file and is used to write data that is being sorted and doesn't fit in memory.
As suggested you could adjust the optional parameter 'Restrict Memory Usage' on the sort stage from the default value of 20MB to a greater value - provided you have enough memory, so that there is less space required in the Scratch area.
Your last post indicates that your Output File is full. This message has come from a combined controller. Add the environment variable APT_DISABLE_COMBINATION and set it to TRUE and rerun the job. The message you get should indicate which stage has run out of space. Be sure to remove this environment variable when you have things working.
I would guess that your are writing a file or dataset somewhere and it is probably to the same disk as your scratch space. Check the available space at these locations.
Your first post highlighted that the Scrtach space was full.
As Ray said the location of the Scratch space is defined in your configuration file and is used to write data that is being sorted and doesn't fit in memory.
As suggested you could adjust the optional parameter 'Restrict Memory Usage' on the sort stage from the default value of 20MB to a greater value - provided you have enough memory, so that there is less space required in the Scratch area.
Your last post indicates that your Output File is full. This message has come from a combined controller. Add the environment variable APT_DISABLE_COMBINATION and set it to TRUE and rerun the job. The message you get should indicate which stage has run out of space. Be sure to remove this environment variable when you have things working.
I would guess that your are writing a file or dataset somewhere and it is probably to the same disk as your scratch space. Check the available space at these locations.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Both sets of error messages indicate that you're running out of disk space. If you're trying to write to a Data Set, File Set or Lookup File Set that's the disk resource defined in the configuration file; if not, then it's the file system on which your file exists.
The solution: add more disk space. More file systems are needed; not more directories on the same file system.
The solution: add more disk space. More file systems are needed; not more directories on the same file system.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia