Unable to start ORCHESTRATE process help!!!

[wojtask]

Hi All!

Today I got following error message:
Unable to start ORCHESTRATE process on node node0 (regan): APT_PMPlayer::APT_PMPlayer: fork() failed, Not enough space [processmgr/player.C:394]

df -k shows that 52% of space is avilable (for both datasets and scratch)

ulimit -a shows:

core file size (blocks) unlimited
data seg size (kbytes) unlimited
file size (blocks) unlimited
open files 1024
pipe size (512 bytes) 10
stack size (kbytes) 8192
cpu time (seconds) unlimited
max user processes 29995
virtual memory (kbytes) unlimited

apt file contains

node "node0"
{
fastname "regan"
pools ""
resource disk "/usr/dsadm/Ascential/DataStage/Datasets" {pools ""}
resource scratchdisk "/usr/dsadm/Ascential/DataStage/Scratch" {pools ""}
}


OS is Solaris.
Amount of processed data is about 1000 records

Do you have any ideas what may cause that error?

Regards
Wojtask

[ray.wurlod]

Are there a lot of other processes on the system when this error occurs? The not enough space message may relate to the process table. Check also to determine whether your inode table is full for the file system in question.

[wojtask]

Hi!

Unfortunately inode table changes didn't help. Here's part of job log ran after DS server reconfiguration and restart.


Regards
Wojtask

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: cs}}}(0),0: Failure during execution of operator logic [api/operator_rep.C:315]

Occurred: 11:04:10 On date: 28-07-2005 Type: Info
Event: cs}}}(0),0: Output 0 produced 0 records

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: cs}}}(0),0: Fatal Error: Fork failed: Not enough space [sort/writer.C:173]

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: cs}}},0: Failure during execution of operator logic [api/operator_rep.C:315]

Occurred: 11:04:10 On date: 28-07-2005 Type: Info
Event: cs}}},0: Output 0 produced 0 records

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: cs}}},0: Fatal Error: mmap failed: Resource temporarily unavailable [sort/writer.C:165]

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: ContainerC55.Change_Capture_390,0: Failure during execution of operator logic [api/operator_rep.C:315]

Occurred: 11:04:10 On date: 28-07-2005 Type: Info
Event: ContainerC55.Change_Capture_390,0: Output 0 produced 0 records

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: ContainerC55.Change_Capture_390,0: Fatal Error: waitForWriteSignal(): Premature EOF on node regan No such file or directory [iomgr/iocomm.C:1518]

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: cs}}},0: Failure during execution of operator logic [api/operator_rep.C:315]

Occurred: 11:04:10 On date: 28-07-2005 Type: Info
Event: cs}}},0: Output 0 produced 0 records

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: cs}}},0: Fatal Error: Fork failed: Not enough space [sort/writer.C:173]

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: subArgs={asc}}},0: Failure during execution of operator logic [api/operator_rep.C:315]

Occurred: 11:04:10 On date: 28-07-2005 Type: Info
Event: subArgs={asc}}},0: Output 0 produced 0 records

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: subArgs={asc}}},0: Fatal Error: Fork failed: Not enough space [sort/writer.C:173]

Occurred: 11:04:10 On date: 28-07-2005 Type: Fatal
Event: subArgs={asc}}},0: Failure during execution of operator logic [api/operator_rep.C:315]

Occurred: 11:04:10 On date: 28-07-2005 Type: Info
Event: subArgs={asc}}},0: Output 0 produced 0 records

[ArndW]

The "not enough space" is happening on a "fork" so I would look at physical and virtual memory space, plus process memory.

[murphykevin]

I recently had a similar fork failure. Take a look at the resources that are being created by DS in your Director output log. My issue was in the JOIN stage, trying to join 6 SybaseOC tables. The job was using 5.1GB during execution. The log showed that the process was creating 35 datasets to perform the join. I changed the JOIN execution mode from parallel(default) to sequential which reduced the number of datasets to 16. The fork failure error disappeared.

[wojtask]

Making all stages to run in sequential mode (where possible) didn't help

[trokosz]

I do not know what operating system your speaking of but each operating systems has a kernel setting called "max processes" and for PX for each node define in your apt there needs to be 100 processes....so 4 nodes implies 400 max proceses.....this could be your issue and I have seen this error.....see your Unix Admin