Transformer BlockSize and node_node1: Player 1 terminated

nmacolin · Post by **nmacolin** » Thu Feb 12, 2009 2:17 am

Hi all,

Here is the situation we have a series of simplistic jobs which extract from Unidata and then load into Db2. This is more of a replication project than true complex ETL.

Anyway we had a problem with the following errors appearing in jobs randomly. Well so I thought.

Message:: SEFJobVariationTypes,0: Internal Error: (shbuf): iomgr\iomgr.C: 1880
Message:: node_node2: Player 1 terminated unexpectedly.
Message:: node_node1: Player 1 terminated unexpectedly.
Message:: main_program: APT_PMsectionLeader(2, node2), player 1 - Unexpected termination by Unix signal 9(SIGKILL).
Message:: main_program: APT_PMsectionLeader(1, node1), player 1 - Unexpected exit status 1.

A search of the site referred to threads where people referred to patches that need to be applied not the case here as we are on 8.01 with FixPatch1A applied.

IBM support couldn't shed any light ... change this do that ... nothing.

Job structure for Parallel Load

SEF --> TRansFormer --> Db2Load

In the Transformer we use a sequence (SurrogateKeyGen) to generate the number and have set the blocksize to something reasonable to Say 200 for a reasonable number of rows to be processed otherwise the performance hit is marked with system block size checked.
With the system block size set to 200 we experience the error above and jobs abort all over the place.

Anyway changing back to system blocksize on these surrogateKeyGen in the transformer removed the problem.

Can someone please give me a rationale as why this may occur ? I can understand that allocating the blocksize means that this number of sequences need to be stored in memory for the duration of the job, but we are only talking about 200

Why would this small obscure change in the Transformer have such a major affect on the DS server and cause the jobs to abort with memory issues.

Any knowledge you could impart that would make sense of this would be appreciated.

Regards
Nick

ray.wurlod · Post by **ray.wurlod** » Thu Feb 12, 2009 1:51 pm

Can you explain more completely what SEF is, and how it extracts data from UniData? Are you using a state file for the surrogate key storage? What syntax are you using for the function that delivers the next surrogate key value in the Transformer stage?

nmacolin · Post by **nmacolin** » Thu Feb 12, 2009 4:32 pm

Hi Ray,

SEF = Sequential File Stage
Extraction from Unidata is via the UnidataStage. The stage is pretty simply it just allows for the multivalued Unidata Structure. Don't have any issues here.

The SurrogateKeyGen is done using a Flat File on the file system. We initially used a call to a DB sequence, but that involved manually editing the SQL for the insert statement and increased the maintenance.
The BlockSize settings for the surrogateKeyGen greatly affect the performance of the transformer. We found allowing the system to decide how many sequences to generate from the file caused the job to perform poorly, but manually selecting the number of blocks to retrieve greatly improved the performance, makes sense the application is pre-fetching a set number of sequences.
Anyway as I mentioned before this manual setting is having a nasty side effect of causing the memory management to go mad and abort the jobs.

So looking for someone who has a better understanding of why this would be the case.

If we leave all the jobs as system size, then the ETL windows blows out, it means jobs take 4 to 5 times longer to complete.

Regards
Nick