Abnormal termination of stage -Transformer

47shailesh · Post by **47shailesh** » Mon Jun 29, 2009 5:23 am

Hello Guys,

Every weekend my jobs runs a full load. Full load generates 60k records in flat file and input it to the Transormer_1, for further lookup and processing.

Intermittently in recent past my job aborts giving error:

Code: Select all

"Abnormal termination of stage <jobname>..Transformer_1 detected"

On reset 'From previous run' says:

Code: Select all

From previous run
DataStage Job 197 Phantom 17904
jobnotify: Error 904 occurred.
DataStage Phantom Finished.
[409654] DSD.StageRun <jobname>.Transformer_1 933 0/0/1 - core dumped.

dsenv has $ORACLE_HOME and LIB_PATH declared and exported.
We've oracle 9.2.0.4.0 installed.

Is the 60k size of file fed to Xfrmer creating issue ?
Any other config I've to check ?

ArndW · Post by **ArndW** » Mon Jun 29, 2009 5:40 am

No, 60K shouldn't cause the transformer to hiccup. You can try to trace the core file with dbx, but that takes a bit of skill and practice. If we approach the error from the other end - what are you doing in your transform stage that is not merely passing through values and could cause problems? External or user routine calls? Stage variable manipulations?

chulett · Post by **chulett** » Mon Jun 29, 2009 5:45 am

What version of Oracle are you loading into? I'm curious if you client version matches your server version and if there is an opportunity here to 'upgrade' the client to a slightly higher version.

47shailesh · Post by **47shailesh** » Mon Jun 29, 2009 5:56 am

@ArndW
The job has no routine calls and doesn't uses any stage variables. There are 3 contraints in the transformer which redirects input from file to 3 other transformers for separate file creation.

@chullet
Job doesn't loads to any database. Job has multiple transformers and outputs multiple files by doing lookups.
Is the Oracle upgrade really required coz it's not in my hand and It would need a strong reason to justify upgrade

chulett · Post by **chulett** » Mon Jun 29, 2009 6:02 am

Ok... wrong choice of words. What version of Oracle are you connecting to? I can't answer your question until you answer mine. And just to double-check, your 'lookups' are all from hashed files, yes?

47shailesh · Post by **47shailesh** » Mon Jun 29, 2009 6:35 am

@craig

My job connects to Oracle 9.2.0.4.0 .
Lookups are from ORA-OCI-9 stage, no hashed files used.

chulett · Post by **chulett** » Mon Jun 29, 2009 6:43 am

Well, at least they match which is a Good Thing but not necessarily always the case, hence the question. And the only reason I brought it up is because I personally have seen intermittent core dump issues like this myself caused by a buggy Oracle client.

So as you're trying to solve this, keep that in the back of your mind. We're not talking anything major here, just (perhaps) a maintenance release up to 9.2.0.5 or 9.2.0.6 - ones that "we" found to be more stable. Another query - what flavor of UNIX are you running - HP-UX by any chance? If so, then I'm going to double-down on my bet.

47shailesh · Post by **47shailesh** » Mon Jun 29, 2009 6:51 am

@craig

We are using AIX v5.3.0.60

ArndW · Post by **ArndW** » Mon Jun 29, 2009 8:28 am

Replace the Oracle output with a sequential file stage. Does the error go away?

chulett · Post by **chulett** » Mon Jun 29, 2009 9:09 am

Unfortunately, Oracle is the source here so hard to swap it out for something else... except pehaps for an ODBC stage.

DSguru2B · Post by **DSguru2B** » Mon Jun 29, 2009 1:33 pm

Make sure that all your lookups and source have identical array sizes set. I know this might sound odd but I have had "Abnormal terminations" before and more than once, the array size fix has worked for me. But then again my issues were with the target. Since your target is a flat file, I cannot assure you that this will work, but is worth a try.
If it fails, change all your array sizes to 1 and see if that helps.

47shailesh · Post by **47shailesh** » Tue Jun 30, 2009 2:42 am

Are there any bugs in Oracle 9.2.0.4, related to this issue which were patched in 9.2.0.5/6 release.
This would be helpful for me to explain to my dba for why I need an upgrade.

Accordingly I'll ask dba for oracle maintenance upgrade to 9.2.0.5/6 and will see if the issue occurs again.
The issue is intermittent, so finding a permanent fix may take some time.

chulett · Post by **chulett** » Tue Jun 30, 2009 6:13 am

As noted earlier, the only way to know for certain would be to have support analyze the core dump.

47shailesh · Post by **47shailesh** » Thu Jul 02, 2009 5:59 am

Can I analyze the core dump ? If yes then how shall I proceed.

ArndW · Post by **ArndW** » Thu Jul 02, 2009 6:03 am

In order to analyze the core you need to understand how an executable program is built together and how runtime memory allocation and processing functions. The dbx program is usually used to do this (it is free with UNIX but not always installed) along with the actual executable that cause the core. While you can do a stack trace and see what is on the top of the stack it takes some practice (and patience) to track down core dumps.