Problem with BASIC transformer in parallel job

[ashwin2912]

I have a BASIC transformer and one i/p file and o/p file in a parallel job...getting the following error...very surprising...I have no clue what is happening!!!

Just attached the job log below:

Code: Select all

@ID
FIELD ( TRANS ( RT_LOG109 , //JOB.STARTED.NO , 1 , X ) , ( char ( 252 ) ) , 1 )


Timestamp. 2005-04-11 17:46:38
Severity.. Warning
Full Text. Job control process (pid 6846) has failed

Timestamp. 2005-04-11 17:46:38
Severity.. Started
Full Text. Job qwe11. aborted

Timestamp. 2005-04-11 17:46:39
Severity.. Info
Full Text. Environment variable settings:
         . APT_ORCHHOME=/export/home/dsadm/Ascential/DataStage/PXEngine
         . DSHOME=/export/home/dsadm/Ascential/DataStage/DSEngine
         . HOME=/export/home/dsadm
         . HZ=
         . LD_LIBRARY_PATH=/export/home/dsadm/Ascential/DataStage/Proje
           cts/Rajeev/RT_BP109.O:/export/home/dsadm/Ascential/DataStage
           /Projects/Rajeev/buildop:/export/home/dsadm/Ascential/DataSt


         . DS_TDM_PIPE_OPEN_TIMEOUT=720
         . APT_COMPILER=/opt/SUNWspro/bin/CC
         . APT_COMPILEOPT=-dalign -O -PIC -library=iostream -c
         . APT_LINKER=/opt/SUNWspro/bin/CC
         . APT_LINKOPT=-G -library=iostream
         . APT_OPERATOR_REGISTRY_PATH=/export/home/dsadm/Ascential/Data
           Stage/Projects/Rajeev/buildop
         . OSH_STDOUT_MSG=1
         . APT_ERROR_CONFIGURATION=severity, !vseverity, !jobid,
           moduleid, errorIndex, timestamp, !ipaddr, !nodeplayer,
           !nodename, opid, message
         .

Timestamp. 2005-04-11 17:46:39
Severity.. Info
Full Text. Parallel job initiated
         . # OSH / orchestrate script for Job qwe11 compiled at
           17:46:34 11 APR 2005


         . ;
         .
         . ############################################################
           #####
         . #### STAGE: BASIC_Transformer_34
         . ## Operator
         . dsjobsh
         . ## Operator options
         . -p '[&DSProjectName]'
         . -h '[&DSHostName]:[&DSTCPPort]'
         . -e '[&DSInstallTag]'
         . -j 'qwe11(BASIC_Transformer_34).[&DSJobInvocationId]'
         . -i record
         . (
         .   Sno:string[max=255];
         .   Name:string[max=255];
         . )
         . -I


Timestamp. 2005-04-11 17:46:41
Severity.. Info
Full Text. main_program: APT configuration file:
           /export/home/dsadm/Ascential/DataStage/Configurations/defaul
           t.apt
         . {
         .      node "node1"
         .      {
         .              fastname "isc2app2u.isc2.co.in"
         .              pools ""
         .              resource disk
           "/export/home/dsadm/Ascential/DataStage/Datasets" {pools ""}
         .              resource scratchdisk
           "/export/home/dsadm/Ascential/DataStage/Scratch" {pools ""}
         .      }
         . }
         .
         . {


Full Text. BASIC_Transformer_34,0: The job did not run successfully.

Timestamp. 2005-04-11 17:46:44
Severity.. Fatal
Full Text. BASIC_Transformer_34,0: The runLocally() of the operator
           failed.

Timestamp. 2005-04-11 17:46:44
Severity.. Info
Full Text. BASIC_Transformer_34,0: Input 0 consumed 0 records.

Timestamp. 2005-04-11 17:46:44
Severity.. Info
Full Text. BASIC_Transformer_34,0: Output 0 produced 0 records.

Timestamp. 2005-04-11 17:46:45
Severity.. Fatal
Full Text. BASIC_Transformer_34,0: Operator terminated abnormally:
           runLocally did not return APT_StatusOk

Timestamp. 2005-04-11 17:46:50
Severity.. Fatal
Full Text. main_program: Unexpected exit status 1
         . Unexpected exit status 1

Timestamp. 2005-04-11 17:46:50
Severity.. Fatal
Full Text. main_program: Step execution finished with status = FAILED.

Timestamp. 2005-04-11 17:46:50
Severity.. Info
Full Text. main_program: Startup time, 0:08; production run time, 0:02.

Timestamp. 2005-04-11 17:46:50
Severity.. Started
Full Text. Job qwe11 aborted.


23 records listed.

[/code]

[ray.wurlod]

I don't really like what you're doing. Principally, what is @ID doing, and how is it actually specified in the expression in the BASIC Transformer?

You really shouldn't, however, be querying the repository tables in this way for every row processed - you can end up with a deadlock situation if DataStage is also trying to update the repository table.

[ashwin2912]

Hi Ray,

I am not querying for every row processed, I am just running the query once the job is over to get the log in a file..thats it!

Also @ID is part of the query and not in transformer....the error I am facing is

Code: Select all

Full Text. BASIC_Transformer_34,0: Operator terminated abnormally: 
           runLocally did not return APT_StatusOk 

Does this indicate anything to you....It is a basic transformer in a parallel job...Parallel transformer works fine.

Thanks,
Ashwin

I don't really like what you're doing. Principally, what is @ID doing, and how is it actually specified in the expression in the BASIC Transformer?

You really shouldn't, however, be querying the repository tables in this way for every row processed - you can end up with a deadlock situation if DataStage is also trying to update the repository table.

[ray.wurlod]

I assumed, because it was in the first two lines of your code, that you were specifying

Code: Select all

@ID 
FIELD ( TRANS ( RT_LOG109 , //JOB.STARTED.NO , 1 , X ) , ( char ( 252 ) ) , 1 ) 

as an (the?) expression in the BASIC Transformer.

So can you advise what you ARE doing in the BASIC Transformer stage that is failing to execute? The operator that it has generated has failed to run successfully, as you can see in the log, but there is no guidance from you as to what it is trying to do (and therefore why it might be failing).

Right now, you've given us "my car doesn't work, can you fix it?".

[Eric]

Timestamp. 2005-04-11 17:46:38
Severity.. Warning
Full Text. Job control process (pid 6846) has failed

It looks like the first log entry implies that things went wrong before the transformer was run?

[ashwin2912]

I am not doing anything in the transformer. I am just mapping i/p cols and o/p cols...I think the real problem (I dont know what it is...hence this query) is not evident in the log.

[ray.wurlod]

OK. If the problem is with a BASIC Transformer stage (as per your subject), then you ARE hitting it every row. But, of course, you're not even getting to process row #1.

Yet the log file is telling us that the runLocally() function associated with the operator generated by compiling the BASIC Transformer stage has something about it that caused it to fail, or that the operator itself has something about it.

If all you're doing is mapping input to output columns, why aren't you using a Transformer stage (rather than a BASIC Transformer stage)?

Do you get this problem with every job that contains a BASIC Transformer stage, or only the one job? If more than one, can you discern anything else that might be a pattern? What happens, for example, if you specify a one-node configuration?

[ashwin2912]

I get this problem with every job that contains a BASIC Transformer stage....I havent tried the node configuration though...I tried to change sequential/parallel option in BASIC tx but still got the same error.
Anyway we will be moving to a different env from now.....will check out this situation there.

[adnaanks]

I've just gone through the process of removing all BASIC transformers from my parallel job. Everything was working fine, until i started integrating the PX jobs with Job sequeneces then i started recieving MUTEX errors, i followed the advise on this board and switched over to Parallel Transformer to resolve the issue.

You might want to consider migrating away from BASIC transformers to avoid unexpected issues further down the line.