Job substituting default params in place of the passed ones

[evee1]

I have a routine activity which picks up a date from a dataset and passes the same to the next job as a parameter.Both my job and routine are added sequentially in the same sequence job.I am calling the routine through a routine activity stage in a sequence job.please find below the routine code(in Basic)

I have a job (it's a server job with Allowed Multiple Instances enabled), let's call it Job_ABC, that has some input parameters, all with the default values. The first of them is of the Parameter Set type, the rest are Strings. This job is performing some standard simple processing, like updating a status field in the database, where table name is passed as a parameter.

I have multiple sequences that use the above job and they pass the appropriate parameters, one of them being a call to a user-defined routine. The sequences are combined in a larger parent and master sequences.

The problem is that I'm getting an intermittent error when one of the instances of the job Job_ABC is not picking up the passed parameters correctly. When I look at the log of the sequence that calls Job_ABC, it has an entry saying "Calling Job_ABC with parameters: x, y, z", and they seem to be correct, but when I look at the log of this particular instance of Job_ABC the entry that details the starting parameters has default values rather than the ones passed by the parent.

Yes, I'm having different Invocation Id set for each instance of the job.
No, I'm not running the same instances of the same job at the very same time.
The problem happens intermittently - sometimes (rarely) the master sequence runs OK, in other runs various instances (with different invocation id) have their parameters missing. There have been three of us looking at this issue in the last couple of days and it does seem like a weird error and we cannot find a pattern, not the cause of this issue.

Any ideas on what might be the cause of this issue will be greatly appreciated.

[chulett]

What exact 8.x version are you running?

ps. There was no code posted but I don't think we need it... yet.

[evee1]

What exact 8.x version are you running?

It's version 8.1.

[chulett]

OK. That's better than it being an 8.0.something version with all the possible bugs that brings to the table. However, for the behaviour you're describing I would open a case with your official support provider.

Do you think the 'user-defined routine' plays any role in this? When you have this problem, are all parameters defaulting in the job or just the parameter value supplied by this routine?

[evee1]

Do you think the 'user-defined routine' plays any role in this? When you have this problem, are all parameters defaulting in the job or just the parameter value supplied by this routine?

It looks like not all parameters get defaulted, only the one with the routine and the one after that. The routine is reading a value from the text file.
The file exists and the routine seem to be reading it fine, hence the log of the calling process displays the correct value.

[chulett]

And there's literally no warnings logged when this happens? Anything about the value being 'not appropriate'?

[evee1]

And there's literally no warnings logged when this happens? Anything about the value being 'not appropriate'? ...

Unfortunately not. It seems like a very unpleasant bug, I'm afraid.
I was thinking of changing the server jobs to containers just to test whether the same problem still exists, but now I don't have time to play with this issue.
I know that containers will run slower so I would not want to go down this path in the long term, but it's better if it runs slower rather than not run at all (or run incorrectly).
I (or someone else) will need to look at it though at some point in the near future.
Thanks.

[chulett]

I know that containers will run slower so I would not want to go down this path in the long term

There's nothing about having things in containers that affects speed, they're just logical blocks of code that get compiled into the job.

[gssr]

Please post the topic in appropriate forum

[chulett]

We're already well on our way, so if you want to play that card it's better to ask the Moderator to move it instead.

Moderator, please move to the Server forum.

(he really likes red)

[evee1]

We're already well on our way, so if you want to play that card it's better to ask the Moderator to move it instead.

Moderator, please move to the Server forum.

(he really likes red)

I have actually replaced all the server jobs with the parallel ones and the same thing happens, i.e. at some point one of the instances of the parallel jobs fails becuase few parameters are not being passed correctly.
Here is an example of the correctly invoked job:

Starting Job ETL_LOG_START.XYZ_DETAILS.
ORA_DSCTRL = (As pre-defined)
$DSCTRL_SID = IBMD1 [$PROJDEF] (Compiled-in default)
$DSCTRL_USER = DSCTRL [$PROJDEF] (Compiled-in default)
$DSCTRL_PWD = ******** (Compiled-in default)
TARGET_TABLE = ETL_RUN_REGISTER
SEQUENCE = ETL_RUN_REGISTER_SEQ
ENTITY = ETL_ENTITY
TARGET_DIRECTORY_PATH = /dat/target/
ETL_NAME = XYZ_DETAILS
BATCH_RUN_KEY = 24
SEQUENCE_RUN_KEY = 585
DSJobController = JOB_SEQ_XYZ_DETAILS

Here is the invocation for the failed job:

Starting Job ETL_LOG_START.ISUS0_ABC_DETAILS.
ORA_DSCTRL = (As pre-defined)
$DSCTRL_SID = IBMD1 [$PROJDEF] (Compiled-in default)
$DSCTRL_USER = DSCTRL [$PROJDEF] (Compiled-in default)
$DSCTRL_PWD = ******** (Compiled-in default)
TARGET_TABLE = ETL_RUN_REGISTER
SEQUENCE = ETL_RUN_REGISTER_SEQ
ENTITY = ETL_ENTITY
TARGET_DIRECTORY_PATH = /dat/target/
ETL_NAME =

And subsequently I'm getting an error :

Occurred: 4:07:24 PM On date: 5/11/2009 Type: Fatal
Event: ETL_LOG_START.ABC_DETAILS: Missing Parameters BATCH_RUN_KEY, SEQUENCE_RUN_KEY.

I have been runnnig it on four nodes and the number of processes was at points quite high (ca 300) - maybe it's running out of some system resources, but I don't know whether it has anything to do with the failure that I occurs.
Will try to run of 2 nodes now and see what happens.

[Kryt0n]

After a long week, I could be reading your issue completely wrong but have you tried getting your routine to send a few strings to the logs?
For your parameter to appear blank on the other side sounds like the routine is somehow bypassing the bit of code that sets the return value.

[evee1]

After a long week, I could be reading your issue completely wrong but have you tried getting your routine to send a few strings to the logs?
For your parameter to appear blank on the other side sounds like the routine is somehow bypassing the bit of code that sets the return value.

No, I haven't tried that. But I think that the routine works correctly as there is an entry in the log with the correct parameters (returned from the routine) AFTER it finishes. Then the sequnce passes these parameters to the next job (and I can see them in the sequnce og) and the job doesn't seem to be getting them (they are missing from the job log).

[chulett]

I don't see how 'nodes' play a role in how a Sequence job passes parameters, neither it nor the Routine Activity have any parallel processing in them. Sounds like it's now time to get into the gory details of your routine.

Can you post the code here now, please? And you really need to include more log entries here from the failed job run. Just showing the bad invocation isn't enough, can you also show the logs from the Routine Activity portion of that failed run? And explain exactly how your data is getting from the one routine to (apparently) three parameters in the Job Activity stage? I could guess but I'd really rather not.