Setting Job Variables in a stage and accessing it in another

[manish1005]

hi all,

What I am trying to do is, just reading a max value from ODBC stage, setting a Job variable to it in one stage and then using that Job Variable in some other stage.


I am setting a Job variable in a transformer stage through a simple routine - setID(Arg1) :

Code: Select all

 max_tid = Arg1;
       Ans = max_tid;

max_tid is defined as a Job Variable in Job Properties window.

In the transformer stage, I am calling this routine while reading a single row from ODBC stage and passing that as an argument to this routine.
So ideally, that input should be set to the max_tid job variable.

But its not happening! I am using max_tid value in another transformer stage but its not giving me the changed value, instead its giving me the default value that I give before execution starts.

Please suggest.

[DSguru2B]

True. This is because stage variables are stage specific. You need to set the user status in your first transformer and retrieve it in the second. Search for setuserstatus. Its been covered a lot.

[chulett]

Well... you would do that to pass data between jobs. Usually, to pass something between stages, all you need to do is add that new field to the link between the two stages. Why make this more difficult than it needs to be?

The other option is 'COMMON' storage. Or one of those five available @USERn variables. Search the forum.

[DSguru2B]

Well... you would do that to pass data between jobs. Usually, to pass something between stages, all you need to do is add that new field to the link between the two stages. Why make this more difficult than it needs to be?

The other option is 'COMMON' storage. Or one of those five available @USERn variables. Search the forum.

Yea that too

[ray.wurlod]

The scope of a stage variable is within a stage. The scope of a job variable is within a job (though there's really no such thing as a job variable). The scope of a COMMON variable is within a single process.
You could use the job's user status area to store a value, and retrieve it subsequently using a routine that calls DSGetJobInfo().

[manish1005]

>Well... you would do that to pass data between jobs. Usually, to pass something between stages, all you need to do is add that new field to the link between the two stages. Why make this more difficult than it needs to be?

Actually, I can not use link for passing this max_tid value because I have to pass this value between two *transformer* stages, so second transformer stage treats the link as a *lookup*.

I searched some forum postings for COMMON, it seems that COMMON is used for passing values between jobs like in a sequence, my problem is passing some values between two transformer stages in the same job.

Ray:
>The scope of a job variable is within a job (though there's really no such thing as a job variable)
By job variable I mean job parameters set from Job properties window.

[ray.wurlod]

Job parameter values can not be changed once the job is running.

[manish1005]

Job parameter values can not be changed once the job is running.

oh! then there must be someway out, I mean if I have more than one such variables then I can not use DSSetUserStatus.

[rleishman]

oh! then there must be someway out, I mean if I have more than one such variables then I can not use DSSetUserStatus.

Yes, and it was given to you above. You declare persistent variables in a routine using COMMON. You set the variables with one routine call from the first transformer, and fetch it back with another routine call from the second transformer. The @USER* variables work similarly, except you don't need a routine to retrieve them - you can reference them directly in any expression.

It might be a problem if you want it to process all of the rows, calculate the variable, and then use the variable against all of the rows in the second transformer. DS passes rows through transformer pipelines in buffers - it doesn't process all rows through transfomer 1, then all rows through transformer 2.

If that's what you want, you can separate the two transformers with a Sequential File stage.

[ArndW]

Remember that if you use the COMMON block method as described above you will need to ensure that you declare your job to not use row buffering or use in-process buffering. If you do not do this, the separate TRANSform stages are started as separate processes and thus won't share the same COMMON blocks.

[chulett]

Actually, I can not use link for passing this max_tid value because I have to pass this value between two *transformer* stages, so second transformer stage treats the link as a *lookup*.

This confuses me. I specifically meant add it to the link between the two *transformer* stages, that's what links do, pass values. And a transformer can't act as a 'lookup' so double lost here.

There are ways to 'look something up' once and then include it on every row - the Initial Value of a stage variable works wonders for that, for instance. Not sure if that would help you as your true needs are still far from clear.

I searched some forum postings for COMMON, it seems that COMMON is used for passing values between jobs like in a sequence, my problem is passing some values between two transformer stages in the same job.

No, you misunderstood. That's exactly what it is for, establishing common storage between transformers in a single job. Within the limits that Arnd notes.

There are routines in the 'sdk' category RowProc which leverage COMMON storage, check them to get some ideas on how or if that technique would work for you in your situation.