How to pass values between jobs?

vivek_rs · Post by **vivek_rs** » Mon Dec 20, 2004 2:14 am

Hi
I need to take in as parameter country name, do a look up on a table and return a country ID to all subsequently running jobs in a sequence.

There are a couple of values like this that I'll have to transmit.

How do I do this?

TIA

ray.wurlod · Post by **ray.wurlod** » Mon Dec 20, 2004 3:43 am

Your basic choice is how you want to write it to disk; in a text file or in a hashed file or in a database table.

Separate jobs run in separate processes. In theory you could have the one job write to a named pipe and a second job (job sequence) reading from that named pipe, but the coordination of job start times and timeouts would be horrible to manage. Named pipes on Windows are ... "forget about it!"

You could write your own routines (in C) to manage a small piece of shared memory, and store things in there. You would then need to incorporate those routines via the General Call Interface (GCI) into the DataStage executables. Yes, it's doable, but the benefit probably isn't worth the cost.

Also, if it's in a hashed file or a text file, you can use a Routine Activity to read it, and return the value to be picked up by the Job Sequence.

Use a hashed file, use a constant as the key value, and write the value you want to transmit into a non-key column. The nice thing about this approach is that you can use the same hashed files for as many different pieces of information as you want to transmit; just choose a different name for each.

vivek_rs · Post by **vivek_rs** » Mon Dec 20, 2004 6:08 am

Hey ray,
I would like to read from a simple sequential file which has a couple of values and put them as parameters into subsequent jobs in the sequence. Please help...

ray.wurlod · Post by **ray.wurlod** » Mon Dec 20, 2004 2:52 pm

In a Command activity you could use cat or head to read the file.

In a Routine activity you could invoke a routine (which you'd have to write) that uses OpenSeq, ReadSeq and CloseSeq statements to read the file.

You could customise the Job Sequence code by turning it into Job Control code in a server job, and adding statements in there to read the file.

Search the forum for examples, particularly of the third approach.

rbanavar · Post by **rbanavar** » Thu May 05, 2005 11:52 am

Good idea. Except, I would be concerned to use Head if you are working on the Ent (Parallel) Edition. I have observed inconsistent results when I used the Head stage to parse the Header of a Seq File, when we ran a parallel job. Love to hear any comments on this.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Thu May 05, 2005 2:28 pm

You can call 'head' command as an external source. This will not affect the nature of the command.

ray.wurlod · Post by **ray.wurlod** » Thu May 05, 2005 9:28 pm

Plus you labelled your post as Server and posted it on the Server jobs forum. So you got a server answer.

talk2shaanc · Post by **talk2shaanc** » Sat May 07, 2005 8:01 am

Heyi created routine, in which you can intialize a variable/value to common and then that variable can be used in the same job or across different job.

FUNCTION StoreValues(Operation,StoreValue,Identifier)

Code: Select all

 Common /SDKRowCompare/ Initialize, LastValue,SeqFile

      Equate RoutineName To 'StoreValues'


      Open "SDKSequences" TO SeqFile Else
         * Open failed. Create the sequence file.
         EXECUTE "CREATE.FILE SDKSequences 2 1 1"
         Open "SDKSequences" TO SeqFile Else Ans = -1
      End




      If Operation="GET" Then
         * Attempt to read the named record from the file.
         Readu LastValue From SeqFile, Identifier then
            Ans=LastValue
         End
      End


      * Increment the sequence value, and write back to file.	
      If Operation='STORE' Then
         Writeu StoreValue On SeqFile, Identifier Else Ans = -1
         Ans='VALUE STORED'
      End
      Close SeqFile

Operation valid values STORE or GET. STORE would store a value given for argument 'StoreValue' for a variable identified by argument 'Identifier'. and get would fetch you a value for a variable identified by argument 'Identifier'.

Say i want to store header date (01-01-2005) and i want to use this header date in other job or stages.

first in one of the transformer i will use the routine as;
StoreValues('STORE','01-01-2005','HeaderDate')

and later on where ever you want to use it call it as;

StoreValues('GET','','HeaderDate')

chulett · Post by **chulett** » Sat May 07, 2005 8:54 am

One caveat. Routines that use COMMON storage can't be used in jobs that have Buffering enabled. At least not reliably used.

Plus the fact that while you can use this in the same job to 'pass' values from one Transformer to another, I'm not sure it's valid to say that it would work 'across different jobs'.

I'd have to go dig though the docs...

ray.wurlod · Post by **ray.wurlod** » Sat May 07, 2005 5:45 pm

Variables declared to be in COMMON definitely can not be used across jobs. Indeed they are limited to one process, which means that they aren't even shared between the job process and a subordindate Transformer stage process. And, unless the Transformers are directly connected (so that they are executed by the same process), variables in COMMON aren't even shared between two processes executing different Transformer stages in the same job.

dstechdev · Post by **dstechdev** » Sat May 07, 2005 9:08 pm

I did something similiar to this but I broke it up into two routines. The first routine stored the keyed value in one job and the second routine retrieved it in a subsequent job

john

ds_developer · Post by **ds_developer** » Mon May 09, 2005 11:15 am

As you can see from the code, he is actually reading and writing to a file. This would allow access from other jobs, stages, etc.

John

talk2shaanc · Post by **talk2shaanc** » Mon May 09, 2005 10:41 pm

Ya exactly I am actually using a Universe file 'SDKSequence', which is used by In-Built, transform-'KeyMgtGetNextValue' and 'KeyMgtGetNextValueConcurrent'.