I would like to use a variable that can be referenced or modified between rows that are processed by a job. Does the concept of persistent variables exist in DataStage?
Thanks,
Peter
Do Persistent Variables exist in DataStage 6?
Moderators: chulett, rschirm, roy
Sure. As long as you are staying within a single Transfomer - Stage Variables should fill the bill. Check your online help.
Across Transformers in a Job you need to write custom routines that leverage COMMON storage. Again, read up on them in your documentation as there are some pretty big caveats to using this approach.
Across Transformers in a Job you need to write custom routines that leverage COMMON storage. Again, read up on them in your documentation as there are some pretty big caveats to using this approach.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The easiest and cheapest "persistent variable" for use between adjacent Transformer stages (or any active stage types) is an extra column on the link.
The job itself has a "user status area" which acts like a single variable accessible from anywhere within the job. Its value is set using the DSSetUserStatus function, and retrieve using the DSGetJobInfo function (with DSJ.ME as the job handle if in the current job). Actually, the job's user status area is in fact disk-based, so that it persists even after the job has finished.
Variables defined to be in a COMMON area are only accessible within the same process. Therefore, if you place a process boundary (a passive stage type or an inter-process stage) between your two active stages, they will run in separate processes. While this can achieve throughput gains due to pipeline parallelism, it means that you can't use COMMON variables.
The cost of using extra columns on links is miniscule. Go that way.
The job itself has a "user status area" which acts like a single variable accessible from anywhere within the job. Its value is set using the DSSetUserStatus function, and retrieve using the DSGetJobInfo function (with DSJ.ME as the job handle if in the current job). Actually, the job's user status area is in fact disk-based, so that it persists even after the job has finished.
Variables defined to be in a COMMON area are only accessible within the same process. Therefore, if you place a process boundary (a passive stage type or an inter-process stage) between your two active stages, they will run in separate processes. While this can achieve throughput gains due to pipeline parallelism, it means that you can't use COMMON variables.
The cost of using extra columns on links is miniscule. Go that way.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Thanks for the feedback.
Just to clarify - I would like to pass the values between rows, not between stages. For example, let's say I want to detect whether I've received 5 or more records with a specific value in a given column. Each time I detect that value in the incoming data, I would like to increment a counter by 1. After processing 100 rows, for example, my counter may have a value of 0 to 100, depending on how may rows matched the criteria I'm providing.
Just to clarify - I would like to pass the values between rows, not between stages. For example, let's say I want to detect whether I've received 5 or more records with a specific value in a given column. Each time I detect that value in the incoming data, I would like to increment a counter by 1. After processing 100 rows, for example, my counter may have a value of 0 to 100, depending on how may rows matched the criteria I'm providing.
Only if you tell it to. 
It's simple enough to keep its current value by assigning it to itself. For example, with one called MyVariable, use something like this in its derivation to only increment it when certain conditions are met:
That make sense? Stage Variables are very powerful, you'll find them very handy.
It's simple enough to keep its current value by assigning it to itself. For example, with one called MyVariable, use something like this in its derivation to only increment it when certain conditions are met:
Code: Select all
If X Then MyVariable + 1 Else MyVariable-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers