setting User Status in a parallel job
Moderators: chulett, rschirm, roy
setting User Status in a parallel job
I've searched for quite a bit, and I've seen a lot of references to setting User Status in a parallel job (for retrieval from a job sequence), but nothing about the mechanics of doing this.
How's it done?
I have a job where a stored procedure stage returns a run number that needs to be passed to other jobs later in the sequence, and I think the User Status would be the easiest way, though I'm open to suggestions.
By the way - I can't just dump it to a file and then read it from the sequence because the solution has to be generic enough to add to a lot of sequences without modification.
How's it done?
I have a job where a stored procedure stage returns a run number that needs to be passed to other jobs later in the sequence, and I think the User Status would be the easiest way, though I'm open to suggestions.
By the way - I can't just dump it to a file and then read it from the sequence because the solution has to be generic enough to add to a lot of sequences without modification.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The user status value is stored in one of the RT_... hashed files in the local DataStage repositories. I have not been able to find anything in the DataStage C API for accessing it. Would a BASIC Transformer be really that bad?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
*Gasps in HoRRoR!*
Did I just hear Ray "Stay away from the Basic Transformer" Wurlod ask me if "using a Basic Transformer would be so bad"?
I guess not....
I'm also looking at the "triggers" section of a transformer stage. It seems to allow setting some sort of user variables, though I've not done it before.
I'll post what I've tried later on.
Thanks!
Did I just hear Ray "Stay away from the Basic Transformer" Wurlod ask me if "using a Basic Transformer would be so bad"?
I guess not....
I'm also looking at the "triggers" section of a transformer stage. It seems to allow setting some sort of user variables, though I've not done it before.
I'll post what I've tried later on.
Thanks!
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
My doctor told me to stay away from cigarettes, so I bought a cigarette holder.
In general avoid BASIC Transformer because of their overhead, but if there's no other way... (Like avoiding sorting unless it's necessary.)
The Variables you found in the stage properties are actually the stage variables, so that won't help. Triggers are additional pieces of C++ code that are fired because of processing in the Transformer stage; unless you can find a C-based technique for updating the user status area then that won't help either.
(You could, of course, create some C code that uses UVread and UVwrite via system calls to update the record containing the job's user status area, but that won't give you much in the way of performance.)
In general avoid BASIC Transformer because of their overhead, but if there's no other way... (Like avoiding sorting unless it's necessary.)
The Variables you found in the stage properties are actually the stage variables, so that won't help. Triggers are additional pieces of C++ code that are fired because of processing in the Transformer stage; unless you can find a C-based technique for updating the user status area then that won't help either.
(You could, of course, create some C code that uses UVread and UVwrite via system calls to update the record containing the job's user status area, but that won't give you much in the way of performance.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Hi Andy,
What did you end up doing?
I have a similar issue. I'm implementing a generic sequence to cater for various steps of file validation. One of them is a routine that verifies the total of a given source field in a file against the total given in a footer.
I need to pass the result of a validation back up to the parent sequence and I was planning to set the UserStatus in a BASIC transformer and then query the status in a sequence.
I can't test it though as there is no DSSetUserStatus routine available in my temporary DS installation (I'm waiting for a proper environment to be installed).
Also what did you do with the output from the BASIC transformer? Did you need it for anything. I don't, as I only care about comparing the stats and returning the result. So in my job I have a BASIC transformer stage that calls (or rather will call, once it is actually available ) DSSetUserStatus routine, but then I need some output from the transformer. Otherwise the job won't compile. I created a dummy outpt Seq file, but I'm not really fond of it.
What did you end up doing?
I have a similar issue. I'm implementing a generic sequence to cater for various steps of file validation. One of them is a routine that verifies the total of a given source field in a file against the total given in a footer.
I need to pass the result of a validation back up to the parent sequence and I was planning to set the UserStatus in a BASIC transformer and then query the status in a sequence.
I can't test it though as there is no DSSetUserStatus routine available in my temporary DS installation (I'm waiting for a proper environment to be installed).
Also what did you do with the output from the BASIC transformer? Did you need it for anything. I don't, as I only care about comparing the stats and returning the result. So in my job I have a BASIC transformer stage that calls (or rather will call, once it is actually available ) DSSetUserStatus routine, but then I need some output from the transformer. Otherwise the job won't compile. I created a dummy outpt Seq file, but I'm not really fond of it.
I once attempted, poorly, to accomplish this. Did not use it and have not gotten a chance to enhance it either. You are more than welcomed to see if it can provide a starting point. The code lies here.
I would advise to get rid of the malloc() command as it provided a lot of grief in pxEreplace().
I would advise to get rid of the malloc() command as it provided a lot of grief in pxEreplace().
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
I tried to use a Routine to set the UserStatus variable of a PX-Job from within a BASIC-Transformer. It just did not work.
Not only do you get a warning in the job-log:
(BASIC_Transformer_14).#0.BASIC_Transformer_14 (DSSetUserStatus): Error reading Job Status record. JobNo = 2336 JobName = Test_CDC.#0 WaveNo = 1.
The UserStatus variable remains empty for the job, so follow-up jobs do not get their parameter-values. Unfortunately it looks like You can only make use of the UserStatus variable from within Server jobs.
Not only do you get a warning in the job-log:
(BASIC_Transformer_14).#0.BASIC_Transformer_14 (DSSetUserStatus): Error reading Job Status record. JobNo = 2336 JobName = Test_CDC.#0 WaveNo = 1.
The UserStatus variable remains empty for the job, so follow-up jobs do not get their parameter-values. Unfortunately it looks like You can only make use of the UserStatus variable from within Server jobs.
"It is not the lucky ones are grateful.
There are the grateful those are happy." Francis Bacon
There are the grateful those are happy." Francis Bacon
Well, I did use a couple of suggestions here to build a work-around. Seems you *can* call a Server Routine and/or access a Server Hash from within a Parallel Job after all.
Absolutely a work-around only (as it relies on current Hash structures), but it gets us past this particular problem.
Absolutely a work-around only (as it relies on current Hash structures), but it gets us past this particular problem.
Michael
Magma Computing Solutions
Magma Computing Solutions
A couple of people have asked what we did... as I said prev. it is in kludge to fit an existing need. If we were designing from scratch, we'd go another way. Also it was an exercise in "can we do it?"
<Before>
SourceStage ----> <Transformer> ----> SequentialFile
written as a Server Job. The transformer would call a user-written routine which invoked DSSetUserStatus. The last row down the Source link would become the actual Job's $UserStatus.
The Sequential file was an append to /dev/null (or NUL if DOS).
<After>
SourceStage ----> ServerSharedContainer
written as a Parallel Job, using a Sever Shared Container with 1 input field. Set the Advanced Properties to 'Sequential' (to avoid Parallel nodes). The Shared Container has the following.
InputLink -----> Transformer -------> SequentialFile
Again, the Sequential File is /dev/null. The Basic Transformer calls user-written code which *pokes* the supplied value directly into the Job's control file (as a Hash File). We couldn't use DSSetUserStatus due to a known problem with Parallel Jobs.
Might look messy, but this was quite useful. The Server Shared Container gave us the ability to call BASIC routines (or use Hash Files). Because it was Shared, we were able to plug it in to a number of other Jobs.
What do you think?
<Before>
SourceStage ----> <Transformer> ----> SequentialFile
written as a Server Job. The transformer would call a user-written routine which invoked DSSetUserStatus. The last row down the Source link would become the actual Job's $UserStatus.
The Sequential file was an append to /dev/null (or NUL if DOS).
<After>
SourceStage ----> ServerSharedContainer
written as a Parallel Job, using a Sever Shared Container with 1 input field. Set the Advanced Properties to 'Sequential' (to avoid Parallel nodes). The Shared Container has the following.
InputLink -----> Transformer -------> SequentialFile
Again, the Sequential File is /dev/null. The Basic Transformer calls user-written code which *pokes* the supplied value directly into the Job's control file (as a Hash File). We couldn't use DSSetUserStatus due to a known problem with Parallel Jobs.
Might look messy, but this was quite useful. The Server Shared Container gave us the ability to call BASIC routines (or use Hash Files). Because it was Shared, we were able to plug it in to a number of other Jobs.
What do you think?
Michael
Magma Computing Solutions
Magma Computing Solutions
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Once you know the location of the user status area - I'm sure I've documented this somewhere on DSXchange - you can write a parallel routine that uses InterCall functions ic_open, ic_recordlock, ic_writev and ic_close to update that particular field (field #9 in one of the RT_STATUSnnn records).
You can find the InterCall Developer's Guide in any set of UniVerse manuals, whether from IBM or Rocket Software, for example here. InterCall has not changed much in the last 20 years.
You can find the InterCall Developer's Guide in any set of UniVerse manuals, whether from IBM or Rocket Software, for example here. InterCall has not changed much in the last 20 years.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.