Page 1 of 1
Wrapper Stage
Posted: Wed Apr 25, 2007 3:34 am
by pravin1581
Hi All,
I have used a new parallel stage- wrapper that calls a C program , takes its input from the sequential file stage preceding it and produces the output in another file. This stage runs fine when in a stand alone job. But when i integrate it in another job with the output from a sort stage as an input to the wrapper stage then it is not working . The job performance monitor shows that from sort stage X no. of rows are entering and 0 rows are getting out of the wrapper stage . i.e. the wrapper stage is not working and the output file has no record. Can anyone provide a solution , may be wrapper stage expects input from a files stage only.
Posted: Wed Apr 25, 2007 7:10 am
by DSguru2B
Is there any error message in the log?
Posted: Wed Apr 25, 2007 8:17 am
by rameshrr3
I have used wrapped stages with Non -File inputs. Dont think that should be the issue.
Thanks
Ramesh
Posted: Wed Apr 25, 2007 11:30 pm
by pravin1581
DSguru2B wrote:Is there any error message in the log?
There is no error message in the log. It is only showing export complete .0 rows exported successfully.
Posted: Wed Apr 25, 2007 11:33 pm
by pravin1581
rameshrr3 wrote:I have used wrapped stages with Non -File inputs. Dont think that should be the issue.
Thanks
Ramesh
In my case the wrapper stage has its input from the sort stage.
Posted: Thu Apr 26, 2007 4:45 am
by nick.bond
Only an idea but try adding the $APT_DISABLE_COMBINATION environment variable to the job, setting it to True.
Posted: Fri Apr 27, 2007 3:58 am
by pravin1581
nick.bond wrote:Only an idea but try adding the $APT_DISABLE_COMBINATION environment variable to the job, setting it to True.
How will the $APT_DISABLE_COMBINATION variable affect the wrapper stage , I was going thru the manuals and they are clearly stating that not to change the $APT_DISABLE_COMBINATION variable as it may affect the performance of the job.
Posted: Fri Apr 27, 2007 4:08 am
by nick.bond
EE will combine oporators in the job to improve performance, I can't see how it would try and combine a sort state and your rapper stage, but if you can see the data going into the sort stage and not going from your sort to your rapper stage i would just add $APT_DISABLE_COMBINATION = True as a means to help debug.
I don't really expect this to fix the problme but whenever somthing is happening in and EE job that can't be explained it is one option to help debug. Why not just give it a go?
I have had a job that had to have this set otherwise it wouldn't run on a single node. It is not something that "should never be used", you should use it when it helps.
Posted: Fri Apr 27, 2007 4:23 am
by pravin1581
nick.bond wrote:EE will combine oporators in the job to improve performance, I can't see how it would try and combine a sort state and your rapper stage, but if you can see the data going into the sort stage and not going from your sort to your rapper stage i would just add $APT_DISABLE_COMBINATION = True as a means to help debug.
I don't really expect this to fix the problme but whenever somthing is happening in and EE job that can't be explained it is one option to help debug. Why not just give it a go?
I have had a job that had to have this set otherwise it wouldn't run on a single node. It is not something that "should never be used", you should use it when it helps.
Ok I will use it in my job.
Posted: Fri Apr 27, 2007 5:59 am
by pravin1581
pravin1581 wrote:nick.bond wrote:EE will combine oporators in the job to improve performance, I can't see how it would try and combine a sort state and your rapper stage, but if you can see the data going into the sort stage and not going from your sort to your rapper stage i would just add $APT_DISABLE_COMBINATION = True as a means to help debug.
I don't really expect this to fix the problme but whenever somthing is happening in and EE job that can't be explained it is one option to help debug. Why not just give it a go?
I have had a job that had to have this set otherwise it wouldn't run on a single node. It is not something that "should never be used", you should use it when it helps.
Ok I will use it in my job.
The property has been set to TRUE , but the job log is hard to debug.
Posted: Fri Apr 27, 2007 6:28 pm
by ray.wurlod
Read each warning carefully - the name of the stage and explanatory text is given, often with other diagnostic information. Why not post one of the warnings, together with your understanding of what it is telling you? If you have version 8.0 there is also an error codes reference manual, though this is incomplete.
Posted: Sun Apr 29, 2007 10:48 pm
by pravin1581
ray.wurlod wrote:Read each warning carefully - the name of the stage and explanatory text is given, often with other diagnostic information. Why not post one of the warnings, together with your understanding of what it is telling you? If you have version 8.0 there is also an error codes reference manual, though this is incomplete.
Here is the entry from the job log where Parallel Job Initiated information is given.....
Code: Select all
STAGE: Median_19
## Operator
Median.op
## General options
[ident('Median_19'); jobmon_ident('Median_19')]
## Inputs
0< [] 'sort_UPC_Org:ln_ip_med_file.v'
## Outputs
0> [modify (
Org:ustring[max=255]=Org;
)] 'Median_19:ln_op_Median.v'
;
The Stage : Median_19 is the wrapper stage used in the job. It takes its input from the SORT stage and produces the output in the sequential file.
Posted: Mon Apr 30, 2007 12:57 am
by ray.wurlod
No it doesn't - it produces its output into the virtual Data Set whose control file name is 'Median_19:ln_op_Median.v'
What you have posted is a small piece of the generated OSH. While it is true that this is logged, it is NOT the event that reports any error.
Is there any event in the log with a severity of Warning (yellow icon) or Error (red icon)? If so, please post the detail of these events. If not, your job is working perfectly (or you've disabled warnings, which is always done at your own peril).