RCP not working for Shared Container
Moderators: chulett, rschirm, roy
RCP not working for Shared Container
I am testing RCP to understand how it works.
I have the following job:
rowGenerator---->copy1---->Container---->copy2---->peek
- the rowGenerator creates 2 columns: col1 and col2
- the copy1 propagates just col1 into the Container
- the Container has a single transformer that alters the col1 and sends it on in the output link
- copy2 inputs col1 and surfaces col2 and out puts both cols to peek
if Container is local, both columns show in the peek (as expected)
if Container is shared (ie I convert the container to shared), only col1 shows in the peek
Why is this not working ?
Thanks a lot in advance,
Steve
I have the following job:
rowGenerator---->copy1---->Container---->copy2---->peek
- the rowGenerator creates 2 columns: col1 and col2
- the copy1 propagates just col1 into the Container
- the Container has a single transformer that alters the col1 and sends it on in the output link
- copy2 inputs col1 and surfaces col2 and out puts both cols to peek
if Container is local, both columns show in the peek (as expected)
if Container is shared (ie I convert the container to shared), only col1 shows in the peek
Why is this not working ?
Thanks a lot in advance,
Steve
This problem has really got me out-foxed.
With the job I mentioned in my original post:
I create the job using a Local Container and RCP works.
I then right click that container and convert it to a SHARED Container and then RCP does NOT work.
If anyone has any clues I would be most grateful.
Thanks in advance,
Steve
With the job I mentioned in my original post:
I create the job using a Local Container and RCP works.
I then right click that container and convert it to a SHARED Container and then RCP does NOT work.
If anyone has any clues I would be most grateful.
Thanks in advance,
Steve
-
- Premium Member
- Posts: 258
- Joined: Tue Jul 04, 2006 10:35 pm
- Location: Toronto
Is your project set to enable RCP by default on all new links?
Also, verify that RCP is enabled on the output link of the Shared Container as well, not just the transformer, especially if RCP is not enabled by default on new links.
Regards,
Also, verify that RCP is enabled on the output link of the Shared Container as well, not just the transformer, especially if RCP is not enabled by default on new links.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
james,jwiles wrote:Is your project set to enable RCP by default on all new links?
Also, verify that RCP is enabled on the output link of the Shared Container as well, not just the transformer, especially if RCP is not enabled by default on new links.
Regards,
its enabled for the project, but NOT enabled for new links. I have however ensured its on for all pertinent output links in the shared container and the job.
i wonder if this could be a datastage bug. i guess at this point i need to engage IBM. **sigh**
thanks again for your ideas.
steve
So, RCP is enabled on the transformer's output link as well as the output link of the shared container itself?
Does your generated OSH, or the OSH shown in the job log, have a copy operator that looks something like this?
The above is an example of where RCP is enabled everywhere except the output link of the shared container (it is enabled on the output link of the transformer within the shared container). Only col1 was kept between the shared container and the next copy.
The only way I could reproduce the problem on an 8.1 FP1 system was to have rcp disabled at some point within the job where only col1 was in the stage's output metadata...in the above case the output link of the shared container.
Set $OSH_PRINT_SCHEMAS = True to find the point at which you lose col2 in your stream.
Regards,
Does your generated OSH, or the OSH shown in the job log, have a copy operator that looks something like this?
Code: Select all
#################################################################
#### CONTAINER MAPPING - CONTAINER STAGE: sctestrcpC2 OUTPUT LINK: DSLink14
## Operator
copy
## General options
[ident('sctestrcpC2.DSLink14')]
## Inputs
0< [] 'sctestrcpC2:Transformer_4:DSLink7.v'
## Outputs
0> [modify (
keep
col1;
)] 'sctestrcpC2:DSLink14.v'
;
The only way I could reproduce the problem on an 8.1 FP1 system was to have rcp disabled at some point within the job where only col1 was in the stage's output metadata...in the above case the output link of the shared container.
Set $OSH_PRINT_SCHEMAS = True to find the point at which you lose col2 in your stream.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
james,
i am not sure what to be looking for here.
this is the osh code for the transformer (which is the only stage in the shared container):
does that look right to you?
btw, there are two columns involved:
i1 - explicitly referenced in the transformer meta data
c1 - should be propagated via RCP
thanks,
steve
i am not sure what to be looking for here.
this is the osh code for the transformer (which is the only stage in the shared container):
Code: Select all
#### CONTAINER MAPPING - CONTAINER STAGE: ContainerC1 OUTPUT LINK: xx
## Operator
copy
## General options
[ident('ContainerC1.xx')]
## Inputs
0< [] 'ContainerC1:Transformer_1:DSLink4.v'
## Outputs
0> [] 'ContainerC1:xx.v'
;
btw, there are two columns involved:
i1 - explicitly referenced in the transformer meta data
c1 - should be propagated via RCP
thanks,
steve
This part in my earlier post: is a "modify adapter", which is used to limit the output columns on a link to those listed in the stage's metadata in Designer. Maybe there's one on a different stage in your job.
Also, look at the schemas printed when using $OSH_PRINT_SCHEMAS = True. The Dataset entry for the above link appears like this:
Regards,
Code: Select all
## Outputs
0> [modify (
keep
col1;
)] 'sctestrcpC2:DSLink14.v'
Also, look at the schemas printed when using $OSH_PRINT_SCHEMAS = True. The Dataset entry for the above link appears like this:
Code: Select all
Data set "sctestrcpC2:DSLink14.v":
record
( col1: int32;
)
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
james,
i checked the OSH and found no string 'modify'.
i changed job stage names to make them more readable.
job is now like:
Row_Generator_0 -> copyBeforeContainer -> Container (TransInContainer) -> copyAfterContainer -> Peek_14
here is the 'schemas' output:
i checked the OSH and found no string 'modify'.
i changed job stage names to make them more readable.
job is now like:
Row_Generator_0 -> copyBeforeContainer -> Container (TransInContainer) -> copyAfterContainer -> Peek_14
here is the 'schemas' output:
Code: Select all
main_program: Schemas:
Data set "ContainerC1:TransInContainer:fromTransformer.v":
record
( i1: int64;
)
Data set "Row_Generator_0:toBeforeCopy.v":
record
( i1: int64;
c1: string;
)
Data set "copyBeforeContainer:toContainer.v":
record
( i1: int64;
c1: string;
)
Data set "copyAfterContainer:fromAfterCopy.v":
record
( i1: int64;
)
Operator "ContainerC1.TransInContainer":
input 0 interface:
record
( i1: int64;
APT_TRinput0Rec0: *;
)
output 0 interface:
record
( i1: int64;
APT_TRoutput0Rec0: *;
)
Operator "Row_Generator_0":
output 0 interface:
record
( i1: int64;
c1: string;
)
Operator "copyBeforeContainer":
input 0 interface:
record
( inRec: *;
)
output 0 interface:
record
( outRec: *;
)
Operator "copyAfterContainer":
input 0 interface:
record
( inRec: *;
)
output 0 interface:
record
( outRec: *;
)
Operator "Peek_14":
input 0 interface:
record
( i1: int64;
inRec: *;
)
.
I was able to recreate it by creating a new shared container while RCP was disabled on the links (when developing, I normally have RCP enabled on all links and disable it when I specifically don't want it).
I think this may be a flaw in the Designer GUI in that when you create a shared container the input link(s) to that container (from the "Input Pin") inherit the RCP setting that was in effect on those links when the container was created. Since the input pin has no editable properties, you can't change the RCP setting in the obvious way. Try the following:
Open the Shared Container for editing, then click on the Job/Container Properties icon on the menu bar.
I'm guessing that the "Enable RCP for new links" option on the General tab is unchecked. If so, check it, click on Ok and answer Yes to the "do you want to enable RCP for all existing links" message window that pops up, then save the container and recompile your job.
If the option was already checked, uncheck it, Ok, then Yes to disable, save the container then go back into properties and re-enable the option and save again.
Regards,
I think this may be a flaw in the Designer GUI in that when you create a shared container the input link(s) to that container (from the "Input Pin") inherit the RCP setting that was in effect on those links when the container was created. Since the input pin has no editable properties, you can't change the RCP setting in the obvious way. Try the following:
Open the Shared Container for editing, then click on the Job/Container Properties icon on the menu bar.
I'm guessing that the "Enable RCP for new links" option on the General tab is unchecked. If so, check it, click on Ok and answer Yes to the "do you want to enable RCP for all existing links" message window that pops up, then save the container and recompile your job.
If the option was already checked, uncheck it, Ok, then Yes to disable, save the container then go back into properties and re-enable the option and save again.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.