RCP not working for Shared Container

evans036 · Post by **evans036** » Thu Mar 24, 2011 10:21 am

I am testing RCP to understand how it works.

I have the following job:

rowGenerator---->copy1---->Container---->copy2---->peek

- the rowGenerator creates 2 columns: col1 and col2
- the copy1 propagates just col1 into the Container
- the Container has a single transformer that alters the col1 and sends it on in the output link
- copy2 inputs col1 and surfaces col2 and out puts both cols to peek

if Container is local, both columns show in the peek (as expected)

if Container is shared (ie I convert the container to shared), only col1 shows in the peek

Why is this not working ?

Thanks a lot in advance,

Steve

evans036 · Post by **evans036** » Fri Mar 25, 2011 7:20 am

This problem has really got me out-foxed.

With the job I mentioned in my original post:

I create the job using a Local Container and RCP works.

I then right click that container and convert it to a SHARED Container and then RCP does NOT work.

If anyone has any clues I would be most grateful.

Thanks in advance,

Steve

evans036 · Post by **evans036** » Mon Mar 28, 2011 7:53 am

has any one observed RCP to work with Share Containers?

thanks,

steve

jwiles · Post by **jwiles** » Mon Mar 28, 2011 8:17 am

RCP works just fine with shared containers. Is RCP properly enabled on all of the stages/links within the shared container?

Regards,

evans036 · Post by **evans036** » Mon Mar 28, 2011 8:24 am

James,

the shared container consists of a single transformer, I just re-verified and indeed 'runtime column propagation' IS set on the only output link.

thanks for the suggestion.

any other thoughts?

steve

samyamkrishna · Post by **samyamkrishna** » Mon Mar 28, 2011 8:40 am

The output of the Copy Stage is RCP enabled?

jwiles · Post by **jwiles** » Mon Mar 28, 2011 9:39 am

Is your project set to enable RCP by default on all new links?

Also, verify that RCP is enabled on the output link of the Shared Container as well, not just the transformer, especially if RCP is not enabled by default on new links.

Regards,

evans036 · Post by **evans036** » Mon Mar 28, 2011 12:27 pm

samyamkrishna wrote:The output of the Copy Stage is RCP enabled?

i just double-checked... Yes it is.

evans036 · Post by **evans036** » Mon Mar 28, 2011 12:32 pm

jwiles wrote:Is your project set to enable RCP by default on all new links?

Also, verify that RCP is enabled on the output link of the Shared Container as well, not just the transformer, especially if RCP is not enabled by default on new links.

Regards,

james,

its enabled for the project, but NOT enabled for new links. I have however ensured its on for all pertinent output links in the shared container and the job.

i wonder if this could be a datastage bug. i guess at this point i need to engage IBM. **sigh**

thanks again for your ideas.

steve

jwiles · Post by **jwiles** » Mon Mar 28, 2011 1:01 pm

So, RCP is enabled on the transformer's output link as well as the output link of the shared container itself?

Does your generated OSH, or the OSH shown in the job log, have a copy operator that looks something like this?

Code: Select all

#################################################################
#### CONTAINER MAPPING - CONTAINER STAGE: sctestrcpC2 OUTPUT LINK: DSLink14
## Operator
copy
## General options
[ident('sctestrcpC2.DSLink14')]
## Inputs
0< [] 'sctestrcpC2:Transformer_4:DSLink7.v'
## Outputs
0> [modify (
keep
  col1;
)] 'sctestrcpC2:DSLink14.v'
;

The above is an example of where RCP is enabled everywhere except the output link of the shared container (it is enabled on the output link of the transformer within the shared container). Only col1 was kept between the shared container and the next copy.

The only way I could reproduce the problem on an 8.1 FP1 system was to have rcp disabled at some point within the job where only col1 was in the stage's output metadata...in the above case the output link of the shared container.

Set $OSH_PRINT_SCHEMAS = True to find the point at which you lose col2 in your stream.

Regards,

evans036 · Post by **evans036** » Mon Mar 28, 2011 1:21 pm

james,

i am not sure what to be looking for here.

this is the osh code for the transformer (which is the only stage in the shared container):

Code: Select all

#### CONTAINER MAPPING - CONTAINER STAGE: ContainerC1 OUTPUT LINK: xx
## Operator
copy
## General options
[ident('ContainerC1.xx')]
## Inputs
0< [] 'ContainerC1:Transformer_1:DSLink4.v'
## Outputs
0> [] 'ContainerC1:xx.v'
;

does that look right to you?

btw, there are two columns involved:

i1 - explicitly referenced in the transformer meta data
c1 - should be propagated via RCP

thanks,

steve

jwiles · Post by **jwiles** » Mon Mar 28, 2011 3:25 pm

This part in my earlier post:

Code: Select all

## Outputs 
0> [modify ( 
keep 
  col1; 
)] 'sctestrcpC2:DSLink14.v'

is a "modify adapter", which is used to limit the output columns on a link to those listed in the stage's metadata in Designer. Maybe there's one on a different stage in your job.

Also, look at the schemas printed when using $OSH_PRINT_SCHEMAS = True. The Dataset entry for the above link appears like this:

Code: Select all

Data set "sctestrcpC2:DSLink14.v": 
record
( col1: int32;
)

Regards,

evans036 · Post by **evans036** » Mon Mar 28, 2011 4:42 pm

james,

i checked the OSH and found no string 'modify'.

i changed job stage names to make them more readable.

job is now like:

Row_Generator_0 -> copyBeforeContainer -> Container (TransInContainer) -> copyAfterContainer -> Peek_14

here is the 'schemas' output:

Code: Select all

main_program: Schemas:
Data set "ContainerC1:TransInContainer:fromTransformer.v": 
record
( i1: int64;
)
Data set "Row_Generator_0:toBeforeCopy.v": 
record
( i1: int64;
  c1: string;
)
Data set "copyBeforeContainer:toContainer.v": 
record
( i1: int64;
  c1: string;
)
Data set "copyAfterContainer:fromAfterCopy.v": 
record
( i1: int64;
)
Operator "ContainerC1.TransInContainer": 
input 0 interface: 
record
( i1: int64;
  APT_TRinput0Rec0: *;
)
output 0 interface: 
record
( i1: int64;
  APT_TRoutput0Rec0: *;
)
Operator "Row_Generator_0": 
output 0 interface: 
record
( i1: int64;
  c1: string;
)
Operator "copyBeforeContainer": 
input 0 interface: 
record
( inRec: *;
)
output 0 interface: 
record
( outRec: *;
)
Operator "copyAfterContainer": 
input 0 interface: 
record
( inRec: *;
)
output 0 interface: 
record
( outRec: *;
)
Operator "Peek_14": 
input 0 interface: 
record
( i1: int64;
  inRec: *;
)
.

jwiles · Post by **jwiles** » Mon Mar 28, 2011 6:06 pm

I was able to recreate it by creating a new shared container while RCP was disabled on the links (when developing, I normally have RCP enabled on all links and disable it when I specifically don't want it).

I think this may be a flaw in the Designer GUI in that when you create a shared container the input link(s) to that container (from the "Input Pin") inherit the RCP setting that was in effect on those links when the container was created. Since the input pin has no editable properties, you can't change the RCP setting in the obvious way. Try the following:

Open the Shared Container for editing, then click on the Job/Container Properties icon on the menu bar.
I'm guessing that the "Enable RCP for new links" option on the General tab is unchecked. If so, check it, click on Ok and answer Yes to the "do you want to enable RCP for all existing links" message window that pops up, then save the container and recompile your job.
If the option was already checked, uncheck it, Ok, then Yes to disable, save the container then go back into properties and re-enable the option and save again.

Regards,

evans036 · Post by **evans036** » Mon Mar 28, 2011 6:15 pm

James,

thankyou thankyou thankyou!!!

i opened the shared container and the 'enable rcp for new links' WAS enabled.

so i disabled it, saved the container, re-enabled it, saved the container and then recompiled my job and ran it.

AND IT WORKED.

thanks again,

steve