Setting up RCP job

iq_etl · Post by **iq_etl** » Mon Mar 03, 2014 11:31 am

We don't have RCP set up for any of our projects or jobs, but since we've got a little downtime, we're interested in understanding the concept...but haven't had much success.

So, in the Administrator I have a test project set to 'Enable Runtime Column Propagation for Parallel jobs'.

In a test job for the same project, I've set 'Enable Runtime Column Propagation for new links'.

My understanding is that with an Oracle Connector stage as the source, we can set this up to be dynamic (variable table name, not provide columns) and that with RCP, the columns of the table name given will ripple through the project.

This seems to work with a straight move from a target to source, but when I look to make a change to one column (in a transformer stage, and only including that column), then just that one column gets written to a target sequential file. I guess I understood that the other columns, not in the transformer, would be written to the sequential file unaltered.

Where is the gap in my knowledge? How do i get the job to take in any table name and write out all of the columns whether I specify a column subset in a middle stage or not?

After I get a successful test case from an Oracle Connector source, I'd like to try using a Sequential File as a source and load a table in an Oracle Connector. I understand that'll require defining part of the schema, which I'll likely have questions on as well.

ray.wurlod · Post by **ray.wurlod** » Mon Mar 03, 2014 4:09 pm

Check that RCP is enabled on the output link from the Transformer stage.

iq_etl · Post by **iq_etl** » Tue Mar 04, 2014 2:17 pm

Ray,

I've got the RCP box checked on the columns tab of the Oracle Connector.

In the Transformer, when I select 'stage properties' (upper left) the RCP option is checked on the Output links tab. _However_, when I go into the transformer pane and into the 'link properties' of the output link, the RCP option doesn't stay checked. I would think it would retain the check from the 'stage property', and if not, at least retain it from the link properties, but it seems to reset to unchecked every time I close out the link property window.

In the target sequential file stage, only that one column I want to modify appears, and I see nothing in that stage for RCP. So, again only that one column given in the transformer (which also appear in the column tabs of the source Oracle Connector and target Sequential File) appears to be written and the other columns from the source table are missing.

Please advise.

ray.wurlod · Post by **ray.wurlod** » Tue Mar 04, 2014 7:55 pm

Check that RCP is enabled on the output link from the Transformer stage.

This is not the same as the stage properties setting - the Transformer supports multiple output links - the stage property asserts the setting for newly-added output links.

iq_etl · Post by **iq_etl** » Wed Mar 05, 2014 11:32 am

Ray, I believe I understand this.

* open the Transformer stage
* right click on the appropriate output like and select 'link properties'
* this opens the Transformer Stage Properties window with the relevant output link tab opened
* check 'Runtime column propagation' in the bottom left which should set RCP for the output link referenced in the 'output name' (also the default output link when opened).

However, whenever I do this, the check doesn't seem to stick. As soon as I close the window, the RCP checkbox seems to 'uncheck'.

When I add a second output link, the RCP checkbox defaults as checked...but when I perform the above steps on that new output link, RCP is unchecked.

Going back to the first output link, opening the link properties, selecting the second, new, link in the dropdown, I see RCP checked again.

I deleted the first output link, and now what was the second link with checked RCP is now the sole link, but RCP is no longer checked. I can't seem to make rhyme or reason as to what's going on here.

Is there another way to enable RCP on the output link? Exiting out of the Transformer, there doesn't seem to be any sort of RCP action I can take on the link in the parallel job designer pane.

Thanks for your patience!

IBM Analytics Champion 2009 - 2020 · Post by **asorrell** » Wed Mar 05, 2014 4:05 pm

Not certain - but this may have to do with you needing an option to define a partial schema for that second link. Look up "Partial Schemas", specifically the intact option for the sequential file stage in the Parallel Job Developers Guide.