Hello
i have a job
Database->Transformer1->Shared Container-> Transformer2->file
The requirement is to pull values of two columns A and B from DB to file
Shared Container uses columns - A , D as input and A ,C , D as outputs
-Enabled Job level RCP -Yes
-Enabled RCP for shared Container - Yes
-Enabled RCP for all links - Yes
When I did a test run and took a link out of Transformer1 to a peek stage without defining columns (metadata) - I can get both A and B in the job logs
When I run the above job , I only see A,C,D as output instead of A , B,C,D
i did look for other posts in the forum but didn't find much help Not sure where I am going wrong - I appreciate your responses on this .
Update - So , I also try this - I remove the transformer 2 and have a peek stage to check and see if I am populating the Column B and yes , I DO see this in the job log along with A,C and D . However , I only want A and B to out put file so , I use the transformer , but when I do that the file only gives me the columns whose metadata is defined in Shared Container output , A C and D .
If I try to define Column B in transformer output ( Which I do not know what to map it from Shared Container output as I do not have a 'B' in the output , obviously the job doesnot compile .
Thank you
RCP with Shared Containers
Moderators: chulett, rschirm, roy
So if I understand correctly from your Shared Container output you have "surfaced" the columns A,C,D? But B remains hidden. Transformer2 would then have columns A,C,D specifically surfaced also? It sounds as though you have RCP turned off in your Transformer2 (hence why it won't compile).
Basically you need to surface column B (either in the shared container,or add a copy or modify stage) before you can turn off RCP in the subsequent stage to output only those values.
Or I have miss the point.
Basically you need to surface column B (either in the shared container,or add a copy or modify stage) before you can turn off RCP in the subsequent stage to output only those values.
Or I have miss the point.
Thank you Shane
"Shared Container output you have "surfaced" the columns A,C,D? "
The shared container takes A,D as input (defined that way ) and outputs Columns 'A,C,D'
Yeah B - remains hidden and I want it to be populated to output file but before that I want to select only column A and B (by using a transformer 2 )
"Transformer2 would then have columns A,C,D specifically surfaced also? It sounds as though you have RCP turned off in your Transformer2 (hence why it won't compile). "
Transformer 2 has input of A,C,D and output as A to file , However I also want B (which obviously is not visible unless I use a peek stage directly from output of Shared container ) and see the log . My question is - If I want to get A and B - Why do I have to enable RCP to the transformer 2 output , won't I get all the columns if I do that ?
Secondly , Even if I try to pass/map A from transformer 2 input to out link and say create a column exactly like B in the output link , I have nothing in derivation (and therefore the job won't compile )
Finally , How would I get A and B in output file ?
"Shared Container output you have "surfaced" the columns A,C,D? "
The shared container takes A,D as input (defined that way ) and outputs Columns 'A,C,D'
Yeah B - remains hidden and I want it to be populated to output file but before that I want to select only column A and B (by using a transformer 2 )
"Transformer2 would then have columns A,C,D specifically surfaced also? It sounds as though you have RCP turned off in your Transformer2 (hence why it won't compile). "
Transformer 2 has input of A,C,D and output as A to file , However I also want B (which obviously is not visible unless I use a peek stage directly from output of Shared container ) and see the log . My question is - If I want to get A and B - Why do I have to enable RCP to the transformer 2 output , won't I get all the columns if I do that ?
Secondly , Even if I try to pass/map A from transformer 2 input to out link and say create a column exactly like B in the output link , I have nothing in derivation (and therefore the job won't compile )
Finally , How would I get A and B in output file ?
You don't have to have Transformer2 set to RCP - I was just pointing out that is probably why it wasn't compiling when you tried to surface column B.
So you have two options
1. Either surface column B using a copy/modify stage prior to the Transformer2, then set the transformer to RCP off and output only Cols A and B
OR
2. As Kashyap suggests, change your transformer2 to a modify stage and specifically DROP the other columns you do not require (this is probably the better way to go - given your initial description of your requirements).
So you have two options
1. Either surface column B using a copy/modify stage prior to the Transformer2, then set the transformer to RCP off and output only Cols A and B
OR
2. As Kashyap suggests, change your transformer2 to a modify stage and specifically DROP the other columns you do not require (this is probably the better way to go - given your initial description of your requirements).