My job has a dataset stage, DS1, as source, a lookup stage in the middle and a sequential file stage as target. The lookup stage is receiving an input link from a lookup file set stage. I also have an output reject link to a dataset stage. What I want to do is, if there is a match on Col1 of the incoming dataset with the reference link coming from the Lookup Fileset stage, I would like Col2 from the incoming Lookup Fileset stage to replace Col2 of the incoming dataset and send all columns to the target sequential file stage. This is not happening.
My lookup fileset stage was created by using a dataset file called DS2. The only difference between DS1 and DS2 is that DS2 has actual values of Col2 which will be used to update whereas DS1 has all dummy values in Col2.
I already am. Eventhough Col2 is coming in to the lookup stage, it doesn't get moved among the output columns of that go into the sequential file stage. Col2 from the Lookup fileset stage get moved to output.
I found the solution. I used a modify operator to exclusively drop the column. Interesting. Just not linking the column in the lookup stage does not work.
