I have a job to take data from source ,lookup on a dataset,if no corresponding entry is present,append that record to same dataset by assigning an unique index to it so that the dataset have unique records..
My job looks as
Code: Select all
dataset
|
seq_file---->lookup---->dataset
Operator initialization: A link between two operators should be named with a .v; insert a copy operator to save a persistent copy of the data
I read some posts and understood that same dataset cannot be used more than once in a job and instead can use virtual dataset.
I am unaware of how to create virtual dataset and please let me know how can i implement it in the above scenario.