Seggregating Input based on the input column
Posted: Fri Jan 28, 2005 2:36 pm
Hi,
I have a requirement like this:
My input is a few million rows. The file sometimes contain only one day's data and some other times, it has two day's data. So, based on the date column in the input row, I want to direct it to one of my two output links.
I can read the entire file before I start the DS job to find out the dates it contains and pass them as parameters to the job, so that I can compare each of the incoming rows against these job-parameters to direct them accordingly. But the input file is too huge and so am looking for an alternate way other than reading the file twice. So, are there any other ways to do this comparison in the transfomer itself?
Can I store the data that I read from the file into some global variables (not stage variables which gets over-written as I read each row)?
Thanks in advance!
Venkatesh
I have a requirement like this:
My input is a few million rows. The file sometimes contain only one day's data and some other times, it has two day's data. So, based on the date column in the input row, I want to direct it to one of my two output links.
I can read the entire file before I start the DS job to find out the dates it contains and pass them as parameters to the job, so that I can compare each of the incoming rows against these job-parameters to direct them accordingly. But the input file is too huge and so am looking for an alternate way other than reading the file twice. So, are there any other ways to do this comparison in the transfomer itself?
Can I store the data that I read from the file into some global variables (not stage variables which gets over-written as I read each row)?
Thanks in advance!
Venkatesh