Aggregator and (maybe) partial schema
Posted: Thu May 26, 2011 9:06 pm
I have searched through the forum and DS documentation but I couldn't find the similar example
I have a parallel job that reads the source file using the Sequential File stage with one varchar column, then splits the records into fields using
Column Import stage according to the dynamically passed schema file.
The source file is has comma separated variable length fields.
I managed to define a schema and I works OK.
Now, I would like to calculate the sum of one column, that I know the position of, for example the 5th field in the file. Is it possible to use aggregator for this?
How do I actually define a calculation to use if I cannot see the fields? The same question stands for a transformer.
I have experimented a bit with a partial schema, just to read the field I am interested in, but cannot make it work so far.
I have a parallel job that reads the source file using the Sequential File stage with one varchar column, then splits the records into fields using
Column Import stage according to the dynamically passed schema file.
The source file is has comma separated variable length fields.
I managed to define a schema and I works OK.
Now, I would like to calculate the sum of one column, that I know the position of, for example the 5th field in the file. Is it possible to use aggregator for this?
How do I actually define a calculation to use if I cannot see the fields? The same question stands for a transformer.
I have experimented a bit with a partial schema, just to read the field I am interested in, but cannot make it work so far.