It has to read the metadata of a file at run time.
It will contain different keys for different file definitions.
Then it has to perform the aggregations at runtime based on the file definitions.
The field to which the aggregation has to be applied is also dynamic.
We have planned to design this component first by creating dynamic schema files which should be read in a custom stage.
Then the custom stage has to concatenate all the key fields and create one generic key which will be used for aggregation.
Can somebody throw some light on the following points
assuming that we receive a CSV input file?)1) How to generate dynamic schema files based on the different input files we receive.
I have gone through parallel developer guide to create a schema file which can be done when we know the format. But we cannot create it at run time by seeing the input file metadata.
2) How can we concatenate only key fields if their position and number is different for each input file?
For example in first case input file may contain 5 columns say A, B, C, D, E out of which C and D can be the key columns.
In second case input file may contain 10 columns M,N,O,P,Q,R,S,T,U,V Out of which M,P,S and U can be key columns.
