Variable Column Set in File
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 39
- Joined: Tue Apr 15, 2014 9:14 am
Variable Column Set in File
I have a file coming in which may have different column sets on different days. It has a header row, so it is possible to determine which columns exist on a given day, but I have not been able to find a way of getting the incoming file stage (HDFS stage) to understand the header and map the input columns to job columns. Is this possible?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Create a preliminary job that reads the header file and creates a schema file (so that the Sequential File stage can use RCP to read the file) and also creates a Modify stage specification to translate from today's columns to the "official" job columns. Provide the Modify stage specification to the real job as a job parameter, perhaps from the User Status area of the preliminary job.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.