hi there
i am relatively new over here, i have come across a very unique problem. what if the format of the sequential file which i read from the remote side changes, for example if i get like 90 coloums today, there is a chance tommorow i can get 95 or 85.
is there a work around for this situation or its not possible in DS.
Any buyers?
What if the File Formats changes
Moderators: chulett, rschirm, roy
-
Sainath.Srinivasan
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
Handled such scenario several times. You can handle these situations by various methods using DataStage.
One simple method is to define the whole record as a single field and split it depending on the requirement.
Other is to accept file with incomplete columns.
But the main question is - if you have different number of columns, how can you know which value represents which field? Will it be part of the data or will you have some logic to derive it.
One simple method is to define the whole record as a single field and split it depending on the requirement.
Other is to accept file with incomplete columns.
But the main question is - if you have different number of columns, how can you know which value represents which field? Will it be part of the data or will you have some logic to derive it.
I faced similar situation where a single file contained some records with X columns, some with Y and ome with Z columns. I had to move the reocrds of different schema into different files.
so i kept a special character as the field delimiter, read the whole record as field and then dependign on the number of delimiters (the ones on which the file was really delimited) the main file was split into different files. however i could do this as i knew taht if it has X columns then which column is what and if it has Y columns then which comuns is what.
So as Sai said you must know which value reprensents which field.
so i kept a special character as the field delimiter, read the whole record as field and then dependign on the number of delimiters (the ones on which the file was really delimited) the main file was split into different files. however i could do this as i knew taht if it has X columns then which column is what and if it has Y columns then which comuns is what.
So as Sai said you must know which value reprensents which field.
well actually i tell you the whole scenario, i am suppose to get the payment schedule of a Customer, now a customer can be in any month of his lease, what the front end does, is sends me a record of the persons information with his payment history too.
so the no. of coloums keep fluctuating. i suggested to create a new file format, bu the frontend guy is like thats the way we do it.
so the no. of coloums keep fluctuating. i suggested to create a new file format, bu the frontend guy is like thats the way we do it.
I think someone will definately need atleast the field in the database table to which the column (85th or 95th) will go to, to figure out how to read the inbound data.
May be you can try to see how they were fetching all the columns in their fields before they got DataStage and probably try to use that same logic.
Sachin
May be you can try to see how they were fetching all the columns in their fields before they got DataStage and probably try to use that same logic.
Sachin
samunik wrote:well actually i tell you the whole scenario, i am suppose to get the payment schedule of a Customer, now a customer can be in any month of his lease, what the front end does, is sends me a record of the persons information with his payment history too.
so the no. of coloums keep fluctuating. i suggested to create a new file format, bu the frontend guy is like thats the way we do it.