Complex Flat File stage - variable record lengths
Posted: Tue Sep 06, 2011 7:30 pm
I'm having a problem variable record lengths on the CFF stage.
I think my problem is the same as this thread, but I don't think respondants there really understood the problem:
viewtopic.php?p=391304
I have several different record types and I have the meta data defined for each record type. All records end with a Unix newline.
The problem is that some of the fields on the end of some records are optional and as a result, some input records end before the end of the definition. In this case, these records are rejected with an input buffer overrun error.
It doesn't seem to matter how I define the fields that are missing from the input record (nullable or set a default), I still get the error.
Whether or not the optional fields exist does not depend on the record type.
For example, if I have record types A, B and C.
Record types A and B are always filled and work fine. Record type C will be rejected if the record is shorter than the metadata.
If I remove extra fields from the definition of record tye C, then these records no longer give an error. As soon as I define even a single field that goes past the physical end of the shortest record, these shorter records are rejected. The longer type C records still come through.
Is there a solution to this or is this a limitation of the CFF stage?
I think my problem is the same as this thread, but I don't think respondants there really understood the problem:
viewtopic.php?p=391304
I have several different record types and I have the meta data defined for each record type. All records end with a Unix newline.
The problem is that some of the fields on the end of some records are optional and as a result, some input records end before the end of the definition. In this case, these records are rejected with an input buffer overrun error.
It doesn't seem to matter how I define the fields that are missing from the input record (nullable or set a default), I still get the error.
Whether or not the optional fields exist does not depend on the record type.
For example, if I have record types A, B and C.
Record types A and B are always filled and work fine. Record type C will be rejected if the record is shorter than the metadata.
If I remove extra fields from the definition of record tye C, then these records no longer give an error. As soon as I define even a single field that goes past the physical end of the shortest record, these shorter records are rejected. The longer type C records still come through.
Is there a solution to this or is this a limitation of the CFF stage?