DSXchange

I am trying to import a file of size 46 KB through Sequential File Stage. It has a single column 'In_Msg' with a length of 50,000 and has a final delimiter as ^ and defined in the Format tab as
Final Delimiter string = ^.

When i run the job I get the following error:

Sequential_File_0,0: Error reading on import.
Sequential_File_0,0: Consumed more than 100,000 bytes looking for record delimiter; aborting
Sequential_File_0,0: Import error at record 0.
Sequential_File_0,0: The runLocally() of the operator failed.

Is there a limit on the size of a record being read into a single field in Sequential File stage?
Is there any APT parameter that can be used to over ride this problem?

The error is that you've told DataStage to look for the record delimiter and it hasn't found it, giving up after twice the described length of your field.

What did you specify as the record delimiter character? Is this correct? Note that final delimiter is NOT the same thing as record delimiter; a final delimiter occurs between the final field and the record delimiter.

There is an environment variable to specify this

APT_MAX_DELIMITED_READ_SIZE
By default, when reading, DataStage will read ahead 500 bytes to get the
next delimiter. If it is not found, DataStage looks ahead 4*500=2000 (1500
more) bytes, and so on (4X) up to 100,000 bytes. This variable controls the
upper bound which is by default 100,000 bytes. Note that this variable
should be used instead of APT_DELIMITED_READ_SIZE when a larger
than 500 bytes read-ahead is desired.


If you have column lengths very high you can also increase APT_DELIMITED_READ_SIZE

APT_DELIMITED_READ_SIZE

By default, the DataStage will read ahead 500 bytes to get the next delimiter. For streaming inputs (socket, FIFO, etc.) this is sub-optimal, since the DataStage may block (and not output any records). DataStage, when reading a delimited record, will read this many bytes (minimum legal value for this is 2) instead of 500. If a delimiter is NOT available within N bytes, N will be incremented by a factor of 2 (when this environment variable is not set, this changes to 4).

According to advanced guide


Hope this helps,
Ajith

Hi Ray/Ajith,

Thanks for the replies. I could not find any of these APT parameters in the Environment Variables list : APT_DELIMITED_READ_SIZE & APT_MAX_DELIMITED_READ_SIZE. I also did a manual search in the DSParams file, but could not find it.

Is there anyway I can add them to the list and use them?


Thanks,

Administrator client. Job properties. Environment...

Add them in the User area.

(Or you can add them to DSParams using a text editor.)