Maximum number of bytes in a record in flat file

Jayakannan · Post by **Jayakannan** » Thu Jun 06, 2013 6:15 am

Hi,

I am trying to create the fixed length .txt file where record length is 4096 bytes/characters and with this length the file is showing wrong record count in that file.

I reduced the record length to around 2500 bytes and then it is showing correct record count in that file.

Is there any limitation in the maximum number of bytes a record can have?

chulett · Post by **chulett** » Thu Jun 06, 2013 7:01 am

Not that I'm aware of. That 4K sounds like a limitation on a single field size - is your record one long CHAR field?

Jayakannan · Post by **Jayakannan** » Thu Jun 06, 2013 7:44 am

No, it has multiple fields. But the last field's (FILLER) length is 3843 bytes to add spaces.

I just tried to split this field into multiple but still it is showing wrong record count (difference of 2).

chulett · Post by **chulett** » Thu Jun 06, 2013 7:49 am

How are you counting the records: wc -l? Do you have any binary data in the record? All it takes is something that looks like a linefeed to change your line count.

ray.wurlod · Post by **ray.wurlod** » Thu Jun 06, 2013 2:31 pm

Does your file have a byte order mark? (There is a property in the Sequential File stage that instructs it to ignore the BOM.)

Jayakannan · Post by **Jayakannan** » Fri Jun 07, 2013 4:43 am

When I run the unix command wc -l then the result is showing correct record count but in viewing the file using WinSCP editor or Notepad++ the record/line count is wrong.

Jayakannan · Post by **Jayakannan** » Fri Jun 07, 2013 6:35 am

Checked the data and there are couple of blank lines in between, not sure how they are populated.

Ray,

I am not familiar with BOM and there is no such explicit setting i could see.

chulett · Post by **chulett** » Fri Jun 07, 2013 6:56 am

So are you having an actual problem with the file on the server or just after you move it over to your PC to look at it?

Jayakannan · Post by **Jayakannan** » Fri Jun 07, 2013 7:21 am

On the server itself because i viewed the data using WinSCP on server.

chulett · Post by **chulett** » Fri Jun 07, 2013 7:26 am

The "Win" part of that name should tell you it's not happening "on the server itself".

What I'm asking is are other processes consuming the file and having issues? I just want make sure we're solving a real problem here and not a perceived one. I'm also curious about the 'couple of blank lines in between' that you mentioned, are they the issue and have you determined their origin yet?

Jayakannan · Post by **Jayakannan** » Fri Jun 07, 2013 7:43 am

This is resolved.

I changed the property of Record Delimiter String = DOS format to Record Delimiter = UNIX newline and the record count is proper now (without blank lines).

This file is going to be consumed by Mainframe system which has not happened yet.

Thanks for your help!!