Missing record delimiter "\n", saw EOF instead
Moderators: chulett, rschirm, roy
Missing record delimiter "\n", saw EOF instead
Our DataStage parallel job has a Sequential File stage that reads a CSV file.
This is a sample of the file content. Each row ends with LF (Line Feed - hex = 0A), except the last row has no LF.
"Name","Currency","Description"LF
"ABC","USD","ABC Description"LF
"DEF","USD","DEF Description"
This is in Output > Properties:
Options:
First line is column names = True
This is in Output > Format:
Record level:
Final delimiter = end
Field defaults:
Delimiter = comma
Quote = double
When I run the DataStage, it imported all records successfully, but it gives me 2 warnings:
Warning: Missing record delimiter "\n", saw EOF instead
Warning: Import warning at record 81.
Import complete; 82 records imported successfully, 0 rejected.
There are 83 rows in the CSV file, with the 1st row as the column names.
The "Import warning at record ..." is always on the number of rows - 2, for example if the number of rows = 8, the Import warning will be at record 6.
It looks like the problem is because all rows ends with Line Feed, except for the last row (it doesn't end with Line Feed).
I have tried all the other Final delimiter (Record Level), but doesn't work.
Can I fix the warning ?
How can I read a CSV file who has all rows end with Line Feed, except for the last row ?
Thank you.
This is a sample of the file content. Each row ends with LF (Line Feed - hex = 0A), except the last row has no LF.
"Name","Currency","Description"LF
"ABC","USD","ABC Description"LF
"DEF","USD","DEF Description"
This is in Output > Properties:
Options:
First line is column names = True
This is in Output > Format:
Record level:
Final delimiter = end
Field defaults:
Delimiter = comma
Quote = double
When I run the DataStage, it imported all records successfully, but it gives me 2 warnings:
Warning: Missing record delimiter "\n", saw EOF instead
Warning: Import warning at record 81.
Import complete; 82 records imported successfully, 0 rejected.
There are 83 rows in the CSV file, with the 1st row as the column names.
The "Import warning at record ..." is always on the number of rows - 2, for example if the number of rows = 8, the Import warning will be at record 6.
It looks like the problem is because all rows ends with Line Feed, except for the last row (it doesn't end with Line Feed).
I have tried all the other Final delimiter (Record Level), but doesn't work.
Can I fix the warning ?
How can I read a CSV file who has all rows end with Line Feed, except for the last row ?
Thank you.
Re: Missing record delimiter "\n", saw EOF instead
I fixed the warning by adding a Line Feed at the end of the file using Before Job subroutine EXECSH sed -i -e '$a\' /dev/inbound/MyFile_*.csv
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
A bit complex a solution, but it is a solution.
Perhaps easier would have been via the ExecSH before-job subroutine.
Perhaps easier would have been
Code: Select all
echo \n >> filename
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Not in the parallel Sequential File stage.
The server Sequential File stage has it handled and, as noted elsewhere here, could be encapsulated in a server Shared Container to be included in a parallel job.
The server Sequential File stage has it handled and, as noted elsewhere here, could be encapsulated in a server Shared Container to be included in a parallel job.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Re: Missing record delimiter "\n", saw EOF instead
I had similar kind of issue and one of the below functions. Please refer
ereplace(ereplace(echoProcDate.$CommandOutput,@FM,"")
CONVERT(Char(13),"",lnk.field)
ereplace(ereplace(echoProcDate.$CommandOutput,@FM,"")
CONVERT(Char(13),"",lnk.field)
Thanks
Sumith
Sumith
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Such an approach is only appropriate if one can read the data in the first place. In the current case, the lack of a linefeed character prevented the last line of the file from being read.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.