Reading a text file but short of last record

[seell]

My DS job reads an input file. This input file is from an external source; so we could not control how the file is created.

The file must reside on Unix;when using the Unix command 'wc -l filename'; the number returned is always n-1 (where n is the number of records).

If we use telnet from dos; then vi the file; the number of records would be n; however if I use emulation software such as REFLECTION on Unix and perform the vi on that file, I get the message [Incomplete last line] n lines .....

Is there any method to allow the DS Job to read the last record without having to amend the input file manually? Reason being that user would like to automate the process from getting the input file followed by the ETL processes.

[WoMaWil]

simply add an empty line with an Unix-command in a before-job- or a before-stage-routine.

[roy]

Hi,
If I undestaned you, you have a columns missing on the last line?
in that case you can use the column definition to populate missing columns wiht defaults (at the rightmost side of the columns definition of the sequential file column definition) oh and don't forget to check the supress truncation of data in the stage 2.

If you ment you have an extra last line (empty probably) this would probably be the last new line.
in this case you could try a unix command/s to truncate that line or the new line from it.
to be more exact you will have to give an actual example so we could get it right.

IHTH

[seell]

My sample file is like this :[done via vi editor in Unix]

F1510849P20030305T01T02S7118151515151515
F1600980Q20030304T37T42T3021151516181616
S8617285B20030305R14S80S8523181818201818
S8630534H20030305R14T38S6227191919191919
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
"regdata_sample.txt" [Incomplete last line] 4 lines, 163 characters

I would like to be able to read in all 4 lines instead of 3 now.

Thank you
Regards

[chulett]

An "incomplete last line" means, from what I recall, that the file is not being created properly and the "EOF" marker is missing. I would think you could do something similar to what WoMaWil suggested and cat the missing piece onto the file in a before-job routine. I believe that would be a ^Z. I could check when I get into work or perhaps someone else could verify the marker?

Just out of curiosity - you never said what happens with DataStage when you process the file, do you get N records or N-1?

[scboyce]

Text files on unix by definition must have a trailing Linefeed (CHR10). All UNIX utilities will make sure that any non-zero byte file will have a trailing LineFeed. Vi for example will always put a LineFeed as the very last character in the file, even if it did not have one to begin with. Nice programs will warn you about "incomplete lines", others will simply ignore that last line quietly. Ouch.

This problem usually arises when a file is transferred to UNIX from some other platform that does not required a trailing LineFeed, like DOS/Windows. Dohhhhh.

Want proof?

Create a new file on UNIX with vi and put "Hello World" in it and save. Boom, 12 bytes. Now do a od -x <file> and you can see the trailing LineFeed.

Now create the same file using Notepad in Windows and save. FTP that file to UNIX in either ASC mode or BIN mode. Since there is only one line without line delimeters, it does not matter. Do od -x on that file and you can see that it is missing the trailing LineFeed.

Now edit the file with vi and just save without changing the text.
Boom, 12 bytes.

Lovely. Make sure when creating text files on other platforms that what ever program is doing it, that it is configured to "append trailing LineFeeds". Most programmer editors have this feature. Anybody write scripts in Windows and then transfer to UNIX?

Hope this helps.
-Steve

[ray.wurlod]

You can add the trailing line feed with echo in a before-job subroutine that uses ExecSH or ExecDOS (depending on your OS). Use the >> redirection operator to append.