Sequential file load problem

abc123 · Post by **abc123** » Thu Jun 07, 2007 1:05 pm

The error I get is:

##I TFCN 000001 11:47:47(000) <main_program>
Ascential DataStage(tm) Enterprise Edition 7.5.1A
Copyright (c) 2004, 1997-2004 Ascential Software Corporation.
All Rights Reserved

##I TUTL 000031 11:47:47(001) <main_program> The open files limit is 1024; raising to 65536.
##I TOSH 000002 11:47:47(002) <main_program> orchgeneral: loaded
##I TOSH 000002 11:47:47(003) <main_program> orchsort: loaded
##I TOSH 000002 11:47:47(004) <main_program> orchstats: loaded
##I TFSC 000001 11:47:47(005) <main_program> APT configuration file: /opt/Ascential/DataStage/Configurations/default.apt
##W TCOS 000049 11:47:47(006) <main_program> Parameter specified but not used in flow: Prm1
##W TCOS 000049 11:47:47(007) <main_program> Parameter specified but not used in flow: Prm2
##W TCOS 000049 11:47:47(008) <main_program> Parameter specified but not used in flow: Prm3
##W TCOS 000049 11:47:47(009) <main_program> Parameter specified but not used in flow: Prm4
##W TCOS 000049 11:47:47(010) <main_program> Parameter specified but not used in flow: Prm5
##W TCOS 000049 11:47:47(011) <main_program> Parameter specified but not used in flow: Prm6
##W TCOS 000049 11:47:47(012) <main_program> Parameter specified but not used in flow: Prm7
##W TCOS 000049 11:47:47(013) <main_program> Parameter specified but not used in flow: Prm8
##W TCOS 000049 11:47:47(014) <main_program> Parameter specified but not used in flow: _APT_DISABLE_COMBINATION
##W TCOS 000049 11:47:47(015) <main_program> Parameter specified but not used in flow: _DS_ENABLE_RESERVED_CHAR_CONVERT
##W TCOS 000049 11:47:47(016) <main_program> Parameter specified but not used in flow: DSProjectMapName
##W TOIX 000170 11:53:24(000) <IDC_Output,0> Import consumed only 99bytes of the record's 100 bytes (no further warnings will be generated from this partition)
##W TOIX 000154 11:53:24(001) <IDC_Output,0> Import warning at record 0.
##I TOIX 000156 11:47:48(002) <IDC_Output,0> Progress: 10 percent.
##I TOIX 000156 11:47:48(003) <IDC_Output,0> Progress: 20 percent.
##I TOIX 000156 11:47:48(004) <IDC_Output,0> Progress: 30 percent.
##I TOIX 000156 11:47:48(005) <IDC_Output,0> Progress: 40 percent.
##I TOIX 000156 11:47:48(005) <IDC_Output,0> Progress: 50 percent.
**VIEWDATA ROW LIMITER HIT**

I have a file that is coming into Linux from Windows environment. Each line is 100 characters long (using awk). So for a file of 5 lines, number of characters should be 500. However, when I do a vi on it, I see 505 characters, that is, every line has 1 additional character. When I do a vi and go to the last character, it is character 99 so the file has 99 visible characters. However, whenever I get the length of the line from Unix using awk, it has 100 characters.

I have tried the following:

1)I set the Field Width and Field Max Width to length of the column on the Edit Column Metadata window.
2)Added a one character dummy column at the end and set "Drop on Input"
3)Went through every post in this forum. None of the problems are similar to mine.
4)Here is the interesting part. When I take the first 102 lines of the file, it loads fine. When I take first 103, it gives the error. When I do a tail on the last 5 lines of this 103 line file, it loads fine again! Exact same situation
on the tail side.

Any help would be appreciated.

kris007 · Post by **kris007** » Thu Jun 07, 2007 1:25 pm

What is the line terminator set to in the sequential file stage?

abc123 · Post by **abc123** » Thu Jun 07, 2007 1:42 pm

Record Level:
Final Delimiter: None
Record Delimiter: Unix Newline

Field Level:
Delimiter: None

abc123 · Post by **abc123** » Thu Jun 07, 2007 1:50 pm

One note. 2 of the 3 source file stages of the job are datasets and 1 sequential file. They are all going into a join.

kris007 · Post by **kris007** » Thu Jun 07, 2007 2:02 pm

Even if you are joining the sequential file with other datasets, your problems arises in your source sequential file. As you have mentioned that the file is coming from Windows environment, I believe the error has something to do with how you are placing the file in UNIX. Are you transferring in the binary mode? Or can you try giving the line terminator as DOS line terminator and check if you are able to read the data?

abc123 · Post by **abc123** » Thu Jun 07, 2007 2:17 pm

I tried by giving DOS Line terminator as well. Still the same error.

kris007 · Post by **kris007** » Thu Jun 07, 2007 6:52 pm

Okay. If that doesn't seem to work we need to take a different approach. Can you provide the details as to what you have set everything on format tab of the sequential file? Also, if the file is a fixed width file, did you mention the datatype as CHAR for all the fields?

abc123 · Post by **abc123** » Tue Jun 19, 2007 4:18 pm

I still haven't been able to resolve this issue. Does anybody have any idea? All my columns are char.

ray.wurlod · Post by **ray.wurlod** » Tue Jun 19, 2007 7:21 pm

Try setting the Record Delimiter String property to \r\n and removing the Record Delimiter property.