Sequential File Size Limit
Posted: Thu Apr 07, 2005 10:22 am
I'm trying to import a 1GB sequential file using the Sequential File Stage in EE 7.5.
The job aborts at the last record saying the data is bad. This is the error msg:
I broke the file into two using sed and the ETL job works fine. I thought this would be related to the ulimit settings for my userid which are:
Am I missing something? Any recommendations on how to fix this?
I read somewhere that for big files the FileSet stage is recommended but I could not find anywhere (including in the documentation) how to use it to read that 1GB file.
Any help would be really appreciated.
Rgds
Marcelo Silva, PMP
JPMorganChase
The job aborts at the last record saying the data is bad. This is the error msg:
Code: Select all
StatementDetailFile,0: Short read encountered on import. This most likely indicates one of the following possibilities:
1) The import schema you specified is incorrect
2) Invalid data (the schema is correct, but there is an error in the data)
Expected 504 record bytes, got 56
StatementDetailFile,0: Import error at record 2130439.
Code: Select all
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2048
nofiles(descriptors) 2000
I read somewhere that for big files the FileSet stage is recommended but I could not find anywhere (including in the documentation) how to use it to read that 1GB file.
Any help would be really appreciated.
Rgds
Marcelo Silva, PMP
JPMorganChase