Page 1 of 1
Is there any max restriction for file size?
Posted: Fri Jul 22, 2005 11:30 am
by kumar_s
Hi,
i have a file with 3 million records. I know this is not very huge file for ETL tool to handle. Though i use very simple sequential stage to another seq stage conversion, i faced error, job got aborted.
so i thought this may be file based restriction in operating system. so i tried with FileSet (as we all know which subdivides the file in max of 2GB each).
Now job runs and also could see the data passing but still it aborts.
I could the follwoing errors
Code: Select all
APT_CombinedOperatorController,1: Unsupported read in APT_FileBufferOutput::spillToNextFile(): Bad file number.
APT_CombinedOperatorController,0: Unsupported read in APT_FileBufferOutput::spillToNextFile(): Bad file number.
CIFWORK_CPC,1: Export failed.
CIFWORK_CPC,1: Output file full, and no more output files
CIFWORK_CPC,0: Export failed.
CIFWORK_CPC,0: Output file full, and no more output files
60% starts
CIFWORK_CPC,1: The runLocally() of the operator failed.
But when the whole job finshed with 100 percent, the following error pops up
Code: Select all
APT_CombinedOperatorController,1: The runLocally() of the operator failed.
APT_CombinedOperatorController,1: Operator terminated abnormally: runLocally did not return APT_StatusOk
CIFWORK_CPC,0: The runLocally() of the operator failed.
and at last i also get the following error.
Code: Select all
CIFWORK_CPC,0: Output 0 produced 2 records.
CIFWORK_CPC,0: Export complete; 0 records exported successfully, 0 rejected.
APT_CombinedOperatorController,0: The runLocally() of the operator failed.
APT_CombinedOperatorController,0: Operator terminated abnormally: runLocally did not return APT_StatusOk
main_program: Step execution finished with status = FAILED.
If any body understand this issue pls give some inputs............
Thanks in advance
-kumar
Posted: Sun Jul 24, 2005 3:16 am
by roy
Hi,
What Unix are you using?
Usually there are file systems that can be defined to hold large files (over the 2GB limit).
So if you can do that I think your better off.
IHTH,
Posted: Sun Jul 24, 2005 3:40 am
by kumar_s
roy wrote:Hi,
What Unix are you using?
Usually there are file systems that can be defined to hold large files (over the 2GB limit).
So if you can do that I think your better off.
IHTH,
Is there a way we can define the limitation of file size in unix.
-------
kumar
Posted: Sun Jul 24, 2005 4:20 am
by ArndW
Kumar,
enter "ulimit -a" to see what your limits are, n.b. this may be different for your background DS processing than for your login UNIX session. Often the users are limited to 2Gb files even though the file system and the operating system support them.
Posted: Sun Jul 24, 2005 5:51 am
by elavenil
Kumar,
Even you use Fileset, Dataset, Seq file stages, the files are created based on the ulimit settings for that user. Login to unix using the user name, which used to run the job and execute the command as suggessted by Arnd and that would tell you the size limit for the particular user id that was used to run the process.
Regards
Saravanan
Posted: Sun Jul 24, 2005 8:39 am
by kumar_s
ArndW wrote:Kumar,
enter "ulimit -a" to see what your limits are, n.b. this may be different for your background DS processing than for your login UNIX session. Often the users are limited to 2Gb files even though the file system and the operating system support them.
thanx Arnd,
i could see the values. it is restricted to a user as well as the os level right.
is there any way we can increaset the max file block. (with the help of admin)
Posted: Sun Jul 24, 2005 9:43 am
by elavenil
Yes. the file size can be increased to unlimited if the admin agrees.
Regards
Saravanan
Posted: Wed Nov 02, 2005 1:38 pm
by bpsprecher
Since this forum has been so helpful over the years, I will overcome my shyness and lend some knowledge.
We are running version 7.5.1 on AIX 5.2. We were encountering the "Output file full" error when a file reached 1 GB, even though the ulimit on our ID's can produce unlimited file sizes.
Taking advice from track_star in post "Parallel Extender Memory Issue" on 10/12/05:
As for the ulimit, you can put this snippet of code into a blank job on the Job control page:
Execute "SH -c 'ulimit -a'" Capturing UOUT
Call DSLogInfo ("ulimit settings: ":UOUT, "JobControl")
Compile and run that job, and it will print the ulimit settings of the user running your DS jobs (because those are the settings you'll need to modify, if necessary).
This showed us that our DataStage was imposing a file size limitation of 2097152 (blocks of 512 bytes), which equates to 1,073,751,824 bytes -- the exact size at which our target files were bombing.
IBM told us to insert this line into our ds.rc file:
The ds.rc file is the DS engine startup/shutdown script, and it can be found in path $DSHOME/sample. We placed this code at the beginning of the script (actually, right after our "umask 002" code). Then, the DS Engine must be recycled. Executing track_star's code now shows that we are able to produce unlimited file sizes.
To find out where your script resides, execute "$DSHOME/bin/uv -admin -info" and refer to the "Startup script" line.
Good luck!
Posted: Wed Nov 02, 2005 1:42 pm
by bpsprecher
Sorry, it was 1,073,741,824 bytes (2,097,152 * 512), not 1,073,751,824.
Posted: Wed Nov 02, 2005 1:48 pm
by kcbland
dt12946 wrote:IBM told us to insert this line into our ds.rc file:
It's a shame, because by default you are supposed to make sure you're ulimit is set to unlimited across the board. Your initial installation 'way back should have included making these adjustments at the same time you verified your kernel parameters. Them pesky kernel adjustments...
