Sequential File error

[yaminids]

Hello there,

I am getting the following errors while writing data from one Sequential File to another Sequential file:

ds_seqput: error in 'write()' - Error 0
ds_seqput: error in 'write()' - File too large

I checked the disk space on the server and only 35% has been used. Can anyone explain to me the reason for getting the above error eventhough there is enough disk space?

Also while transforming data from one Sequential file to another, is the data stored in any kind of temporary files before been written into the target Sequential file?

Thanks a lot in advance
Yamini

[ray.wurlod]

File too large usually means that the file has reached the maximum size permitted by one of your ulimit setting or by the operating system. For example, if the O/S limit is 4GB and the file is already 4GB, this error will occur for any attempt to append to the file.

[yaminids]

Ray,

I checked the permitted file size and it was set to "Unlimited". Is there a possiblity that a different limit (for file size) is set at the Operating System level and it is overriding the value at the user level?

Thanks a lot in advance
Yamini

[ArndW]

How large is your sequential file when the job aborts? Could it be 2Gb and you are hitting a system-level or file-system level limit? Which OS and filesystem are you using?

[Sainath.Srinivasan]

Are you running on AIX?

How did you check your ulimit? Maybe that DS was initiated with lower ulimit value than the one in your profile.

[ray.wurlod]

Test Sainath's theory by invoking ExecSH as a before-job subroutine to run the ulimit -a command and observe its output in the job log.

[yaminids]

Hello friends,

I checked ulimit -a as before job sub-routine and found that the file size is not 'Unlimited'.

Sai, how can I change the file size set at DS level to 'Unlimited'?

Thanks a lot in advance
Yamini

[Sainath.Srinivasan]

Are you working with AIX?

What is the file size? Is it zero bytes? Maybe the file shrinked after the error

So can you run the job again and monitor the file size in background.

[yaminids]

Sai,

Yes, I am working with AIX. I am trying to write data into a Sequential file from SQL server. The job is aborted after processing 3 million rows with the error 'file too large'

When the job was aborted the file size was close to 2GB. Can you advise me on changing the file size limit at DS level to 'Unlimited'?

Thanks a lot in advance
Yamini

[Sainath.Srinivasan]

From what you said earlier, it does seem that your ds environment ulimit is set to unlimited.

But as policy of AIX, you may have limitations creating files more than 2Gb. You can attempt to create a file for more than 2 Gb outside DataStage. This will give you an idea of whether the limitation is with AIX or DataStage.

[yaminids]

Sai,

I am still confused with file size limit. When I checked on AIX server I got the following result:

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) 32768
coredump(blocks) unlimited
nofiles(descriptors) 2000

But when I checked through DataStage (by executing before job sub-routine), I got the following result:

*** Output from command was: ***
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000


I have couple of questions:

1) Is there a limit on File size on AIX?
2) Is there a limit on File size in DataStage?

Thanks a lot for you help
Yamini

[ArndW]

The file size limits depend upon your OS (AIX in your case), the version and perhaps the file system type. You will need to read the UNIX documentation for your version to see what the file size limits are, but if you are using a recent DS version I will assume that your system can handle files >2Gb. DataStage itself uses whatever the implementation's file size limits are.

Have you looked into your UVCONFIG file to see if you set the limits there?

[ray.wurlod]

The ulimit reported in the log suggests about 2GB as the file size limit. Have your UNIX administrator check the ulimit for this user, and change it to unlimited.

[Sainath.Srinivasan]

As Ray has pointed out, your ulimit in DataStage is set to 2Gb.

You need to
1.) Change your Unix ulimit
2.) Bounce DataStage

If you run your job after this, it must work fine.

[yaminids]

Hello there,

Thank you all for your suggestions. We figured out that the problem is with 32 bit file system(AIX) which is imposing 2GB file limit. We are upgrading to DS 7.5 to overcome this

Yamini