Sort stage error: Error in merging files

[j.anderson@thomson.com]

I'm having an error in a sort stage where i'm getting the following error:

cl5928Fac00262..sortFac: Error in merging files
Unix error: /ascential/DataStage/Projects/advprod/sokZqlaa/err: Error 0

I've tried bumping up the number of rows in virtual memory and using the max number of sort files, but nothing seems to be helping. Any ideas? The file that is going through this stage has about 6 million rows.

Thanks!

[chulett]

Running out of temp space? Make sure you specify in the 'Temporary Director' property somewhere that has plenty o' room. Leaving that blank defaults to the Project, which seems like what you are doing. It is generally considered to be a Bad Thing to fill that space up.

[j.anderson@thomson.com]

Yeah, we're in very good shape as far as space. I'd venture to guess we have 60-70 GB free. Any other things I could check?

[tonystark622]

Check that the kernel parameters are set to at least the recommended values according to the Install and Upgrade Guide. We also had to increase the 'maxdsiz' kernel parameter (We're on HP/UX) to up the ulimit value. I don't recall if we were getting the exact same error, but we were definitely seeing "Abnormal termination of stage".

Good Luck,
Tony

[ray.wurlod]

Is /ascential/DataStage/Projects/advprod/sokZqlaa/err a file? If so, what's in it? (I'm thinking it might be an error log containing extra diagnostic information. I could be off target there.)

[j.anderson@thomson.com]

No, there isn't a /ascential/DataStage/Projects/advprod/sokZqlaa/err file. I did notice, however, that one of the files in /ascential/DataStage/Projects/advprod/sokZqlaa/ is irregularly sized. All of the opv__xx are one size except for one which is around 2GB. Is there a file size limit on these sort temp files?

[chulett]

What O/S? 2GB is a suspicious number, being the typical file size limit for sequential files on UNIX systems. What is your "ulimit"?

[j.anderson@thomson.com]

We are running AIX 5.2. My ulimit is set to "unlimited".

[chulett]

Out of curiousity, have you verified that it is unlimited in the environment your jobs run in? Not just from the same user at the command line as they can be different.

[j.anderson@thomson.com]

I'm pretty sure its unlimited to, but I'm not sure how to check.

[tonystark622]

Create a new server job.

Open the Job Properties dialog.
Go to the Job Control tab.
Paste in the following code:

Code: Select all

Execute "SH -c 'ulimit -a'" Capturing Uout

Call DSLogInfo("Ulimit ":Uout,"Job Control")

Compile and run the job. The ulimit will be in the job log.

HTH,
Tony

[chulett]

Thanks Tony.

[j.anderson@thomson.com]

CopyOfcl9998Prov..JobControl (Job Control): Ulimit time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000

I think this is what you wanted. Does this mean it's unlimited then?

[tonystark622]

I'm not a UNIX expert, but that's what I'd say.

Tony