Ulimit problem

DWH-M · Post by **DWH-M** » Sat Jan 05, 2008 4:17 am

Hi,

I went thru couple of posts on the same problem as this http://dsxchange.com/viewtopic.php?t=93 ... a4bea76809.

Problem: Not able to create a sequential file more than 1.5 gb and aborts with the message "File too large".

The "ulimit -a;id" from DS before job routine results as

Time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000
uid=208(dsadm) gid=208(dstage) groups=1(staff)

At AIX level for dsadm,

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 245760
stack(kbytes) 16384
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) 2000

Even after bouncing the DS server after the ulimit was changed at the OS level, the DS log message shows the old value.

Environment : DS 7.5.2 on AIX 5.3

I could create a file > 2gb at os level.

Please help.

Thanks!

ArndW · Post by **ArndW** » Sat Jan 05, 2008 5:58 am

I think that your ulimit settings for root are limited, since that user spawns the other processes this is where you limitations are inherited from at runtime.

DWH-M · Post by **DWH-M** » Mon Jan 07, 2008 5:50 am

Hi Arnd,

Since i confirmed that the user dsadm has unlimited privilege for fileblocks at OS level, how does the root's privilege interfere with DS execution for creating large files using the user dsadm? Could you please elaborate.

Thanks

ArndW · Post by **ArndW** » Mon Jan 07, 2008 6:00 am

please re-read my post. I think it is the "root" user causing the issue. If a process has ulimits and spawns another, then the new one cannot have a higher ulimit (even if the settings are unlimited) than the parent.

DWH-M · Post by **DWH-M** » Mon Jan 07, 2008 7:00 am

Hi Arndw,

If the root user ulimit settings are overriding the dsadm user's setting, then how are files > 2gb could be created using dsadm at the back end?

thanks

DWH-M · Post by **DWH-M** » Mon Jan 07, 2008 7:17 am

Hi ArndW,

Just to add, even after changing the root setting of fileblocks to unlimited, the problem persists.

Thanks

chulett · Post by **chulett** » Mon Jan 07, 2008 7:25 am

He is saying the override only happens during job execution, not all the time. What are the ulimit settings for root on that box?

Ok, I see you've changed them. Did you stop and restart DataStage after making the change?

DWH-M · Post by **DWH-M** » Mon Jan 07, 2008 10:34 pm

Hi,

I have changed the fileblocks for root user to unlimited and restarted the DS server. I still find the ulimit for dsadm in DS log shows 2 gb though it is unlimited at OS level.

Please help.

Thanks!

ArndW · Post by **ArndW** » Tue Jan 08, 2008 3:33 am

Strange. Did you restart DataStage as root or as your dsadm user? Also, check your $DSHOME/dsenv file to see if it contains a ulimit command.

DWH-M · Post by **DWH-M** » Tue Jan 08, 2008 4:00 am

Hi ArndW,

I know its really weird, but i dont know how to resolve this.

I restarted the DS server as dsadm. There are no entries for ulimit in dsenv file though there are entries in uvconfig file which you know will not impact if the root and dsadm has unlimited settings for fileblocks.

Thanks

ArndW · Post by **ArndW** » Tue Jan 08, 2008 4:15 am

If you have root access try to run your job and see what the ulimit settings display.

Also, if you execute ". $DSHOME/dsenv" as dsadm do the ulimit settings remain the same before & after the command?

Known Facts
========
1. root user has unlimited ulimit
2. dsadm user has unlimited ulimit
3. $DSHOME/dsenv contains no ulimit command
4. System and filesystem limits allow creation of files >2Gb by dsadm/root from OS
5. Display of "ulimit -a" in before job shows limited settings for dsadm user

DWH-M · Post by **DWH-M** » Tue Jan 08, 2008 4:29 am

Hi,

I do not have the root access, but i confirmed with my admin that he has granted unlimited to all properties of ulimit.

I executed ulimit -a for dsadm at os level before and after executing dsenv. the results are the same. I confirm the known facts which you have mentioned.

thanks!

DWH-M · Post by **DWH-M** » Tue Jan 08, 2008 4:30 am

Hi,

I do not have the root access, but i confirmed with my admin that he has granted unlimited to all properties of ulimit.

I executed ulimit -a for dsadm at os level before and after executing dsenv. the results are the same. I confirm the known facts which you have mentioned.

thanks!

ArndW · Post by **ArndW** » Tue Jan 08, 2008 4:47 am

Would you be able to get someone with root login to try logging into DataStage at your workstation and test the runtime limit? Also, could you try the following from UNIX

Code: Select all

cd $DSHOME
. ./dsenv
bin/dssh          {This will put you into the DS shell}
SH -c "ulimit -a"
QUIT               {This will bring you back out to your shell}

Is the ulimit still unlimited inside the DS command environment?

chulett · Post by **chulett** » Tue Jan 08, 2008 6:52 am

At some point soon here, I would think you'd be better served by opening a case with your official Support provider rather than continuing to thrash here.