NOFILES and MAXUPROC Kernel Parameters in DS 7.5

Oli · Post by **Oli** » Wed Sep 29, 2004 7:06 am

In the latest DataStage Installation Guide there are 2 new kernel parameter configuration recommended, the MAXUPROC and NOFILES. For Solaris the following values are reccommended

MAXUPROC - Minimum Value of 100 per CPU
NOFILES - Minimum Value 1000

The upgrade/install guide does not state the purpose of these kernel parameters for DataStage. I have searched through SUN documentation and found that the MAXUPROC is the limit of no of allowed process per user but I cant seem to find out the NOFILES parameter.

Have you some information on these kernel parameters that might come in handy? What happens if they are not specified, will the upper value be unlimited or...?

roy · Post by **roy** » Fri Oct 08, 2004 12:13 pm

Hi,
theese are OS kernel parameters that you need to consult with sys admins to reconfigure.
they will also mention that the NOFILES is the limit of simultaniously open files (I can't recall right now if its a total limit or per process)

in future cases like this consult your sys admins for OS stuff to get a quick answer (if your not yet familiar with theese persons get to be their friend asap

)

IHTH,

kduke · Post by **kduke** » Fri Oct 08, 2004 12:42 pm

NOFILES has a direct relationship to MFILES in uvconfig. I think MFILES can be 2 less than NOFILES. Anyway DataStage performs best when you make MFILES as large as possible otherwise it will close a file before it opens another when it gets close the MFILES limit.

ketfos · Post by **ketfos** » Fri Oct 08, 2004 2:53 pm

Hi Kim,
What do MFILES and NOFILES parameters mean? What do they signify?
Can you please throw some light for users who do not know?
Thanks

Ketfos

kduke · Post by **kduke** » Fri Oct 08, 2004 10:27 pm

MFILES is the number of files which Universe can have open within one process. This has to live under the limit at the UNIX level. Some UNIX platforms do not have a limit. NOFILES is the UNIX kernel parameter which sets the limit at the UNIX level. If MFILES allows a process to exceed the UNIX level then it will core dump. So you have to set these together. Most UNIXs have this limit. I think AIX may be the only one which does not. I am doing all this from memory. I would have to look it up to be sure. The UNIX command sysdef will usually tell what the UNIX kernel parameters are set all.

The uvconfig file sets these for Universe. You have to do a uvregen to make these parameters permanent. At TCL in the UV account in Universe or the DataStage engine type in CONFIG ALL to see these parameter settings.

Do a search. I think Ray and Ken both have talked about these parameters and the rotating file pool.

All this is covered in both the DataStage and Universe manuals. I explained above how files get closed when a Universe process gets close to the MFILES limit. You can count how many files are opened or need to be opened in a job by counting the links. If a file gets opened and closed a lot then your job runs slower. Your rows per second go down especially in processing with hash files.

s_r_chandru · Post by **s_r_chandru** » Sun Oct 10, 2004 9:56 pm

Hi Kduke,
Thanks for your info.One more thing to add.If I increase the NOFILES in Sunsolaris 2.8 will there be any impact in OS(like kernel parameter to be tunned or changed).
Following is my Config
$ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) unlimited
nofiles(descriptors) 256
memory(kbytes) unlimited
Thanks
-Chandru

kduke · Post by **kduke** » Sun Oct 10, 2004 10:14 pm

What do you mean? Sun Solaris 2.8 is the OS. It is also UNIX. NOFILES is also a UNIX kernel parameter. You need to change MFILES to 200 or more which should be plenty. NOFILES is large enough. Do you have jobs with more than 100 links? There is another parameter for the number of DYNAMIC hash files. If you edit uvconfig then you will see it in the comments. I would up this as well. Do a search Ray has covered changing these several times. He has a step by step post to walk you through changing these.

s_r_chandru · Post by **s_r_chandru** » Mon Oct 11, 2004 12:04 am

Hi Kduke,
Thanks for your reply.
I mean if I change the NOFILES in Unix.Will I need to change any other kernel parameter in Unix level as becoz ,we run other application ,so I doent want to Impact it.
Thanks
-Chandru

kduke · Post by **kduke** » Mon Oct 11, 2004 6:39 am

No. There is a direct releationship between that one parameter NOFILES and MFILES. It looks like NOFILES is large enough. Just need to increase MFILES.