How to increase memory usage on HPUX

dhwankim · Post by **dhwankim** » Wed May 11, 2005 3:08 am

I want to use px sort stage on HPUX.but I found something wrong with using memory.
It means that HP Box hase 60GB free memory. so I wanted to increase memory usage to around 1GB. (Parallel Degree = 12)
But When I set Restrict Memory Usage Parameter to 1000 (MB). But this job is aborted with some message. but Message was there is no space.

But this box has a lot of disk storage. and free memory.

so I thought that to use large memory on HPUX. I should set some special parameter for HPUX and Parallel Extender Engine.

For your information.
I aready checked kernal parameters( It statified ascential recommaned parameter(written in Ascenial Menual).

Please Let me know it, how to solve it.

Thanks in advance

Dhwankim

ArndW · Post by **ArndW** » Wed May 11, 2005 3:14 am

Dhwankim,

could you post the exact error message - I think that the message you are getting is referring to temporary disk space (see your $APT_CONFIG values) and not virtual memory - although your system might have a lot of disk space left free, it might not be on the partition that is being used by Px.

dhwankim · Post by **dhwankim** » Wed May 11, 2005 3:21 am

ArndW

I checked Scratch space for each node. But each pointed disk area has a lot of free space.
Each Scratched Area has 300GB left more. Is not It enough?
If so, Please give how to calculate the amount of needed disk space.

and Other thing is, When I decreased Restirict memory limit to around 60 (MG), than Job was well.

Please give your hands more.

Thanks in Advance.

ArndW wrote:Dhwankim,

could you post the exact error message - I think that the message you are getting is referring to temporary disk space (see your $APT_CONFIG values) and not virtual memory - although your system might have a lot of disk space left free, it might not be on the partition that is being used by Px.

ArndW · Post by **ArndW** » Wed May 11, 2005 3:30 am

dhwankim,

what I think is happening is that the stage is trying to access 1Gb of virtual memory for the process, but the OS will not allow the process so much space. What is your ulimit -a displayed value for memory? You either need to specify an amount in the Restrict memory limit that is smaller and/or you need to increase your process' virtual memory limit.

T42 · Post by **T42** » Wed May 11, 2005 12:02 pm

You also need to remember, not only ONE process is used for sorting, especially with multiple nodes. It is not a cumulative value, but an individual value you are setting there.

mmanes · Post by **mmanes** » Thu May 12, 2005 4:57 am

Hi dhwankim,
you can increase memory usage on HP-UX using memory windows.
Here the issue:

- In the file /etc/hosts: at the line
<IP> <hostname> add <host2> <host3> <host4> ...

- In /etc/services.window add the lines:
<hostname> 50
<host2> 51
<host3> 52
<host4> 53
...

- In .rhosts of your user add:
<hostname> <your user>
<host2> <your user>
<host3> <your user>
<host4> <your user>

- In the Config file:
the first node may have "fastname <hostname>"
and the others may have the aliases.
Example:
{
node "node1"
{
fastname "hostname"
pools ""
resource disk "/dstageeetl3/Dataset" {pools ""}
resource scratchdisk "/dstageeetl3/Scratch" {pools ""}
}

node "node2"
{
fastname "host2"
pools ""
resource disk "/dstageeetl3/Dataset" {pools ""}
resource scratchdisk "/dstageeetl3/Scratch" {pools ""}
}

node "node3"
{
fastname "host3"
pools ""
resource disk "/dstageeetl3/Dataset" {pools ""}
resource scratchdisk "/dstageeetl3/Scratch" {pools ""}
}
...

In this way you can use 1,7 GB per node.

By
Matteo.

dhwankim · Post by **dhwankim** » Thu May 12, 2005 5:07 am

Thank you . It's so helpful.
[

T42 · Post by **T42** » Thu May 12, 2005 4:23 pm

Oh, so this was a MPP setup?

dhwankim · Post by **dhwankim** » Thu May 12, 2005 6:15 pm

Hi
I tested this functionality on SMP.

but I used memorywindow concept for HPUX 11.x

This concept is announced at releasing for HPUX 11.x .

How to overcome limit of memory usage for 32 bit application.
(I just read an article.)

Anyway, It could be applied for HPUX box.

so I had a enterprise job used some gb memory.

T42 wrote:Oh, so this was a MPP setup?

sanjay · Post by **sanjay** » Thu Jun 11, 2009 11:23 pm

Hi mmanes

We are implementing memory window option in HP unix server

Just changing max_mem_window will work or do i need to logically divide Hostname and assign each memory window

Thanks
Sanjay

ray.wurlod · Post by **ray.wurlod** » Fri Jun 12, 2009 12:10 am

Even so, your tsort operators will still be constrained by any limit set explicitly in the Sort stage or by APT_TSORT_STRESS_BLOCKSIZE environment variable. You will probably need to increase one of these also.

hsahay · Post by **hsahay** » Wed Jul 22, 2009 4:07 pm

ray.wurlod wrote:Even so, your tsort operators will still be constrained by any limit set explicitly in the Sort stage or by APT_TSORT_STRESS_BLOCKSIZE environment variable. You will probably need to increase one of these also.

Hi Ray,

We want to use APT_TSORT_STRESS_BLOCKSIZE to increase the memory usage and hence performance.

Can you please tell us what is the default value for APT_TSORT_STRESS_BLOCKSIZE variable and upto what value we can increase the size of this variable? We could not find the details about this variable in product manual

Thanks for the help in advance

hsahay · Post by **hsahay** » Wed Jul 22, 2009 4:08 pm

ray.wurlod wrote:Even so, your tsort operators will still be constrained by any limit set explicitly in the Sort stage or by APT_TSORT_STRESS_BLOCKSIZE environment variable. You will probably need to increase one of these also.

Hi Ray,

We want to use APT_TSORT_STRESS_BLOCKSIZE to increase the memory usage and hence performance.

Can you please tell us what is the default value for APT_TSORT_STRESS_BLOCKSIZE variable and upto what value we can increase the size of this variable? We could not find the details about this variable in product manual

Thanks for the help in advance

ray.wurlod · Post by **ray.wurlod** » Wed Jul 22, 2009 5:04 pm

This question is unrelated to the thread that you've hijacked. Please begin a new thread. Or just consult the manuals.

hsahay · Post by **hsahay** » Thu Jul 23, 2009 2:37 pm

ray.wurlod wrote:This question is unrelated to the thread that you've hijacked. Please begin a new thread. Or just consult the manuals.

Sorry.

I have started a new thread for this topic