Not Enough space
Moderators: chulett, rschirm, roy
Not Enough space
Hi,
I am facing a unique problem when i am using a 4 node configuration problem for Datastage 7.5.2 on HP-UX Itanium 64 bit box. We are running our jobs in production successfully till yesterday and all of a sudden we are facing below error:
Could not map table file "/scratch/proj/DataSets/lookuptable.20090714.mf0ymnc": Not enough space
It looks simple and though of space problem on the partition but i was wrong as i ran the job and started monitoring the server. 250GB of free space is available for this partition.
When i use single node the job is running fine but is very slower than a 4 node configuration file.
Any suggestion on where the problem could be?
I am facing a unique problem when i am using a 4 node configuration problem for Datastage 7.5.2 on HP-UX Itanium 64 bit box. We are running our jobs in production successfully till yesterday and all of a sudden we are facing below error:
Could not map table file "/scratch/proj/DataSets/lookuptable.20090714.mf0ymnc": Not enough space
It looks simple and though of space problem on the partition but i was wrong as i ran the job and started monitoring the server. 250GB of free space is available for this partition.
When i use single node the job is running fine but is very slower than a 4 node configuration file.
Any suggestion on where the problem could be?
-
- Premium Member
- Posts: 79
- Joined: Thu Mar 22, 2007 4:58 pm
- Location: USA
While it could be a disk space issue, it is more likely to be running out of memory.
A recommendation to try would be to monitor the swap memory space while the process is running using both the single node and the 4-node configurations.
A recommendation to try would be to monitor the swap memory space while the process is running using both the single node and the 4-node configurations.
Jack Thornton
----------------
Spectacular achievement is always preceded by spectacular preparation - Robert H. Schuller
----------------
Spectacular achievement is always preceded by spectacular preparation - Robert H. Schuller
Do you have a lookup stage and if so, how big is your lookup data (approximately)?
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
If you are on AIX then this thread will help solve the problem. I thought that the solution was in another thread, with some more details, but couldn't find it.
I think that if the lookup exceeds 512Mb you get this error unless extended the space. Note that if you have large VarChar columns in the lookup you might save memory space by removing the explicit length declarations.
I think that if the lookup exceeds 512Mb you get this error unless extended the space. Note that if you have large VarChar columns in the lookup you might save memory space by removing the explicit length declarations.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
I ran my process and below is the glance report
GlancePlus C.04.50.00 07:38:27 usplcpd1 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SU | 7% 7% 19%
Disk Util F F |100% 50% 100%
Mem Util S SU UB B | 31% 31% 31%
Swap Util U UR R | 10% 10% 10%
------------------------------------------------------------------------------------------------------------------------------------------------------
IO BY DISK Users= 4
GlancePlus C.04.50.00 07:38:27 usplcpd1 ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SU | 7% 7% 19%
Disk Util F F |100% 50% 100%
Mem Util S SU UB B | 31% 31% 31%
Swap Util U UR R | 10% 10% 10%
------------------------------------------------------------------------------------------------------------------------------------------------------
IO BY DISK Users= 4
Sree
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
sanjay wrote:you have implement memory window option in HP unix to resolve above issue below issue .
Please explain how you believe the memory window option solves lack of physical disk space.
I'm not really certain what you mean by "above issue below issue" either.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
The "Not enough space" message almost always refers to disk space. I have fought with this issue before. After intensive checking during runtime it became clear that we had no disk issues, but that the lookup stage we were using tried to allocate more than 512 Mb. The solution is to use "ldedit" to change the osh executable so that it can allocate more virtual memory using the MAXDATA subcommand.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Ray , Arndw
Ray its not lack of physical disk space 250GB of free space is available
above issue means
Could not map table file "/scratch/proj/DataSets/lookuptable.20090714.mf0ymnc": Not enough space
In HP unix it can allocate only 1 GB of shared memory for 32 bit application . so if it goes above 1 GB job aborts . so there is feature available in HP unix i.e memory window which will allocate 1 GB for each node suppose u have 4 nodes
in datastage 7.5.2 osh is a 32 bit application file . not sure about datastage 8 ldedit is available only in AIX . equivalent change in HP unix in maxdsize but changing maxdsize doesn't reslove issue
Thanks
Sanjay
Ray its not lack of physical disk space 250GB of free space is available
above issue means
Could not map table file "/scratch/proj/DataSets/lookuptable.20090714.mf0ymnc": Not enough space
In HP unix it can allocate only 1 GB of shared memory for 32 bit application . so if it goes above 1 GB job aborts . so there is feature available in HP unix i.e memory window which will allocate 1 GB for each node suppose u have 4 nodes
in datastage 7.5.2 osh is a 32 bit application file . not sure about datastage 8 ldedit is available only in AIX . equivalent change in HP unix in maxdsize but changing maxdsize doesn't reslove issue
Thanks
Sanjay
ArndW wrote:The "Not enough space" message almost always refers to disk space. I have fought with this issue before. After intensive checking during runtime it became clear that we had no disk issues, but that the lookup stage we were using tried to allocate more than 512 Mb. The solution is to use "ldedit" to change the osh executable so that it can allocate more virtual memory using the MAXDATA subcommand.