why this fatal won't abort a job???
Moderators: chulett, rschirm, roy
why this fatal won't abort a job???
Dear all,
I meet this fatal in a parallel job, but the job is in finished status like normal... and have not aborted...
aggregate_1,0: Caught exception from runLocally(): APT_BadAlloc: Heap allocation failed..
Why this fatal won't abort the job??
many thanks...
I meet this fatal in a parallel job, but the job is in finished status like normal... and have not aborted...
aggregate_1,0: Caught exception from runLocally(): APT_BadAlloc: Heap allocation failed..
Why this fatal won't abort the job??
many thanks...
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You cannot do that. It is an internal message from the PX engine. It really shouldn't be an error message, but just a warning message.
Why do you want this to abort your job? As Ray has explained, it means that there is no more virtual memory space to hold the data in your aggregation (might you be able to raise your ulimit values?) so PX has begun using disk space to hold temporary data.
You can use PX message handling to demote this message to a warning, or you could pre-sort your data coming into the aggregator stage so that no temporary space is required.
Why do you want this to abort your job? As Ray has explained, it means that there is no more virtual memory space to hold the data in your aggregation (might you be able to raise your ulimit values?) so PX has begun using disk space to hold temporary data.
You can use PX message handling to demote this message to a warning, or you could pre-sort your data coming into the aggregator stage so that no temporary space is required.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Ray and I have been saying that this error message seems to be a non-fatal one - meaning that your data is going to be the same regardless of whether this message is in the log or not.
Your incoming data needs to be sorted upon the key columns for your aggregation. It is not sorted that way, because if it were, the aggregation stage would have no need for interim storage and you wouldn't be getting this error message. If you check your sorting attributes in your job you can make this whole issue a moot point.
Your incoming data needs to be sorted upon the key columns for your aggregation. It is not sorted that way, because if it were, the aggregation stage would have no need for interim storage and you wouldn't be getting this error message. If you check your sorting attributes in your job you can make this whole issue a moot point.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You are not losing any data when this message appears. All that is happening is that DataStage is being forced to use disk because there is not enough virtual memory available.
I agree with Arnd that it should not be a Fatal message.
Do check your ulimit settings for the user ID that processes DataStage jobs; you may be able to increase the amount of memory that that user can allocated. Your UNIX admininstrator will need to be involved, because only superuser can increase a ulimit.
I agree with Arnd that it should not be a Fatal message.
Do check your ulimit settings for the user ID that processes DataStage jobs; you may be able to increase the amount of memory that that user can allocated. Your UNIX admininstrator will need to be involved, because only superuser can increase a ulimit.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 407
- Joined: Mon Jun 27, 2005 8:54 am
- Location: Walker, Michigan
- Contact:
You can set ulimit for a shell and inherited shells will have the new ulimit. That is, you could set it in your profile, logout and login, then bounce datastage.
If osh is running into memory allocation problems (lookup stage will fail when you use over 512 MB on reference link), you can also change osh to use large memory allocation.
On AIX, you would enter the following to change from 512 MB to 2GB.
Code: Select all
cat ~/.profile
## DATASTAGE PROFILE
unset ENV
## DATA
ulimit -d 1048576
## MEMORY
ulimit -m 1048576
## NOFILES
ulimit -n 10000
## STACK
ulimit -s 262144
## CORE DUMP SIZE
ulimit -c 4194304
. ~/.dsadm
if [ -s "$MAIL" ] # This is at Shell startup. In normal
then echo "$MAILMSG" # operation, the Shell checks
fi # periodically.
## DISPLAY SOME HELP INFO ON LOGIN
. ~/.menu
On AIX, you would enter the following to change from 512 MB to 2GB.
Code: Select all
/usr/ccs/bin/ldedit -bmaxdata:0x80000000/dsa $APT_ORCHHOME/bin/osh
/usr/ccs/bin/ldedit: File /Ascential/DataStage/PXEngine/bin/osh updated.