Page 1 of 1
Datastage is eating all my memory
Posted: Wed Nov 02, 2005 2:37 pm
by nnes
Hi all,
I have a very simple job that reads a sequential file 40Million rows at 1500kbyte/row= 60GByte. Goes through a modify stage to eliminate a few rows and transform date into timestamp and then loads it into SQL server using ODBC enterprise.
As I run the job it increasingly takes more and more memory( 8GB swap), until it runs out of memory and is killed by the OS (AIX).
Is there a way of setting a limit for the amounth of memory Datastage will use?
thanks all
Posted: Wed Nov 02, 2005 6:14 pm
by kduke
You probably have a memory leak. You need to figure out where. It may be in the ODBC driver.
Posted: Wed Nov 02, 2005 6:15 pm
by kduke
You can work around it by spliting the feed into multiple files or feeds.
Posted: Thu Nov 03, 2005 7:42 am
by nnes
kduke wrote:You probably have a memory leak. You need to figure out where. It may be in the ODBC driver.
That makes sense. I guess the ODBC stage has the memory leak since it is the one we use the least. They say those memory leaks are pretty common in C++.
Posted: Thu Nov 03, 2005 7:48 am
by nnes
kduke wrote:You can work around it by spliting the feed into multiple files or feeds.
I thought about that. It is a real pain to have to split the file in 5 parts and run 5 separate jobs for that though. Idealy you want to split depending on the size of the file because it is likely that the size of the input file will grow over time. That means you would have to create some kind of script that deals with that.
Posted: Thu Nov 03, 2005 7:43 pm
by Ultramundane
Don't know if this related or not...
I just got a patch for memory leaks for datetime, numeric/decimal datatypes. We encountered the memory leaks when working with the Sybase, Teradata, and ODBC stages in PX. This patch was for DS751A on AIX. Ecase 76017.
Posted: Fri Nov 04, 2005 7:50 am
by nnes
Ultramundane wrote:Don't know if this related or not...
I just got a patch for memory leaks for datetime, numeric/decimal datatypes. We encountered the memory leaks when working with the Sybase, Teradata, and ODBC stages in PX. This patch was for DS751A on AIX. Ecase 76017.
AIX+ODBC+Datetime in our case but the leaky code is apparently used in other stages too. We talked to a tech and he promised to send us a patch. I am hoping to get it soon.