Datastage is eating all my memory

nnes · Post by **nnes** » Wed Nov 02, 2005 2:37 pm

Hi all,

I have a very simple job that reads a sequential file 40Million rows at 1500kbyte/row= 60GByte. Goes through a modify stage to eliminate a few rows and transform date into timestamp and then loads it into SQL server using ODBC enterprise.

As I run the job it increasingly takes more and more memory( 8GB swap), until it runs out of memory and is killed by the OS (AIX).

Is there a way of setting a limit for the amounth of memory Datastage will use?

thanks all

kduke · Post by **kduke** » Wed Nov 02, 2005 6:14 pm

You probably have a memory leak. You need to figure out where. It may be in the ODBC driver.

kduke · Post by **kduke** » Wed Nov 02, 2005 6:15 pm

You can work around it by spliting the feed into multiple files or feeds.

nnes · Post by **nnes** » Thu Nov 03, 2005 7:42 am

kduke wrote:You probably have a memory leak. You need to figure out where. It may be in the ODBC driver.

That makes sense. I guess the ODBC stage has the memory leak since it is the one we use the least. They say those memory leaks are pretty common in C++.

nnes · Post by **nnes** » Thu Nov 03, 2005 7:48 am

kduke wrote:You can work around it by spliting the feed into multiple files or feeds.

I thought about that. It is a real pain to have to split the file in 5 parts and run 5 separate jobs for that though. Idealy you want to split depending on the size of the file because it is likely that the size of the input file will grow over time. That means you would have to create some kind of script that deals with that.

Ultramundane · Post by **Ultramundane** » Thu Nov 03, 2005 7:43 pm

Don't know if this related or not...

I just got a patch for memory leaks for datetime, numeric/decimal datatypes. We encountered the memory leaks when working with the Sybase, Teradata, and ODBC stages in PX. This patch was for DS751A on AIX. Ecase 76017.

nnes · Post by **nnes** » Fri Nov 04, 2005 7:50 am

Ultramundane wrote:Don't know if this related or not...

I just got a patch for memory leaks for datetime, numeric/decimal datatypes. We encountered the memory leaks when working with the Sybase, Teradata, and ODBC stages in PX. This patch was for DS751A on AIX. Ecase 76017.

AIX+ODBC+Datetime in our case but the leaky code is apparently used in other stages too. We talked to a tech and he promised to send us a patch. I am hoping to get it soon.