Freeing returned pointer in parallel routine

[zulfi123786]

Hi,

Rather than posting in a four year old thread of same subject which is marked "Resolved" felt it appropriate to start a new one.

Curious to know if DataStage (8.5) has the feature of freeing the memory of pointers returned by parallel routine on a per row basis (or some interval) rather than doing it at then end of transformer process.

I am running a load test over a newly written routine and the job aborts after processing around 55 million records.

Also I made an attempt to free the pointer of the routines input argument before the routines 'return' statement and it didn't give me any issue, I was expecting the input pointer to be marked 'const' but it isn't also wondering if free'ing an input argument would cause any issue.

Thanks
Zulfi

[asorrell]

Out of curiosity - why are you trying to free the pointers? I have jobs that have processed hundreds of millions of rows and never had to do that. What problem are you trying to solve?

[zulfi123786]

Hi Andy,

The routine performs Base64 Encoding of input argument and then performs AES 256 bit encryption using opessl, in the same job I am decrypting the cipher after performing base64 decoding.

Every pointer I have used in my code except the returned one is free'd and the job aborts after 55 million rows with heap allocation failure which leaves only one possibility that the returned pointer might not have been freed.

Looking forward to your valuble inputs.

Thanks
Zulfi

[zulfi123786]

I have jobs that have processed hundreds of millions of rows and never had to do that.

The above is intriguing as how that is working, I can clearly see that transformer is not freeing up the returned pointer in my case.

Code: Select all

APT_CombinedOperatorController,0: Current heap size: 1,208,487,600 bytes in 25,129,226 blocks
Transformer_3,0: Input 0 consumed 25120499 records.

The above appears just before my job dies of memory hunger. In the routine a char* pointer is passed which is of 47 bytes and the job aborts after processing 25120499 rows.

47 bytes * 25120499 =1180663453

Close to where my job dies.

Thinking of declaring a global variable for the returned pointer but I would have to run it from a sequential transformer which I am not comfortable with

Any help is highly appreciated.

[zulfi123786]

Below is a simple routine I wrote to check if my actual routine had any other memory leak or was it really due to transformer not free'ing up the returned pointer.

A simple job with

row generator ---> transformer --->sequential file

was designed and a call to this routine was made from the transformer

The routine returns the reverse of input string and the size of input was made same as that was given to my original routine.

This test job also aborted after almost the same number of rows as my earlier job.

Code: Select all

#include <stdio.h>
#include <stdlib.h>
#include <string.h>


char *ReverseString(char *In){
int i=0,len=0;
char *res;

len=strlen(In);
//res=(char *)malloc((sizeof(char)*len) + 1);

res=new char[((sizeof(char)*len) + 1)] ;

for (i=0; i<len; i++){
res[i]=In[len-i-1];
}
res[i]=0;

return res;

}

Tried both new[] and malloc.

Also Is there a way to look at the actual C++ code the transformer generates. (not the .trx code which looks to be only a pseudo code)