Hi,
Rather than posting in a four year old thread of same subject which is marked "Resolved" felt it appropriate to start a new one.
Curious to know if DataStage (8.5) has the feature of freeing the memory of pointers returned by parallel routine on a per row basis (or some interval) rather than doing it at then end of transformer process.
I am running a load test over a newly written routine and the job aborts after processing around 55 million records.
Also I made an attempt to free the pointer of the routines input argument before the routines 'return' statement and it didn't give me any issue, I was expecting the input pointer to be marked 'const' but it isn't also wondering if free'ing an input argument would cause any issue.
Thanks
Zulfi
Freeing returned pointer in parallel routine
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
Freeing returned pointer in parallel routine
Last edited by zulfi123786 on Mon Apr 14, 2014 12:14 am, edited 1 time in total.
- Zulfi
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
Hi Andy,
The routine performs Base64 Encoding of input argument and then performs AES 256 bit encryption using opessl, in the same job I am decrypting the cipher after performing base64 decoding.
Every pointer I have used in my code except the returned one is free'd and the job aborts after 55 million rows with heap allocation failure which leaves only one possibility that the returned pointer might not have been freed.
Looking forward to your valuble inputs.
Thanks
Zulfi
The routine performs Base64 Encoding of input argument and then performs AES 256 bit encryption using opessl, in the same job I am decrypting the cipher after performing base64 decoding.
Every pointer I have used in my code except the returned one is free'd and the job aborts after 55 million rows with heap allocation failure which leaves only one possibility that the returned pointer might not have been freed.
Looking forward to your valuble inputs.
Thanks
Zulfi
- Zulfi
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
The above is intriguing as how that is working, I can clearly see that transformer is not freeing up the returned pointer in my case.asorrell wrote:I have jobs that have processed hundreds of millions of rows and never had to do that.
Code: Select all
APT_CombinedOperatorController,0: Current heap size: 1,208,487,600 bytes in 25,129,226 blocks
Transformer_3,0: Input 0 consumed 25120499 records.
47 bytes * 25120499 =1180663453
Close to where my job dies.
Thinking of declaring a global variable for the returned pointer but I would have to run it from a sequential transformer which I am not comfortable with
![Sad :(](./images/smilies/icon_sad.gif)
Any help is highly appreciated.
- Zulfi
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
Below is a simple routine I wrote to check if my actual routine had any other memory leak or was it really due to transformer not free'ing up the returned pointer.
A simple job with
row generator ---> transformer --->sequential file
was designed and a call to this routine was made from the transformer
The routine returns the reverse of input string and the size of input was made same as that was given to my original routine.
This test job also aborted after almost the same number of rows as my earlier job.
Tried both new[] and malloc.
Also Is there a way to look at the actual C++ code the transformer generates. (not the .trx code which looks to be only a pseudo code)
A simple job with
row generator ---> transformer --->sequential file
was designed and a call to this routine was made from the transformer
The routine returns the reverse of input string and the size of input was made same as that was given to my original routine.
This test job also aborted after almost the same number of rows as my earlier job.
Code: Select all
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
char *ReverseString(char *In){
int i=0,len=0;
char *res;
len=strlen(In);
//res=(char *)malloc((sizeof(char)*len) + 1);
res=new char[((sizeof(char)*len) + 1)] ;
for (i=0; i<len; i++){
res[i]=In[len-i-1];
}
res[i]=0;
return res;
}
Also Is there a way to look at the actual C++ code the transformer generates. (not the .trx code which looks to be only a pseudo code)
- Zulfi