When I tried to run a large file > 200MB through XML Pluggin Stage i get the error
:
APT_CombinedOperatorController,0: Fatal Error: Throwing exception: APT_BadAlloc: Heap allocation failed.
I confirmed all the hard and soft ulimits were set to unlimited but still the 'data' resource said it was constrained. I my brain for days on this and IBM couldn't find a solution , but I found out the problem and solution:
In the dsenv file there is a couple lines near the bottom:
LDR_CNTRL=MAXDATA=0x60000000@USERREGS
export LDR_CNTRL
You need to comment it out otherwise the number of memory segments allocated is 6 and not unlimited. I can't say what are the downsides, but this is what I needed to do to get my jobs working in the test environment.
True, there's still an upper limit of around 400 or 500 MBs as noted, and it seems to vary by platform. You really should be getting many small XML files rather than a small number of really big ones.
-craig
"You can never have too many knives" -- Logan Nine Fingers