"mmap failed: Not enough space"
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 145
- Joined: Fri Sep 15, 2006 12:23 pm
"mmap failed: Not enough space"
Hi,
I am getting the error when I am trying to perform a join or a lookup in a particular job. but thats not the case with all the jobs. The source data is read from a sequential file which passes the data to transformer and then the data is passed to row generator which inturn passes the data to the join/Lookup but both fail with the error message ""mmap failed: Not enough space"" Could any one please let me know what is the reason for this?
Thanks
I am getting the error when I am trying to perform a join or a lookup in a particular job. but thats not the case with all the jobs. The source data is read from a sequential file which passes the data to transformer and then the data is passed to row generator which inturn passes the data to the join/Lookup but both fail with the error message ""mmap failed: Not enough space"" Could any one please let me know what is the reason for this?
Thanks
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
There's probably not enough space available at the time on whatever directory you have specified for scratch disk. Or, in the case of a Lookup stage, not enough space to load the reference data into memory.
Please post the entire (detailed) error message so we can get a better idea.
Please post the entire (detailed) error message so we can get a better idea.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 145
- Joined: Fri Sep 15, 2006 12:23 pm
Well, I was able to solve this issue. Just for the reference for the people who would be facing this issue in future. It was not the issue with the scratch disk or the swap space. But in the sort I had used "restrict memory usage" which was Causing this error. Anyways, Dsxchange is a place where in I can post when i find the end of the road.
Thanks people
Thanks people
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
"mmap failed: Not enough space"
Hello ahmedwaseem2000:ahmedwaseem2000 wrote:Well, I was able to solve this issue. Just for the reference for the people who would be facing this issue in future. It was not the issue with the scratch disk or the swap space. But in the sort I had used "restrict memory usage" which was Causing this error. Anyways, Dsxchange is a place where in I can post when i find the end of the road.
Thanks people
I am getting the same error "mmap failed: Not enough space". I am using a Join stage.
Can you please let me know how you remove "restrict memory usage" in this join.
Thanks in advance.
Thanks in advance,
DS_MJ wrote:Hello ahmedwaseem2000:
I am doing a Hash partitioning in this join and not using a sort stage.
When I remove this join I dont get this error ("mmap failed: Not enough space")
Thanks again.
Set the environment variable "APT_TSORT_STRESS_BLOCKSIZE" higher value and give a try..
Accept that some days you're the pigeon and some days you're the statue.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Ray,Thanks for looking..ray.wurlod wrote:Note that APT_TSORT_STRESS_BLOCKSIZE, if set, overrides any Restrict Memory Usage setting in the Sort stage. ...
"DS_MJ" said he dont have SORT stage in his design but still getting the above said Error.i told him to set the variable and verify whether that works or not.
Thanks again.
Accept that some days you're the pigeon and some days you're the statue.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Thank you all, so much, appreciate your responses.dsedi wrote:Ray,Thanks for looking..ray.wurlod wrote:Note that APT_TSORT_STRESS_BLOCKSIZE, if set, overrides any Restrict Memory Usage setting in the Sort stage. ...
"DS_MJ" said he dont have SORT stage in his design but still getting the above said Error.i told him to set the variable and verify whether that works or not.
Thanks again.
However, I had a aggregator stage before the join stage. I removed the sort in the aggregator stage and now I dont get this error.
Thanks and Best Regards,
Mamta
Thanks in advance,
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
Disabling operator combination also fixed the issue
I too was facing this mmap issue with a job where a join stage has 32 links each with an inline sort, The most curious thing is that this job ran fine in our Development and Testing environment but failed in Production. when the operator combination was disabled it ran fine in Production
Very curious to know why ?
Very curious to know why ?
Re: Disabling operator combination also fixed the issue
32 links into a Join is unusual. In your case, Orchestrate was likely combining most/all of the sort operators (added by the inline sort) with the join operator into a single process ("Combined Operator"), and that process ran out of addressable memory. When you disabled operator combination, each sort was run as a separate process with it's own memory space.zulfi123786 wrote:I too was facing this mmap issue with a job where a join stage has 32 links each with an inline sort, The most curious thing is that this job ran fine in our Development and Testing environment but failed in Production. when the operator combination was disabled it ran fine in Production
Very curious to know why ?
"Inline" sort and Sort stage both utilize the tsort operator. Inline sort just provides a more convenient way to add a sort to a link, but provides fewer options to the developer.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
Re: Disabling operator combination also fixed the issue
Does it mean all inline sorts together form one tsort oerator ?jwiles wrote: Orchestrate was likely combining most/all of the sort operators (added by the inline sort) with the join operator into a single process ("Combined Operator"), and that process ran out of addressable memory. When you disabled operator combination, each sort was run as a separate process with it's own memory space.
Regards,
and what is the difference in terms of memory usage if operators are combined into one as the amount of data is still the same
Re: Disabling operator combination also fixed the issue
No, it just means that one process (per partitiion) is performing all of the work. Each sort is still a separate "entity".zulfi123786 wrote:Does it mean all inline sorts together form one tsort oerator ?
Actually, for multiple input streams the sorts may not be combined as I am thinking. I'll need to verify the combination rules to be certain, but you can see for yourself by enabling the APT_DUMP_SCORE environment variable and examining the job score for combined operators.
It's hard to quantify beyond some generalizations as it partially depends upon how the operating system allocates memory for individual processes. Also, each operator may have it's own memory requirements beyond normal buffering/link (each tsort, for example, allocates memory for sorting workspace). This gets into a deeply technical area which is really beyond the scope of this forum and really requires a good understanding of how the parallel engine operates.and what is the difference in terms of memory usage if operators are combined into one as the amount of data is still the same
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.