Lookup Space problem in Windows
Moderators: chulett, rschirm, roy
Lookup Space problem in Windows
How can I increase the look up space in windows environment. I would really appreciate if you can provide me the command.
I am getting the following error:
LKP_CSD_Tracking_Number,0: Could not map table file "K:/Ascential/DataStage/Datasets/lookuptable.20061229.4zlwfxa (size 553973128 bytes)": Not enough space
Error finalizing / saving table /tmp/dynLUT5560af42d249
Operator's runLocally() failed.
Any help is greatly appreciated.
Thanks,
Sri
I am getting the following error:
LKP_CSD_Tracking_Number,0: Could not map table file "K:/Ascential/DataStage/Datasets/lookuptable.20061229.4zlwfxa (size 553973128 bytes)": Not enough space
Error finalizing / saving table /tmp/dynLUT5560af42d249
Operator's runLocally() failed.
Any help is greatly appreciated.
Thanks,
Sri
-
- Participant
- Posts: 56
- Joined: Mon Oct 16, 2006 7:32 am
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Ray,
I added one more disk to the resouce pools:
{
node "node1"
{
fastname "serever"
pools ""
resource disk "K:/Ascential/DataStage/Datasets" { pools "" }
resource disk "M:/Ascential/DataStage/Datasets" { pools "" }
resource scratchdisk "K:/Ascential/DataStage/Scratch" { pools "" }
resource scratchdisk "M:/Ascential/DataStage/Scratch" { pools "" }
}
node "node2"
{
fastname "server"
pools ""
resource disk "K:/Ascential/DataStage/Datasets" { pools "" }
resource disk "M:/Ascential/DataStage/Datasets" { pools "" }
resource scratchdisk "K:/Ascential/DataStage/Scratch" { pools "" }
resource scratchdisk "M:/Ascential/DataStage/Scratch" { pools "" }
}
}
but I am getting the same error again...any ideas of how to implement LMM in Windows....
Thanks,
Sri
I added one more disk to the resouce pools:
{
node "node1"
{
fastname "serever"
pools ""
resource disk "K:/Ascential/DataStage/Datasets" { pools "" }
resource disk "M:/Ascential/DataStage/Datasets" { pools "" }
resource scratchdisk "K:/Ascential/DataStage/Scratch" { pools "" }
resource scratchdisk "M:/Ascential/DataStage/Scratch" { pools "" }
}
node "node2"
{
fastname "server"
pools ""
resource disk "K:/Ascential/DataStage/Datasets" { pools "" }
resource disk "M:/Ascential/DataStage/Datasets" { pools "" }
resource scratchdisk "K:/Ascential/DataStage/Scratch" { pools "" }
resource scratchdisk "M:/Ascential/DataStage/Scratch" { pools "" }
}
}
but I am getting the same error again...any ideas of how to implement LMM in Windows....
Thanks,
Sri
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The reference Data Set for a Lookup stage must fit into available physical memory. It may be this amount of space that you are exhausting.
If this is the case, consider whether you are carrying unnecessary data in the reference Data Set and, if not, whether you can solve your requirement using a Join stage or a sparse lookup. (If you are carrying unnecessary columns, the solution is obvious.)
If this is the case, consider whether you are carrying unnecessary data in the reference Data Set and, if not, whether you can solve your requirement using a Join stage or a sparse lookup. (If you are carrying unnecessary columns, the solution is obvious.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54
- Joined: Wed Oct 25, 2006 11:07 pm
- Location: Hyderabad
Re: Lookup Space problem in Windows
This happens wen ur lookup try to get many records from the refernce link to the main memory. That is when the memory gives up. If ur refernce link is having say more than a million records then I would suggest go for "Join" rather than lookup
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Where did you pull "say more than a million records" from?
The error message in the original post refers to (approx) 500MB. I have a vague recollection that there's an environment variable that controls this - would be surprised if there weren't - but don't have access to research it at the moment.
The error message in the original post refers to (approx) 500MB. I have a vague recollection that there's an environment variable that controls this - would be surprised if there weren't - but don't have access to research it at the moment.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The reference Data Stage for a Lookup stage MUST be able to fit in physical memory. If you can not reduce the size of the reference Data Set so that it can fit in physical memory, then you can not use a Lookup stage.
Are there any columns you are not using in the job? Don't put them in the reference Data Set.
Are there any rows that you will never look up? Don't put them in the reference Data Set.
Are there relatively few distinct lookup values in your stream input compared to the number of key values in the reference Data Set, and is the reference table in DB2 or Oracle? If so, you might contemplate a sparse lookup.
Are there any columns you are not using in the job? Don't put them in the reference Data Set.
Are there any rows that you will never look up? Don't put them in the reference Data Set.
Are there relatively few distinct lookup values in your stream input compared to the number of key values in the reference Data Set, and is the reference table in DB2 or Oracle? If so, you might contemplate a sparse lookup.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
That RAM is doing a lot of other things at the same time!
I have yet to find a reference to 500MB as the limit (per partition?) for the reference Data Set for a Lookup stage, and how to increase that, but I'm fairly sure I've seen it somewhere.
I have yet to find a reference to 500MB as the limit (per partition?) for the reference Data Set for a Lookup stage, and how to increase that, but I'm fairly sure I've seen it somewhere.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Have you check K:/Ascential/DataStage/Datasets/ and /tmp for space. Try placing some 500MB-1GB file in these directries and check if it can hold without any error or waring. Per Ray's comment, try running the job alone in the server when RAM is idle for this job to get utilize.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'