Lookup Space problem in Windows

[sri1111]

How can I increase the look up space in windows environment. I would really appreciate if you can provide me the command.

I am getting the following error:

LKP_CSD_Tracking_Number,0: Could not map table file "K:/Ascential/DataStage/Datasets/lookuptable.20061229.4zlwfxa (size 553973128 bytes)": Not enough space
Error finalizing / saving table /tmp/dynLUT5560af42d249
Operator's runLocally() failed.

Any help is greatly appreciated.

Thanks,
Sri

[DSguru2B]

Do an exact search on 'Not enough space'.

[sri1111]

Thanks DSGuru.

I am not able to find the OSH command which I can use for windows to accomodate large memory models.

I found this for unix:

/usr/ccs/bin/ldedit -bmaxdata:0x80000000/dsa FullPathOf /osh

so just wondering what might be the command ...

[johnthomas]

it seems you are running out of diskspace , since available space is less than "size 553973128 bytes"

[ray.wurlod]

Add more directories (on different drives) to the resource disk in your configuration file.

[sri1111]

Hello John,

I think the problem is not with the disk space as I am having more than 200GB on that drive.

Ray,

Let me follow your suggestion and add more directories.

Thanks guys

[sri1111]

Ray,

I added one more disk to the resouce pools:

{
node "node1"
{
fastname "serever"
pools ""
resource disk "K:/Ascential/DataStage/Datasets" { pools "" }
resource disk "M:/Ascential/DataStage/Datasets" { pools "" }
resource scratchdisk "K:/Ascential/DataStage/Scratch" { pools "" }
resource scratchdisk "M:/Ascential/DataStage/Scratch" { pools "" }
}
node "node2"
{
fastname "server"
pools ""
resource disk "K:/Ascential/DataStage/Datasets" { pools "" }
resource disk "M:/Ascential/DataStage/Datasets" { pools "" }
resource scratchdisk "K:/Ascential/DataStage/Scratch" { pools "" }
resource scratchdisk "M:/Ascential/DataStage/Scratch" { pools "" }
}
}


but I am getting the same error again...any ideas of how to implement LMM in Windows....

Thanks,
Sri

[ray.wurlod]

The reference Data Set for a Lookup stage must fit into available physical memory. It may be this amount of space that you are exhausting.

If this is the case, consider whether you are carrying unnecessary data in the reference Data Set and, if not, whether you can solve your requirement using a Join stage or a sparse lookup. (If you are carrying unnecessary columns, the solution is obvious.)

[sudeepmantri]

This happens wen ur lookup try to get many records from the refernce link to the main memory. That is when the memory gives up. If ur refernce link is having say more than a million records then I would suggest go for "Join" rather than lookup

[ray.wurlod]

Where did you pull "say more than a million records" from?

The error message in the original post refers to (approx) 500MB. I have a vague recollection that there's an environment variable that controls this - would be surprised if there weren't - but don't have access to research it at the moment.

[sri1111]

Thanks RAY and Sudeep for your suggestions.

I would like to go for LOOKUP rather than JOIN and if I cannot figure out the solution for this problem then I will redesign my job. I opened a ticket in IBM and awaiting their response.

Thanks,
Sri1111

[ray.wurlod]

The reference Data Stage for a Lookup stage MUST be able to fit in physical memory. If you can not reduce the size of the reference Data Set so that it can fit in physical memory, then you can not use a Lookup stage.

Are there any columns you are not using in the job? Don't put them in the reference Data Set.

Are there any rows that you will never look up? Don't put them in the reference Data Set.

Are there relatively few distinct lookup values in your stream input compared to the number of key values in the reference Data Set, and is the reference table in DB2 or Oracle? If so, you might contemplate a sparse lookup.

[sri1111]

Ray,

I have 16GB of RAM on my box. I am using four columns in my lookup stage which are very much needed so I cannot remove any of the columns. IBM suggested me to use JOIN stage instead of lookup. So I am redesigning my job again.

Thanks for your support.

Regards,
Sri

[ray.wurlod]

That RAM is doing a lot of other things at the same time!

I have yet to find a reference to 500MB as the limit (per partition?) for the reference Data Set for a Lookup stage, and how to increase that, but I'm fairly sure I've seen it somewhere.

[kumar_s]

Have you check K:/Ascential/DataStage/Datasets/ and /tmp for space. Try placing some 500MB-1GB file in these directries and check if it can hold without any error or waring. Per Ray's comment, try running the job alone in the server when RAM is idle for this job to get utilize.