Memory usage by lookup stage
Posted: Mon Apr 02, 2018 9:36 am
So, I informed a developer that the lookup stage (non-sparse) should typically be used when referencing data in smaller tables, which allows the structure to fit into memory.
He asked two very good questions for which I didn't have an answer.
1) Does it use standard virtual memory to buffer the lookup records, or does it have a "reserved" buffer area.
2) If it has a reserved area, what is the size and is there anyway to set or increase the size of that area.
The reason he's probably asking is that we're running in a BigIntegrate (DataStage on Yarn) environment with really large (5 GB) container sizes, so if memory is there and available, why not use it for a larger lookup buffer?
I think its using general virtual memory, but wanted to confirm...
He asked two very good questions for which I didn't have an answer.
1) Does it use standard virtual memory to buffer the lookup records, or does it have a "reserved" buffer area.
2) If it has a reserved area, what is the size and is there anyway to set or increase the size of that area.
The reason he's probably asking is that we're running in a BigIntegrate (DataStage on Yarn) environment with really large (5 GB) container sizes, so if memory is there and available, why not use it for a larger lookup buffer?
I think its using general virtual memory, but wanted to confirm...