I'm not in a position to test this directly and didn't have any luck with a search so I'll ask the question here:
Is there a 2Gb limit to the reference data to to SCD stage? Our source and reference data will be in excess of 5Gb but the documentation indicates an in-memory table is built which makes me think along the lines of the lookup stage and it's inherent memory limit.
Does anyone know for certain that this limit does, or does not, exist?
Slowly Changing Dimension Stage - size limitations?
Moderators: chulett, rschirm, roy
Slowly Changing Dimension Stage - size limitations?
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Yes, but if the stage was written to support larger amounts of data it could, even with a 32bit pointer, get around this limitation (writing to disk, for example). If it uses the old lookup-stage functionality then I'd be "SOL".
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Would it make any difference if your reference data were partitioned three or more ways, so that any one process had less than 2GB of reference data with which to deal?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
If it uses the same code as the lookup stage, then the 2Gb limit wouldn't be affected by the number of processing nodes. I guess we'll have to code an example and see if it blows up on us. Once we have a result I'll post it here.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>