Hi,
We have a basic question on the way a Database stage works. Suppose we have a SQL on a source database stage that returns 2 million rows, how does DataStage handles it?
Does DataStage executes the SQL and holds the 2 million rows in its memory and passes a chunk by chunk of rows to the next stage or does it process it some way different? If the former happens, then it requires a lot of memory to hold the 2 million records in its memory, right...
Thanks.
A basic question on Database stage
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Typically the database server will establish the result set and the DataStage job will establish a cursor into the result set, fetching an array of rows at a time. There are variations on this theme, but in no case (in server jobs) does DataStage cache the entire data set in memory.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: