Hi,
I'm using Lookup stage in my job,there are 10 lookup tables and one primary table.Please suggest any tips for better performance on higher volumes.
Thanks,
Sreedhar S.
Lookup Performance
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 4
- Joined: Tue Jan 15, 2013 7:21 am
- Location: Bangalore
Lookup Performance
Cheers,
Forrest.
Forrest.
The number of lookups you perform isn't as important as the type of data source behind the lookups (is it a dataset? A database table?) and, more importantly, how big the data volumes are in the lookups themselves. Without knowing the size of the reference data (in MB, average record length times number of records) no suggestions or recommendation can be made.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Re: Lookup Performance
What is the issue that you are facing ... What is the volume of incoming records vs lookup table size? How long is the execution time?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The answer will depend on many factors, including what other load there is on the server(s).
For example you could use 10 Lookup stages and disable operator combination, so that the individual lookups are being performed in separate processes. If not much else is happening on the machine, this would tend to be quite quick, particularly if you have a large volume of data and relatively small reference data sets.
For example you could use 10 Lookup stages and disable operator combination, so that the individual lookups are being performed in separate processes. If not much else is happening on the machine, this would tend to be quite quick, particularly if you have a large volume of data and relatively small reference data sets.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
If your lookups have small number of rows then it works fine. Your inbound data stream can be huge without issue. If your lookup table size grows then it becomes necessary to use other stage types no matter how large the inbound stream. You can use sparse lookups with small inbound stream and large lookup tables. Should have indexes on the keys in a sparse lookup.
Mamu Kim