Hi,
my job starts with a flat file and enriches data by 20-30 lookups.
For performance reasons I have selected the data into hashfiles so the lookup source is alsways a hashfile.
Because I have so many lookups and hashfiles the configured memory is not big enough to hold all data in RAM.
The option pre-load to memory is check in every hash file but the log of cause shows that it can not be cashed for all of the files.
Any ideas how I could increase the throughput for that job?
Currenty it is processing 30-40 rows /s.
I tried to split the job into smaller ones with 10-15 lookups with no success.
I tried to use multiple processes for that job - with even worse performance.
Which memory configuration sould I increase for the best performance increase?
Any help and ideas are appreciated.
kind regards
Michael
Performance for multiple Lookups
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Try using four or five Transformer stages, with four or five lookups each, and enable inter-process row buffering. That way you will have four or five separate processes each with its own memory allocation.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
My guess would be that your hashed files are ineffeciently sized or too big. You need to also monitor your CPUs and see what the job is doing. If other things are running you may be contributing your poor performance to the wrong reason.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle