It would seem that the overhead actually lies with writing to the table. You might want to see if there is a way of performing a bulk load to the table.
Additionally, check out increasing the array size and transaction size of the DB2 stage. These determine how much data is transferred in each read and how much is written to the table. Try increasing in multiples of 5000
Also you might want to stick an interprocess stage between the two too.
Performance
Moderators: chulett, rschirm, roy
I have something similar.
I put a InterProcess Stage between the source and the LP and an InterProcess between the LC and the target. You should get more performance.
NOTE!: However,
1) I've noticed the the InterProcess is a little unstable. Keep your rows below 2million and you should be fine.
2) Check you hardware, do you have more than one processor and is it using them all otherwise you don't gain much than a time-splitting processing, mulithreaded job. More processors, means more processing.
I put a InterProcess Stage between the source and the LP and an InterProcess between the LC and the target. You should get more performance.
NOTE!: However,
1) I've noticed the the InterProcess is a little unstable. Keep your rows below 2million and you should be fine.
2) Check you hardware, do you have more than one processor and is it using them all otherwise you don't gain much than a time-splitting processing, mulithreaded job. More processors, means more processing.
~The simpliest solutions are always the best~
~Trick is to understand the complexity to implement simplicity~
~Trick is to understand the complexity to implement simplicity~
-
- Participant
- Posts: 612
- Joined: Thu May 03, 2007 4:59 am
- Location: Melbourne
Performance of link partitioner with multiple transformers depends on the number of CPUs of server. Use the inter-process (IPC) stage which provides a communication channel between processes running.
Joshy George
<a href="http://www.linkedin.com/in/joshygeorge1" ><img src="http://www.linkedin.com/img/webpromo/bt ... _80x15.gif" width="80" height="15" border="0"></a>
<a href="http://www.linkedin.com/in/joshygeorge1" ><img src="http://www.linkedin.com/img/webpromo/bt ... _80x15.gif" width="80" height="15" border="0"></a>
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
IPC, Link Partitioner and Link Collector have had some bad press here. If you're worried about that, use a Transformer stage to implement round robin partitioning (Mod(@INROWNUM,3) for 3 outputs). Enable inter process row buffering if you have two active stages, otherwise enable in process row buffering.
But I still suspect the major delays are in the database. Are there many indexes and constraints on the table? Have you tried bulk load?
But I still suspect the major delays are in the database. Are there many indexes and constraints on the table? Have you tried bulk load?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 273
- Joined: Wed Oct 18, 2006 12:20 pm
- Location: Porto
Even I agree with Ray's post...
Its a real good approach... you must desgin your own logic to split the data into different sets and process them running multiple instance of your job...
I am implementing a similar logic for partitioning the input data for my process...where I have to load apporx 30Million data in daily basis....Mod(@INROWNUM,3)
Its a real good approach... you must desgin your own logic to split the data into different sets and process them running multiple instance of your job...
Share to Learn, and Learn to Share.