hi ,
i have a problem with the performance.
My source is a csv file and it loads into a oracle database.
we have a 100000 records only daily basis.
It is taking a quite long time to load these files.
the number of columns we have are 15.
please le tme know if any one can help me.
thanks in advance
Improving the performance of transformer stage
Moderators: chulett, rschirm, roy
It depends up on the Job design, Transformaions used (if at all), length of the columns (which is not a very big factor unless other wise its LOB).
You need to identify the bottleneck. If the source is in different nodes which involves network bandwidth you ll need to work for that. If its on Database, you can use Bulk load as suggested. If its on transformations, perhaps you can post the list of options you use, and get suggestions from the form.
You can try Multiple readers per node and other similar options available to read parallely for IO contention.
You need to identify the bottleneck. If the source is in different nodes which involves network bandwidth you ll need to work for that. If its on Database, you can use Bulk load as suggested. If its on transformations, perhaps you can post the list of options you use, and get suggestions from the form.
You can try Multiple readers per node and other similar options available to read parallely for IO contention.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Are you sure the bottleneck is in the Transformer stage, or is this simply a convenient scapegoat? What throughput do you get if you replace the Oracle stage with a Peek stage? What does that reveal to you?
What settings are you using in your Oracle stage? What are the settings of the two environment variables that control Oracle commit intervals?
What settings are you using in your Oracle stage? What are the settings of the two environment variables that control Oracle commit intervals?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.