Hi all,
I am extracting aggregated data from the warehouse using oracle sql ( invovled joining dimension and fact tables , summing up some measures, some translation of data column and sorting )
Now i need to build the DS job for it.
-should i get better performance if I break the logic down into a series of stages ( like ext, join, xfm, aggre, sort...etc )?
-Or let the Oracle handle all the processing? ( I am pulling data from one source only )
Thanks and Regards,
william
Extraction Job which one is faster?
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 11
- Joined: Mon Dec 05, 2005 4:11 am
It also depeds on where you database resides. If it is costly have carry huge number of rows from database server to DS server, it might be effecient to do the necessay aggregation steps from DS by either stroed procedure or using your SQL logic, and carry the necessay records for transformation to DS server.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Also, you need to always keep in mind, what time, or what part of day or night is your process going to run when it goes live. If during that time, not much database activity is taking place, then i would suggest doing aggregations sorting and joins at the database level.Transformations can be done in datastage.
My 2 centy.
My 2 centy.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.