DSXchange

What are the various parameters needed for ETL performance

vikas gupta wrote:What are the various parameters needed for ETL performance

Hi Vikas

There are many parameters on which your ETL performance depends. I can list some of them here.

The flexibility of the architecture. It shouldn't be tool dependent.
Which database engine will be used for the transformations?
What happens to ETL developments if the physical address of a server changes or if a database name changes?
How you promote code from development to production.
Complexity of business rules.
Number of dimensions and their nature.
Platform.

There is a long list. I hope it helps u.

Regards
Ashwin

Also check for APT_BUFFER_MAXIMUM_MEMORY, APT_BUFFER_FREE_RUN, Buffering policies, Database parameters.

There are a lot of posts covered on ETL performance. I just did a search and got several posts. Also, your answer depends upon what do you mean by performance.

The original question is like asking "What can I do to tune my vehicle" in a car forum and can't really be answered, at least directly.

Reading and writing data takes a long time. Try to read and write and "land" data to disk as few times as possible. This approach (while not always correct) usually works pretty well for a starting point.

vikas gupta wrote:What are the various parameters needed for ETL performance

The first thing you need is some definition of "performance". In ETL the key performance indicator is usually the ability to meet some time window in which access to the databases in question is permissible. Nothing else really matters.

For example, you may be able to process the same volume of data faster just by throwing more processing nodes at it.

Intelligent design and some awareness of which techniques are more efficient than others. Some knowledge of how the ETL tool interacts with data sources (and what alternatives are available) is also vital.

..

..

Arent you having fun with those dots