DSXchange

Hi

i would like to hear from experts here on what exactly is a cofigurable ETL engine or Dynamic schema driven etl engine and how does Datastage fit into it.

Can you please explain the actual need for a configurable ETL solution and the basic necessity that drives these ideas and any insights on practical implementation?

thanks in advance.

No Inputs ?????

Sleep ?????

Your post is not simple nor short in answer. The shortest I can come is that it's NOT currently within the DataStage portfolio to support a fully dnamic schema driven etl engine. DS is compiled code within most cases, and while it can use built in operators with dynamic capabilities, once you move into things like Transformers and custom operators you move away from dynamic.

As for explaining the need and necessity and insights, that's a long answer.

Google?

A configurable ETL engine is an ETL engine that can be configured.

DataStage can be configured in a number of different ways, including uvconfig, dsenv, osh configuration file, and so on. It is, therefore, by any reasonable definition, a configurable ETL engine.

Some stages allow a schema file to define the record layout being processed. Assuming that the schema file can change, runtime column propagation might allow a certain degree of dynamism though, of course, and transformation expression must name an explicit input column.

I've never seen the need for it. Others have, and have posted on the forum. Search is your friend. In practice the metadata of the data flowing through an ETL process do not change much over time. Sufficiently infrequently, in fact, that re-design is usually the more efficient way to react.