Hi
i would like to hear from experts here on what exactly is a cofigurable ETL engine or Dynamic schema driven etl engine and how does Datastage fit into it.
Can you please explain the actual need for a configurable ETL solution and the basic necessity that drives these ideas and any insights on practical implementation?
thanks in advance.
Configurable ETL/ Dynamic Schema Driven ETL
Moderators: chulett, rschirm, roy
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Your post is not simple nor short in answer. The shortest I can come is that it's NOT currently within the DataStage portfolio to support a fully dnamic schema driven etl engine. DS is compiled code within most cases, and while it can use built in operators with dynamic capabilities, once you move into things like Transformers and custom operators you move away from dynamic.
As for explaining the need and necessity and insights, that's a long answer.
As for explaining the need and necessity and insights, that's a long answer.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
A configurable ETL engine is an ETL engine that can be configured.
DataStage can be configured in a number of different ways, including uvconfig, dsenv, osh configuration file, and so on. It is, therefore, by any reasonable definition, a configurable ETL engine.
Some stages allow a schema file to define the record layout being processed. Assuming that the schema file can change, runtime column propagation might allow a certain degree of dynamism though, of course, and transformation expression must name an explicit input column.
I've never seen the need for it. Others have, and have posted on the forum. Search is your friend. In practice the metadata of the data flowing through an ETL process do not change much over time. Sufficiently infrequently, in fact, that re-design is usually the more efficient way to react.
DataStage can be configured in a number of different ways, including uvconfig, dsenv, osh configuration file, and so on. It is, therefore, by any reasonable definition, a configurable ETL engine.
Some stages allow a schema file to define the record layout being processed. Assuming that the schema file can change, runtime column propagation might allow a certain degree of dynamism though, of course, and transformation expression must name an explicit input column.
I've never seen the need for it. Others have, and have posted on the forum. Search is your friend. In practice the metadata of the data flowing through an ETL process do not change much over time. Sufficiently infrequently, in fact, that re-design is usually the more efficient way to react.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.