Insight for Project Review

[Saad]

Dear All,

We are going to conduct a review of an ETL implementation soon. Would like to know if somebody has gone through a similar experience and what important areas we should be focusing first.

Our plan is to review certain areas such as:
1. Reviewing movement of data from landing to staging and then target area.
2. Restartability and scheduling of jobs (internal/external).
3. Error handling.
4. DataStage Implementation.
5. Initiation of ETL.
etc.

Thanks,

Saad

[ray.wurlod]

Welcome aboard.

This is, of course, a huge topic. Many of us here have the experience you mention, often more than once.

IBM has a specific methodology that they use, called ITERATIONS, which begins with fundamental questions like "is the enterprise ready for a data warehouse?". You might do well to contact them, I don't know whether they sell the methodology separately or only as a service. But it is a good, structured and - above all - self-documenting (mainly using Microsoft Office products) set of techniques.

Or involve an experienced consultant in the evaluation phase. The cost will easily be justified in the amount of time not wasted because you have leveraged that experience.

[Saad]

Ray,

Thanks for sharing your views. Actually by 'Review' i mean that we are going to do the Audit of an existing ETL implementation. We would be looking into certain areas as mentioned in my email and then recommending based upon our findings.

What I would like to know are other related areas of importance and which should be analyzed carefully. Any insight would be highly appreciated.

Thanks,

Saad

[ray.wurlod]

You have missed probably THE most important questions, if it's a review of an existing implementation. These are two.

1. Are the end-users' requirements being met and, if not, why not and how could they be?

2. Are the key performance indicators of the ETL being met (usually these are expressed in terms of successfully meeting the available time windows for the extraction and loading phases)? Again, if not why not, and is there anything that can ameliorate this situation.

If there are known future plans for the enterprise, these too can be accommodated in the review as well as in ongoing ETL development.

There are many other aspects. You may be interested in reviewing what my business offers in the way of a DataStage health check.

[ray.wurlod]

You have missed probably THE most important questions, if it's a review of an existing implementation. These are two.

1. Are the end-users' requirements being met and, if not, why not and how could they be?

2. Are the key performance indicators of the ETL being met (usually these are expressed in terms of successfully meeting the available time windows for the extraction and loading phases)? Again, if not why not, and is there anything that can ameliorate this situation.

If there are known future plans for the enterprise, these too can be accommodated in the review as well as in ongoing ETL development.

There are many other aspects. You may be interested in reviewing what my business offers in the way of a DataStage health check.

[Saad]

Thanks Ray for a very thorough and useful insight. This will definitely help us in our new assignment.

Saad