ETL Specification

MattKeevil · Post by **MattKeevil** » Tue Jun 24, 2008 1:56 pm

I am trying to put together design documents for ETL jobs, was wondering if anyone has any good templates... didn't want to recreate the wheel.

sreddy · Post by **sreddy** » Tue Jun 24, 2008 3:21 pm

MattKeevil

Go through your BRD, discuss with your BA/Team Lead.
They have to clarify all your droughts. Then start with TDD.
In TDD you have given complete flow of mapping (column level)
Transformation Rules based on Business logic, Pseudocode, description purpose of this job.
Check with your team if they have any standard template that is good.

MattKeevil wrote:I am trying to put together design documents for ETL jobs, was wondering if anyone has any good templates... didn't want to recreate the wheel.

ray.wurlod · Post by **ray.wurlod** » Tue Jun 24, 2008 4:07 pm

Welcome aboard. :D

The best kind of design document for ETL is a target-from-source mapping. Investigate the FastTrack product which gives a spreadsheet-like interface to doing just that and can then generate DataStage jobs at the click of a command button.

MattKeevil · Post by **MattKeevil** » Tue Jun 24, 2008 6:29 pm

Thanks... glad to be here...

So what I'm looking for is a good template to create design specifications for ETL (Datastage) jobs. Something I can write which would contain the ETL source mappings/transformations and also the Database changes. This document needs to be done (pre development) so I can hand it off to a developer. The FastTrack software looks like it's a POST development documentation tool, which is not what I'm looking for. We have datafeed specifications that define the source data but I'm lacking the overall design documentation, which is what I'm creating... just looking to see if anyone has any good templates.

chulett · Post by **chulett** » Tue Jun 24, 2008 6:32 pm

MattKeevil wrote:The FastTrack software looks like it's a POST development documentation tool

Nope. Pre.

vmcburney · Post by **vmcburney** » Tue Jun 24, 2008 10:29 pm

IBM are going to release a version of FastTrack that can reverse engineer existing jobs and create a mapping specification from them. According to the [urlhttp://blogs.ittoolbox.com/bi/websphere/archives/ibm-information-server-and-datastage-roadmap-for-2008-and-2009-24854]roadmap[/url] it's in Information Server 8.2 and not due until the start of 2009.

A DataStage Design document is just a standard interface spec with source, target, business rules, rollback and recovery, scheduling etc.

MattKeevil · Post by **MattKeevil** » Wed Jun 25, 2008 5:02 pm

Thanks for the information but it's not what I'm looking for... not interested right now in purchasing software... guess no one has just a document for design documentation.

vmcburney · Post by **vmcburney** » Wed Jun 25, 2008 6:57 pm

Not sure if you mean functional design, technical design, requirements, detailed design but there are lots of free resources online such as www.openmethodology.com or www.ganttproject.org or maybe www.datamigrationpro.com that describe the design phase of data integration/warehouse projects.

MattKeevil · Post by **MattKeevil** » Wed Jun 25, 2008 8:13 pm

DSXchange

ETL Specification

ETL Specification

Re: ETL Specification