How to Handle DataQuality?

kishorebhulokam · Post by **kishorebhulokam** » Sun Jul 20, 2008 5:58 pm

Hi,

can any one please tell me how to handle the dataquality in ETL jobs.

Thanks in Advance...

ray.wurlod · Post by **ray.wurlod** » Sun Jul 20, 2008 6:08 pm

Initial skepticism is what I've always found to be best. Disbelieve everything you are assured about data quality (if it's good) and CHECK. Profile the data to learn what's really there (ProfileStage is good here, and AuditStage is good for verifying compliance with business rules*). Based on the profile, design any requirements that may exist for cleaning and standardizing the data, and implement those requirements with DataStage/QualityStage.

* Your organization does have its business rules documented, doesn't it? The process of profiling may well uncover other, undocumented characteristics of data that turn out also to be business rules.

whenry6000 · Post by **whenry6000** » Thu Jul 31, 2008 7:15 am

ray.wurlod wrote:Initial skepticism is what I've always found to be best. Disbelieve everything you are assured about data quality (if it's good) and CHECK. Profile the data to learn what's really there (ProfileStage is good here, and AuditStage is good for verifying compliance with business rules*). Based on the profile, design any requirements that may exist for cleaning and standardizing the data, and implement those requirements with DataStage/QualityStage.

* Your organization does have its business rules documented, doesn't it? The process of profiling may well uncover other, undocumented characteristics of data that turn out also to be business rules.

Do ProfileStage and AuditStage still exist in 8.0?? I thought ProfileStage was replaced by Information Analyzer, and I'm not sure about what has replaced AuditStage, if anything.

ray.wurlod · Post by **ray.wurlod** » Thu Jul 31, 2008 4:59 pm

Original post was for 7.x. You are correct that ProfileStage no longer exists in 8.0 having been morphed into Information Analyzer. Elements of QualityStage and AuditStage will also migrate into Information Analyzer over the next couple of years. In the meantime, AuditStage continues to exist as a separate product.