QC environment in ETL

[thumsup9]

Hi,

In our company, we have Dev, QC, and PROD environments for ETL. PROD is the only environment that is scheduled. QA is manually run by the developers based on the jobs they need to kick off. They do a prod data restore onto QC database before running their fixes. We are facing challenges in ETL testing. One of the problems is that multiple changes are happening on the same table as part of two different change requests and multiple developers working on the same table in QA. For some reason due to the data loads they are doing independently of each other, the test cases pass on one day and fail on the other day.

This issue is happening more because the QC environment is not scheduled but manual and the developers are not running all the dependent jobs as required. In some cases the developers need to reload the historical data and they think scheduling the jobs nightly would mean only incremental data, but my question was why cant they set the date parameter to a back date and run the loads or have one time jobs to have historical data loaded but stillhave the incremental jobs scheduled like in production.

Where can i get information on the ETL QC environment standards that should be followed ? Please advice

Thanks

[PaulVL]

Wow, that's a fair degree fo chaos you have going on over there.

Do you have a team responsible for Database table design? Who are the custodians to ensure that 1 version of the table is under developement and coordinating the PROD deployment of that table structure.

An ETL standards document is not going to help you much with your situation.

You want your QA / QC environment to mirror your PROD environment for existing job creation and migration.

You want to also be able to test the new and upcoming table layout that will be migrated to PROD at some point in the near future.

The fact that you have two competing table layouts in your non production environment... may mean you need to request another non-prod database to be a firm mirror of what PROD is.

Talk to your DBAs to see what can be done to accomodate for your competing table layouts.

[ray.wurlod]

In a word, "management". You need to manage (co-ordinate) your QC environment.

As far as I am aware, there are no standards for this.