Production data extarction issue

[vinaymanchinila]

Hi,
Most of the development data is extracted out of production as data quality and quantity is not even close in test and development environments. This makes the real production jobs run slow!

If we need to synchronize data in test or development with production every week on the weekend or so, what is the best process.

1) Is it better for the DBA to do it with scripts or
2) Design DS jobs to do it.

I know this is not a fullfledged DS question but hope you guys can share your ideas.

Thanks,

[ketfos]

Hi,
Extracting the production data using DataStage jobs will give you a better picture of the data contents, as you will put some check/balances/transforms to view the end data.

Ketfos

[kduke]

Depends on how much money you have. If you have enough money to mirror your entire database more than once then it changes everything. You can then break one set of mirrors and unmount them from one machine like production and then mount them to test. This is very fast and very cool.

You can do similar things with database replication. Oracle has some cool software to copy databases. If you are not using Oracle or have lots of money then DataStage is a good solution.

[vinaymanchinila]

well I donot know about the money but instead of calling for meetings with DBAs and all team members (which will take more a week) , I thought it would take the same time to just build individual jobs in DS to load Test from Prod!

[kduke]

Another good reason is control. If you do it yourself then you can control when and where.

[vinaymanchinila]

Thanks for your idea , have alreadu started it, wish there wa s a developer who wants to learn DS , they are so monotonus jobs

[ray.wurlod]

The boredom can be relieved somewhat by creating a job template with all the stage types, links and job parameters already in place. All that's required is to import then load the table definitions.