DSXchange

Guys,

I need advice. We had someone else (who is more or less gone now) set up our TEST environment, with the understanding that our DataStage Project would be copied down from PROD to TEST.

This may not have been done. We have found a TEST Job that was modified extensively, but never successfully run in TEST. The much simplified version was never successfully run in PROD either. This brings into question the entire DS Project in TEST.

My searches here uncovered an interesting backup batch file from Kim Duke, who we tragically lost recently. If I could get a copy of that somehow, that would be great.

I've played around with the Export / Import in the Designer Client, but I have not warmed up to it yet. I have gotten some frustrating messages when copying one Job into TEST, and I'm still working out the Export options to use.

Can someone give me beginner-level tips on using the Designer Export / Import utility, or give me any advice you wish?

Thank you,
john3

I thought Kim's script was posted here but I couldn't turn it up in the time I had to search for it. From what I recall it was bundled into his ETLStats package, which I'm pretty sure Ray still hosts. It may also be something I have at home, I'll try to remember to check later.

There could be significant differences between our environments, so the following is offered with a grain of salt.

First: does your PROD project include both executables and designs for every job component? If not, that will be your first problem to solve. I spent several weeks working out the best "match" between orphan PROD executables and non-PROD project designs.

If you are lucky to have no PROD orphans, my advice is to do a full backup of the TEST project, then do a full export of all PROD design components only. Import them to TEST and compile them there.

Here, we have a hard rule: never compile in PROD. 100% test coverage is the goal, and putting those executables in PROD means if there's a defect, it is nearly always because of an environmental difference between TEST and PROD.

I have one project with confirmed no orphans. We are in the midst of migrating from 8.7 to 11.5. The first step was to import the PROD designs to the new DEV server, do a multi-job compile, and work out the kinks from there. I know of no better way to handle a migration, and it fits directly from the discipline of having exactly the same designs and executables in TEST and PROD.

I'm also a big fan of no compiles in prod.

We do not allow Operations folks to have developer role in prod. Super Op only.

"IF" something needs to be compiled in prod, admins do it.


now to export from prod and import to test...

from the sounds of your story I would research why your job is failing to begin with. Seems that would be an easier step to accomplish and a good learning experience as well.

I've done this at a couple of sites where their code migrations had poor controls. When I walked in, they were basically having to copy code from Prod all the time because they had no confidence in their processes.

Recomendations:

1) Do a full backup of Test. This includes Jobs, Routines, Parameter Sets, UNIX scripts and anything else used by the jobs (table def's, QualityStage stuff, etc.). Also dump Environment variables from the Admin client just to be safe. That way you can always restore anything if required.
2) We actually deleted / recreated projects in Test, but that's not really recommended, and requires lots of additional steps. If you want to make sure you have no "odd" code that never made it to prod, then you will need to delete all the existing jobs. Then run a DS.REINDEX ALL after the clean-up.
3. Then load the jobs from production (source code only, no object code). Do NOT load the environment variables from prod. If you do load the parameter sets, be very careful to inspect them to insure nothing accidentally points to Prod.
4. Mass compile. If you have missed something, that should point it out. Note: if you have a lot of jobs, mass compile will fail mid-way through, just restart it telling it to skip compiled jobs.
5. Check scripts. I have seen lots of sites where scripts are a real problem because they are manually migrated and not in the source code control system. If you decide to synchronize scripts from Prod be VERY careful, because hard-coding Prod systems / DB's in scripts is unfortunately very common and can be extremely difficult to identify (for example "P" instead of a "T" in some names). If hard-coded Prod settings are identified, change them to use the correct settings for Dev / Test / Prod based on the server that is running the script.

Note - per your request I have added a new FAQ that delivers the Backup script that used to be a part of ETLStats:
viewtopic.php?p=473930

Because there is a risk of impacting Production, this needs to be done very carefully and with lots of oversight.

asorrell wrote:Note - per your request I have added a new FAQ that delivers the Backup script that used to be a part of ETLStats

Why tanks, Mista Andy!

Many thanks everyone. I am humbled and intimidated.

Lots of good information here. I hope we can do this.

Fingers crossed,
john3

Franklin,
Luckily we seem to have designs and executables for all Jobs in PRD. Thank you for your advice.
john3

Andy Sorrell,

Thank you, thank you, thank you! Your advice looks excellent. I do wonder about my ability. I'm thinking this resolves my general advice question. However please feel free to continue contributing as you see fit.

john3