There was a recent post inquiring about a UNIX (server-side) DS export utility, which doesn't seem to exist. I think the person who posted the question was thinking along the same lines that I was for a backup strategy. That it would be nice to export all the projects to the file system and then back everything up.
I had always heard that it was necessary to have both (exports and file system backup), but based on some of the responses to that post, I'm wondering if this is correct. Can someone shed some light on the basic steps for a backup process that can be run every night? I'd especially be interested in any thoughts about how you deal with existing connections, or how the backup procedure may be different on the production ETL server vs. the development server.
Regards, Kevin
basic backup strategy
Moderators: chulett, rschirm, roy
Hi,
in your docs directory read the man_gde.pdf file in the "Importing, Exporting, and Packaging Jobs" section for the command line usage of exporting etc'.
theese utilities are client side utilities, never the less, once you have the .dsx file you can transfer it to a "safe" place of your choice.
IHTH (I Hope This Helps),
Roy R.
in your docs directory read the man_gde.pdf file in the "Importing, Exporting, and Packaging Jobs" section for the command line usage of exporting etc'.
theese utilities are client side utilities, never the less, once you have the .dsx file you can transfer it to a "safe" place of your choice.
IHTH (I Hope This Helps),
Roy R.
There is no server side export that produces a .dsx file on the server. The only way to get a .dsx is through the FAT client export in DS Manager. The only way to automate that is through a PC script on a client machine. If you're NT server, then you can have the DS Client on the NT server do the export. But if you're UNIX server, you have to use the DS Client which resides on a PC.
Now an UNIX or NT file system backup is a different thing. You cannot selectively archive jobs, logs, routines, etc. It's pretty much an all or nothing backup. The problem is that if during a backup jobs are actively running or being edited, you could run a backup that captures hash files in flux, and therefore if you tried to restore the backup you would restore internal DS files out of sync, and potentially corrupt said hash files. The file system level backup is only good on an absolutely quiesced file system, and only gets you back THAT EXACT IMAGE. Which means, in order to restore a job design from two days ago, you would have to revert your entire image back to that day.
I still suggest a file system backup only as a disaster recovery plan. You should be using a version control tool such as PVCS for storing the system of record code. It is nuts to spend several years and tens of millions building a data warehouse and fail on the simple things like software development lifecycle, source code control, and disaster recovery plans.
My recommended practices include daily end-of-day exports of development projects snapshotted into an archive by day, for selective restore if required. We never use the automated scripts, as we want a guaranteed export, not a client hung waiting for you to click on okay if a job is locked. We never leave source code (jobs, scripts, sql scripts, routines, etc) open in any editor overnight. At the end of a release development, that version is checked into PVCS. Always restore and promote from PVCS. We "package" from PVCS, never using DataStage's internals functionality.
Good luck!
Kenneth Bland
Now an UNIX or NT file system backup is a different thing. You cannot selectively archive jobs, logs, routines, etc. It's pretty much an all or nothing backup. The problem is that if during a backup jobs are actively running or being edited, you could run a backup that captures hash files in flux, and therefore if you tried to restore the backup you would restore internal DS files out of sync, and potentially corrupt said hash files. The file system level backup is only good on an absolutely quiesced file system, and only gets you back THAT EXACT IMAGE. Which means, in order to restore a job design from two days ago, you would have to revert your entire image back to that day.
I still suggest a file system backup only as a disaster recovery plan. You should be using a version control tool such as PVCS for storing the system of record code. It is nuts to spend several years and tens of millions building a data warehouse and fail on the simple things like software development lifecycle, source code control, and disaster recovery plans.
My recommended practices include daily end-of-day exports of development projects snapshotted into an archive by day, for selective restore if required. We never use the automated scripts, as we want a guaranteed export, not a client hung waiting for you to click on okay if a job is locked. We never leave source code (jobs, scripts, sql scripts, routines, etc) open in any editor overnight. At the end of a release development, that version is checked into PVCS. Always restore and promote from PVCS. We "package" from PVCS, never using DataStage's internals functionality.
Good luck!
Kenneth Bland