DataStage Backup and Recovery from Server

ray.wurlod · Post by **ray.wurlod** » Tue Jun 10, 2008 4:38 pm

Before I provide a greater level of detailed feedback, have you tested all of these commands in version 8 parallel environment?

If so please provide detail concerning your test cases, expected and actual results. If not, please say so so that people will understand that this is purely a theoretical discussion not based on actual practice.

Did you attempt to restore after damaging/deleting a component?

ag_ram · Post by **ag_ram** » Tue Jun 10, 2008 11:01 pm

I have tested all of these commands in DataStage v7.5.x2.(Windows Server).

Is there any restriction beyond our views on using these commands as to mean that this would be of theoretical at all point of time?

Did you attempt to restore after damaging/deleting a component?

Can your question be much clearer? (Restoring to where?)

ray.wurlod · Post by **ray.wurlod** » Tue Jun 10, 2008 11:08 pm

Let's say you delete a job and need to restore it. Please explain the process. You know the job name but no other information.

ray.wurlod · Post by **ray.wurlod** » Thu Jun 12, 2008 12:57 am

I have deleted the job. I have no idea what the job number is, much less the key values for DS_JOBOBJECTS to specify to uvrestore. How, indeed, can you be certain that you have identified ALL the DS_JOBOBJECTS records for this job?

Further questions. What if I want to restore the job executable (either instead of or as well as the job design)? What if I want to restore the job log and status information?

ArndW · Post by **ArndW** » Thu Jun 12, 2008 1:00 am

And after you finish thinking about Ray's questions, think about Parameter Sets and Shared Containers.

ray.wurlod · Post by **ray.wurlod** » Thu Jun 12, 2008 1:07 am

... and Routines upon which the job depends.

Oh, yes, I'm a great believer in using Data Elements to map my business metadata into DataStage. And I'd like to have each table definition that refers to the job restored as well.

ag_ram, can you please advise how to do all those things in uvrestore, with no prior knowledge of the job number?

ag_ram · Post by **ag_ram** » Fri Jun 13, 2008 7:32 am

ray.wurlod wrote:I have deleted the job. I have no idea what the job number is, much less the key values for DS_JOBOBJECTS to specify to uvrestore. How, indeed, can you be certain that you have identified ALL the DS_JOBOBJECTS records for this job?

Though you are not aware of the Job no of the deleted Job, when restoring that particular record from DS_JOBS, you will be able to find the orininal Job no. But, with this, you will NOT be able to find the record IDs which are deleted from DS_JOBOBJECTS.

When you need to restore a deleted Job, you must have known the Job no and corresponding record IDs of the deleted Job in DS_JOBOBJECTS.

ray.wurlod wrote:Further questions. What if I want to restore the job executable (either instead of or as well as the job design)? What if I want to restore the job log and status information?

Because Job executables do include the RT_BPnnn, RT_SCnnn and RT_BPnnn.O and uvbackup does NOT ensure taking back up of Type 19 hashed files, we can not take back up of a complete Job executable of a Compiled/run Job using uvbackup. (till my knowledge)

Respective Job Log(RT_LOGnnn), Job Status(RT_STATUSnnn), Job Config(RT_CONFIGnnn) files could be restored correctly but the tragedy is that those could not be identified by DataStage, though we change the file names with the new job no and adding VOC entries for file defintion records as below.

Code: Select all

uvrestore -F C:\Ascential\DataStage\Projects\Development\RT_CONFIG456=C:\Ascential\DataStage\Projects\Test\RT_CONFIG108 -F C:\Ascential\DataStage\Projects\Development\RT_STATUS456=C:\Ascential\DataStage\Projects\Test\RT_STATUS108 -F C:\Ascential\DataStage\Projects\Development\RT_LOG456=C:\Ascential\DataStage\Projects\Test\RT_LOG108 -F C:\Ascential\DataStage\Projects\Development\DS_TEMP456=C:\Ascential\DataStage\Projects\Test\DS_TEMP108 -V -U "D:\Project_Development_full"

Code: Select all

uvrestore -F C:\Ascential\DataStage\Projects\Development\VOC=C:\Ascential\DataStage\Projects\Test\VOC -R "RT_CONFIG4"="RT_CONFIG1" -R "RT_LOG4"="RT_LOG1" -R "RT_STATUS4"="RT_STATUS1" -R "DS_TEMP4"="DS_TEMP1" -V -U "D:\Project_Development_full"

But this attempt may be a workaround now as those four files are completly restored but not identified by the respective restored Job.

ag_ram · Post by **ag_ram** » Fri Jun 13, 2008 7:35 am

ArndW wrote:And after you finish thinking about Ray's questions, think about Parameter Sets and Shared Containers.

Regarding Parameters Set, I have NOT tested all these commands in DataStage v8.0. But anyway in DS V7.5 all Shared Containers could be backed up and restored correctly as the Job Design information of a Shared Container is stored in DS_CONTAINERS and DS_JOBOBJECTS files.

ArndW · Post by **ArndW** » Fri Jun 13, 2008 7:38 am

But I am talking about restoring one job that uses a shared container. You can't just overwrite the hashed files.

ag_ram · Post by **ag_ram** » Fri Jun 13, 2008 7:49 am

Any Shared Container which the Job uses, can be backed up and restored with the Job from the recent back up file if we know Job No, Container No and Record IDs of that Job Design in DS_JOBOBJECTS and Container's Job Design in DS_JOBOBJECTS.

ArndW · Post by **ArndW** » Fri Jun 13, 2008 8:56 am

ag_ram: You don't seem to get the point I am making

You backup the DS_JOBOBJECTS as you've described, then decide to restore your job a half day later, after your 10 colleagues have spent time modifying their jobs.

You use the complex methods you have explained earlier in this thread to identify the job in the backups that you want to restore.

But you cannot just overwrite the DS_JOBOBJECTS as that would corrupt the system and your 10 colleagues will probably start discussing lynch justice. If you don't get the shared container information from the backed up DS_JOBJOBJECTS then your restore won't work.

ag_ram · Post by **ag_ram** » Sat Jun 14, 2008 10:35 am

No ArndW.

You did not make any point out there but instead, asked me to check the range of possibilities on using uvbackup/uvrestore method by raising two questions regarding Shared Containers. And for that, i replied to you with pleasure.

Shall i ask you one question - Did you see the contents(especially Step 8 ) under heading "Job level backup and recovery: (Job Design)" in my first post. And may i recommend you to remember what -R record=newrecord option does at the moment of having selective recovery?

Please stay away from initial thoughts on restoring what we need as we reached a momentum to have a break.

ArndW · Post by **ArndW** » Sat Jun 14, 2008 10:43 am

ag_ram - I asked you a question and instead of answering it, you wrote a couple of sentences that neither apply to the question nor answer anything else.

For clariity, I shall repeat my question: Using your proposed method listed above, how can I restore a job that has a shared container without disrupting work that has been done in the project since the backup.

Kindly respond directly and do not pose unrelated questions or other obfuscations. Just answer the question.

kduke · Post by **kduke** » Sat Jun 14, 2008 3:58 pm

ag_ram

I have been using Universe since 1986 and I would never do any of this. Export/Import is by far the best way to recover a job. Your advice seems to be misleading on purpose. If you keep giving bad advice then we need to ban you or do something drastic.

You argue with Ray, Arnd and others, our most devoted posters. I wish you would just quit or leave this site. Our best posters do not need to argue with you on a regular basis. Most of the posters here are or try to be professional.

ArndW · Post by **ArndW** » Sun Jun 15, 2008 3:23 am

ag_ram - Even though you have described the involved procedure, you seem to be getting further and further away from the original precepts to this whole discussion.

The original question was how to do this from the server and the inital responses were "you cannot do this without resorting to programming and using undocumented functionality". You stated that it could be done and proceeded to put a lot of effort into using undocumented tables, columns and facets of the product that are guaranteed to change in the near future.

Several DSXchangers (including myself) have written and implemented such server-side programs in past - but with the explicit warning that "this will probably cease to work with some future version". If I were asked to do this again today I would fight hands-and-teeth to avoid such a solution.

Each time someone points out an ommission in functionality you are forced to delve deeper into the internal table structures and use undocumented and unsupported features; something every consultant avoids like the plague. And, having worked at support for Universe at Prime/VMark/Ardent/Ascential from 1985 onwards, I can guarantee you that any support call with issues related to using these tables will be closed within minutes of discovering code using unspported features - even if the problem has nothing to do with that part of the code.

If you look at your described procedures, processes and programs do you actually think that it can be implemented over a solution of keeping a PC turned on at night and doing an automated scheduled full backup per "at" and then having an operator (with no knowledge of DataStage) restore a single job from the client using just 7 mouse clicks?