We are using DCS product to migrate our data stage jobs.
Since this is our first attempt to move items to production, we were wondering what migration practices other Data Stage users had in place.
If using DCS, what do you do with the Jobs that were migrated up from DEV - to TEST - to PROD... (tedious, since you can only be in Dev to make any changes, i.e. ..OCI Prod ids, pswds etc...before migrating up Test, then to Prod) Do you leave them in those Projects(Dev & Test...?), Do you delete them out after migrated to PROD ?
Any simple incites would be appreciated.
Thank You,
Patricia Desiano
Anheuser-Busch, Inc.
MSG - BudNET Data Warehouse
314-577-7558
Process for Migration
Patricia,
This part of our system is not very sophisticated. We simply use the DataStage export/import. However, we do have some practices that might answer you questions.
We make all our changes to jobs in dev. We then migrate the jobs to test and prod (and yes, it is VERY tedious).
As a rule, we do NOT make any changes to jobs in test or prod. We do leave the jobs in dev and test. We also keep a copy (history) of the export files so that we can roll back to a previous version should there be a major problem with a release.
We do not change OCI DSNs, Userids or passwords when migrating to prod. These are all parameters. The defaults are always set to the dev oracle database, except perhaps for some top-level control jobs. As these parameters are passed down from a calling job to a called job, we only need to set the parameters correctly once in the top-level job when scheduling it.
Anything that needs to be different between dev and prod (including file
paths) is generally driven by parameters (we do use a little slight-of-hand
here) so that there is never any need to change a job when migrating it from dev to prod. If you had to change the job, that would defeat the purpose of all the testing, and seriously compromise the integrity of the release process.
Hope this helps.
David Barham
Information Technology Consultant
InformAtect Pty Ltd
Brisbane, Australia
e-mail: david@barham.hm
-----Original Message-----
From: Desiano, Patricia [mailto:Patricia.Desiano@anheuser-busch.com]
Sent: Tuesday, 31 July 2001 7:25 AM
To: datastage-users-list
Subject: Process for Migration
We are using DCS product to migrate our data stage jobs.
Since this is our first attempt to move items to production, we were wondering what migration practices other Data Stage users had in place.
If using DCS, what do you do with the Jobs that were migrated up from DEV - to TEST - to PROD... (tedious, since you can only be in Dev to make any changes, i.e. ..OCI Prod ids, pswds etc...before migrating up Test, then to Prod) Do you leave them in those Projects(Dev & Test...?), Do you delete them out after migrated to PROD ?
Any simple incites would be appreciated.
Thank You,
Patricia Desiano
Anheuser-Busch, Inc.
MSG - BudNET Data Warehouse
314-577-7558
This part of our system is not very sophisticated. We simply use the DataStage export/import. However, we do have some practices that might answer you questions.
We make all our changes to jobs in dev. We then migrate the jobs to test and prod (and yes, it is VERY tedious).
As a rule, we do NOT make any changes to jobs in test or prod. We do leave the jobs in dev and test. We also keep a copy (history) of the export files so that we can roll back to a previous version should there be a major problem with a release.
We do not change OCI DSNs, Userids or passwords when migrating to prod. These are all parameters. The defaults are always set to the dev oracle database, except perhaps for some top-level control jobs. As these parameters are passed down from a calling job to a called job, we only need to set the parameters correctly once in the top-level job when scheduling it.
Anything that needs to be different between dev and prod (including file
paths) is generally driven by parameters (we do use a little slight-of-hand
here) so that there is never any need to change a job when migrating it from dev to prod. If you had to change the job, that would defeat the purpose of all the testing, and seriously compromise the integrity of the release process.
Hope this helps.
David Barham
Information Technology Consultant
InformAtect Pty Ltd
Brisbane, Australia
e-mail: david@barham.hm
-----Original Message-----
From: Desiano, Patricia [mailto:Patricia.Desiano@anheuser-busch.com]
Sent: Tuesday, 31 July 2001 7:25 AM
To: datastage-users-list
Subject: Process for Migration
We are using DCS product to migrate our data stage jobs.
Since this is our first attempt to move items to production, we were wondering what migration practices other Data Stage users had in place.
If using DCS, what do you do with the Jobs that were migrated up from DEV - to TEST - to PROD... (tedious, since you can only be in Dev to make any changes, i.e. ..OCI Prod ids, pswds etc...before migrating up Test, then to Prod) Do you leave them in those Projects(Dev & Test...?), Do you delete them out after migrated to PROD ?
Any simple incites would be appreciated.
Thank You,
Patricia Desiano
Anheuser-Busch, Inc.
MSG - BudNET Data Warehouse
314-577-7558
My 2-cents worth. Build a parameter file for each environment, something like a simple text file. Set permissions on the file so that peeking eyes cant read it. Write a simple routine to set parameters in your job at runtime to the values in the parameter file. This allows you to build a migratable parameter file for each environment, have a single point to change any passwords, etc, and further simplify what youre trying to do. THIS IS THE BEST SOLUTION.
---------------------------------------------------------------------------------------------------------------------------------------
* PROD parameters ini
[Directories] InputFileDirectory=/DataStageWork/hif_prod/inputs/
SourceFileDirectory=/DataStageWork/hif_prod/source/
LoadFileDirectory=/DataStageWork/hif_prod/load/
UnloadFileDirectory=/DataStageWork/hif_prod/unload/
ErrorFileDirectory=/DataStageWork/hif_prod/errors/
LogFileDirectory=/DataStageWork/hif_prod/logs/
WarningFileDirectory=/DataStageWork/hif_prod/warnings/
ViolationFileDirectory=/DataStageWork/hif_prod/violations/
TempFileDirectory=/DataStageWork/hif_prod/temp/
RejectFileDirectory=/DataStageWork/hif_prod/rejects/
ArchiveFileDirectory=/DataStageWork/archive/
CommonScriptDirectory=/DataStageWork/hif_prod/scripts/
[Servers]
StagingDSN=Training-MSA
StagingUserID=kxb32
StagingPassword=abcdefg
ODSDSN=ODS
ODSUserID=kxb32
ODSPassword=abcdefg
DMDSN=dm
DMUserID=kxb32
DMPassword=abcdefg
---------------------------------------------------------------------------------------------------------------------------------------
The second best solution is to have a program mass replace the default values that been HARD-CODED in to your jobs. (As you can see I dont agree with this approach. Paste the following into a batch job called Batch::UTILSetDefaultParameters. This utility will read a job of your choice and mass replace other jobs with those parameters or values in your chosen job. It will not remove existing parameters, only update matching parameters and add missing. I recommend setting up a job to house parameters for each of your environments. You must recompile all jobs
after mass-deploying this parameter change. Simply run this utility in
each environment to mass replace the default values. Setup four parameters in the job to prompt for the following:
Update: User has to answer Y to actually run the job, default to N
Folder: Folder to which the job will isolate selection of jobs
SearchText: Case-sensitive piece of selection text in job name
PrototypeJob: Jobname of prototype that holds parameters
---------------------------------------------------------------------------------------------------------------------------------------
If Update = "Y" Then
Open "DS_JOBS" TO F.JOBS Then
Read prototypenbr from F.JOBS,PrototypeJob Then
PROTOFILE = "DS_JOB":prototypenbr
Open PROTOFILE TO F.PROTO Then
Read prototype From F.PROTO,"ROOT" Then
Call DSLogInfo(PrototypeJob:" contains: ":prototype,
"Msg")
TCLcmd = SSELECT DS_JOBS LIKE "...:Searchtext:..."
If Folder # "" Then TCLcmd := WITH F3
= ":Folder:"
Call DSLogInfo(TCLcmd, "Msg")
EXECUTE TCLcmd CAPTURING OUTPUT
Done = @FALSE
Loop
Readnext ID Else Done = @TRUE
Until Done Do
If ID # "NextNumber" Then
Read job From F.JOBS,ID Then
job.number = job
JOBFILE = "DS_JOB":job.number
Open JOBFILE TO F.FILE Then
Readu rec From F.FILE,"ROOT" LOCKED
Call DSLogInfo("Job ":ID:" is locked for update","Msg")
End Then
Call DSLogInfo("Updating Job
":ID,"Msg")
before = rec
ColNames = prototype
ColCount = DCount(ColNames,@VM)
recColCount = Dcount(rec, @VM)
For i=1 to ColCount
ColName = ColNames
Locate ColName IN rec SETTING PTR Then
For j=15 to 20
rec = prototype
Next j
If ColName = "JobName" Then rec = ID
End
End Else
recColCount += 1
rec = field(rec,"/",1):"/":recColCount
For j=14 to 20
rec = prototype
Next j
If ColName = "JobName" Then rec = ID
End
Next i
Write rec on F.FILE,"ROOT"
End Else
Call DSLogInfo("Job ":ID:" does not have a ROOT record","Msg")
Release F.FILE,"ROOT"
End
End
End
End
Repeat
End Else
Call DSLogInfo("Unable to read ":PrototypeJob:" record
ROOT","Msg")
End
End Else
Call DSLogInfo("Unable to open ":PrototypeJob:" job file
":PROTOFILE,"Msg")
End
End Else
Call DSLogInfo("Unable to read ":PrototypeJob:" job record in DS_JOBS","Msg")
End
End Else
Call DSLogInfo("Unable to open DS_JOBS file","Msg")
End
End Else
Call DSLogInfo("No processing, update not set to Y","Msg")
End
---------------------------------------------------------------------------------------------------------------------------------------
If youre interested in having this utility and a mass-recompile utility job, set me an email and Ill reply with a .dsx file attached. I also have the job control logic for setting job parameters at runtime, doing all these bells and whistles. Its fully documented so you should have no trouble setting it up and using it.
Thanks,
-Ken
david@barham.hm on 30-Jul-2001 19:25
Please respond to datastage-users@oliver.com
To: datastage-users
cc:
Subject: RE: Process for Migration
Patricia,
This part of our system is not very sophisticated. We simply use the DataStage export/import. However, we do have some practices that might answer you questions.
We make all our changes to jobs in dev. We then migrate the jobs to test and prod (and yes, it is VERY tedious).
As a rule, we do NOT make any changes to jobs in test or prod. We do leave the jobs in dev and test. We also keep a copy (history) of the export files so that we can roll back to a previous version should there be a major problem with a release.
We do not change OCI DSNs, Userids or passwords when migrating to prod. These are all parameters. The defaults are always set to the dev oracle database, except perhaps for some top-level control jobs. As these parameters are passed down from a calling job to a called job, we only need to set the parameters correctly once in the top-level job when scheduling it.
Anything that needs to be different between dev and prod (including file
paths) is generally driven by parameters (we do use a little slight-of-hand
here) so that there is never any need to change a job when migrating it from dev to prod. If you had to change the job, that would defeat the purpose of all the testing, and seriously compromise the integrity of the release process.
Hope this helps.
David Barham
Information Technology Consultant
InformAtect Pty Ltd
Brisbane, Australia
e-mail: david@barham.hm
-----Original Message-----
From: Desiano, Patricia [mailto:Patricia.Desiano@anheuser-busch.com]
Sent: Tuesday, 31 July 2001 7:25 AM
To: datastage-users-list
Subject: Process for Migration
We are using DCS product to migrate our data stage jobs.
Since this is our first attempt to move items to production, we were wondering what migration practices other Data Stage users had in place.
If using DCS, what do you do with the Jobs that were migrated up from DEV
-
to
TEST - to PROD...
(tedious, since you can only be in Dev to make any changes, i.e. ..OCI Prod ids, pswds etc...before migrating up Test, then to Prod) Do you leave them in those Projects(Dev & Test...?), Do you delete them out after migrated to PROD ?
Any simple incites would be appreciated.
Thank You,
Patricia Desiano
Anheuser-Busch, Inc.
MSG - BudNET Data Warehouse
314-577-7558
---------------------------------------------------------------------------------------------------------------------------------------
* PROD parameters ini
[Directories] InputFileDirectory=/DataStageWork/hif_prod/inputs/
SourceFileDirectory=/DataStageWork/hif_prod/source/
LoadFileDirectory=/DataStageWork/hif_prod/load/
UnloadFileDirectory=/DataStageWork/hif_prod/unload/
ErrorFileDirectory=/DataStageWork/hif_prod/errors/
LogFileDirectory=/DataStageWork/hif_prod/logs/
WarningFileDirectory=/DataStageWork/hif_prod/warnings/
ViolationFileDirectory=/DataStageWork/hif_prod/violations/
TempFileDirectory=/DataStageWork/hif_prod/temp/
RejectFileDirectory=/DataStageWork/hif_prod/rejects/
ArchiveFileDirectory=/DataStageWork/archive/
CommonScriptDirectory=/DataStageWork/hif_prod/scripts/
[Servers]
StagingDSN=Training-MSA
StagingUserID=kxb32
StagingPassword=abcdefg
ODSDSN=ODS
ODSUserID=kxb32
ODSPassword=abcdefg
DMDSN=dm
DMUserID=kxb32
DMPassword=abcdefg
---------------------------------------------------------------------------------------------------------------------------------------
The second best solution is to have a program mass replace the default values that been HARD-CODED in to your jobs. (As you can see I dont agree with this approach. Paste the following into a batch job called Batch::UTILSetDefaultParameters. This utility will read a job of your choice and mass replace other jobs with those parameters or values in your chosen job. It will not remove existing parameters, only update matching parameters and add missing. I recommend setting up a job to house parameters for each of your environments. You must recompile all jobs
after mass-deploying this parameter change. Simply run this utility in
each environment to mass replace the default values. Setup four parameters in the job to prompt for the following:
Update: User has to answer Y to actually run the job, default to N
Folder: Folder to which the job will isolate selection of jobs
SearchText: Case-sensitive piece of selection text in job name
PrototypeJob: Jobname of prototype that holds parameters
---------------------------------------------------------------------------------------------------------------------------------------
If Update = "Y" Then
Open "DS_JOBS" TO F.JOBS Then
Read prototypenbr from F.JOBS,PrototypeJob Then
PROTOFILE = "DS_JOB":prototypenbr
Open PROTOFILE TO F.PROTO Then
Read prototype From F.PROTO,"ROOT" Then
Call DSLogInfo(PrototypeJob:" contains: ":prototype,
"Msg")
TCLcmd = SSELECT DS_JOBS LIKE "...:Searchtext:..."
If Folder # "" Then TCLcmd := WITH F3
= ":Folder:"
Call DSLogInfo(TCLcmd, "Msg")
EXECUTE TCLcmd CAPTURING OUTPUT
Done = @FALSE
Loop
Readnext ID Else Done = @TRUE
Until Done Do
If ID # "NextNumber" Then
Read job From F.JOBS,ID Then
job.number = job
JOBFILE = "DS_JOB":job.number
Open JOBFILE TO F.FILE Then
Readu rec From F.FILE,"ROOT" LOCKED
Call DSLogInfo("Job ":ID:" is locked for update","Msg")
End Then
Call DSLogInfo("Updating Job
":ID,"Msg")
before = rec
ColNames = prototype
ColCount = DCount(ColNames,@VM)
recColCount = Dcount(rec, @VM)
For i=1 to ColCount
ColName = ColNames
Locate ColName IN rec SETTING PTR Then
For j=15 to 20
rec = prototype
Next j
If ColName = "JobName" Then rec = ID
End
End Else
recColCount += 1
rec = field(rec,"/",1):"/":recColCount
For j=14 to 20
rec = prototype
Next j
If ColName = "JobName" Then rec = ID
End
Next i
Write rec on F.FILE,"ROOT"
End Else
Call DSLogInfo("Job ":ID:" does not have a ROOT record","Msg")
Release F.FILE,"ROOT"
End
End
End
End
Repeat
End Else
Call DSLogInfo("Unable to read ":PrototypeJob:" record
ROOT","Msg")
End
End Else
Call DSLogInfo("Unable to open ":PrototypeJob:" job file
":PROTOFILE,"Msg")
End
End Else
Call DSLogInfo("Unable to read ":PrototypeJob:" job record in DS_JOBS","Msg")
End
End Else
Call DSLogInfo("Unable to open DS_JOBS file","Msg")
End
End Else
Call DSLogInfo("No processing, update not set to Y","Msg")
End
---------------------------------------------------------------------------------------------------------------------------------------
If youre interested in having this utility and a mass-recompile utility job, set me an email and Ill reply with a .dsx file attached. I also have the job control logic for setting job parameters at runtime, doing all these bells and whistles. Its fully documented so you should have no trouble setting it up and using it.
Thanks,
-Ken
david@barham.hm on 30-Jul-2001 19:25
Please respond to datastage-users@oliver.com
To: datastage-users
cc:
Subject: RE: Process for Migration
Patricia,
This part of our system is not very sophisticated. We simply use the DataStage export/import. However, we do have some practices that might answer you questions.
We make all our changes to jobs in dev. We then migrate the jobs to test and prod (and yes, it is VERY tedious).
As a rule, we do NOT make any changes to jobs in test or prod. We do leave the jobs in dev and test. We also keep a copy (history) of the export files so that we can roll back to a previous version should there be a major problem with a release.
We do not change OCI DSNs, Userids or passwords when migrating to prod. These are all parameters. The defaults are always set to the dev oracle database, except perhaps for some top-level control jobs. As these parameters are passed down from a calling job to a called job, we only need to set the parameters correctly once in the top-level job when scheduling it.
Anything that needs to be different between dev and prod (including file
paths) is generally driven by parameters (we do use a little slight-of-hand
here) so that there is never any need to change a job when migrating it from dev to prod. If you had to change the job, that would defeat the purpose of all the testing, and seriously compromise the integrity of the release process.
Hope this helps.
David Barham
Information Technology Consultant
InformAtect Pty Ltd
Brisbane, Australia
e-mail: david@barham.hm
-----Original Message-----
From: Desiano, Patricia [mailto:Patricia.Desiano@anheuser-busch.com]
Sent: Tuesday, 31 July 2001 7:25 AM
To: datastage-users-list
Subject: Process for Migration
We are using DCS product to migrate our data stage jobs.
Since this is our first attempt to move items to production, we were wondering what migration practices other Data Stage users had in place.
If using DCS, what do you do with the Jobs that were migrated up from DEV
-
to
TEST - to PROD...
(tedious, since you can only be in Dev to make any changes, i.e. ..OCI Prod ids, pswds etc...before migrating up Test, then to Prod) Do you leave them in those Projects(Dev & Test...?), Do you delete them out after migrated to PROD ?
Any simple incites would be appreciated.
Thank You,
Patricia Desiano
Anheuser-Busch, Inc.
MSG - BudNET Data Warehouse
314-577-7558
Ricks got a good point about exposing passwords in a text file, but UNIX and NT permissions do allow an easy method for controlling access to specific files. Setting up a parameter as type ENCRYPTED will keep DataStage from exposing a password. A sequential parameter file is EASILY source-code controlled in any software (Razor, PVCS, SCCS), so you can track changes to this file. DataStage has always exposed passwords in the UNIX ps table, you can mitigate this by positioning passwords as the last parameters set in the job control logic. As for third-party software in beta and a non-shipping release of DataStage, you have to make choices now. I recommend NOT hard-coding parameters and instead set the appropriate values at runtime either from a parameter file or parameter table.
Good luck!
Kenneth Bland
(Embedded image moved to file: pic05565.pcx)
Principal Consultant
Kenneth.Bland@AscentialSoftware.com
www.AscentialSoftware.com
rick@schirms.net on 31-Jul-2001 13:44
Please respond to datastage-users@oliver.com
To: datastage-users
cc:
Subject: RE: Process for Migration
While Kens suggestions are very viable there is a concern with doing the first option. First in most sites that I have been at the mention of storing passwords in a flat file even with the permissions set tight is frowned upon. Then you have to worry about DS having the correct permissions to read the file. If you are in an environment that this type of security is not an issue, then it will work fine. But if your location has a strict auditing requirement it may not pass the test. Secondly this information does not get retained with the runtime metadata of the job. When you run a job that has parameters in it there is an entry in the log that gets created that displays the parameter that the job is currently running with. If you are setting these values after the job starts then the information in the Log is not valid.
The second option mentioned will also work until Kobe (Release 5.0) where they have changed the structure of the job storage so the routines used to update these values would have to be changed.
We are developing an application (Parameter Management for DataStage) that allows you to manage the parameters used from a global aspect and update the parameters based on the server, project, and job that the parameter resides in. What this does is updates both the job design defaults as well as the runtime defaults for each job which eliminates the requirements for a recompile. The application is communicating with DataStage using the dll functions from DataStage. This means that from one release of DataStage to the next the underlying changes to DataStage do not effect the application.
This product is nearing the Beta stage which Patricia from AB will be part of that Beta.
This interface allows you to analyze a project to identify all the parameters in all of the jobs and determine what the default values should be based on the server/project that they are running in.
Another little security issue to keep in mind that anytime the DSRunjob is performed where there are parameters being passed those parameter values are visible to the operating system. So by making sure the default runtime values are updated and correct means that the job can be ran without parameters specified and they will not be exposed.
This product will be further integrated with Version Control for DataStage that we developed so that when using this to mi-grate the jobs to various projects the default parameter values will be updated at the same time. This will be a future enhancement.
We do not have any literature to speak of at this time for Parameter Management (first comes the product then the lit) but if anyone has any specific questions you can email me at the address below. ______________________________
Rick Schirm
VP Operations
2201 Harwood Rd.
Bedford, TX 76021
mailto:rick.schirm@performart.com
http://www.performart.com
Phone 817-318-6278
______________________________
______________________________
Good luck!
Kenneth Bland
(Embedded image moved to file: pic05565.pcx)
Principal Consultant
Kenneth.Bland@AscentialSoftware.com
www.AscentialSoftware.com
rick@schirms.net on 31-Jul-2001 13:44
Please respond to datastage-users@oliver.com
To: datastage-users
cc:
Subject: RE: Process for Migration
While Kens suggestions are very viable there is a concern with doing the first option. First in most sites that I have been at the mention of storing passwords in a flat file even with the permissions set tight is frowned upon. Then you have to worry about DS having the correct permissions to read the file. If you are in an environment that this type of security is not an issue, then it will work fine. But if your location has a strict auditing requirement it may not pass the test. Secondly this information does not get retained with the runtime metadata of the job. When you run a job that has parameters in it there is an entry in the log that gets created that displays the parameter that the job is currently running with. If you are setting these values after the job starts then the information in the Log is not valid.
The second option mentioned will also work until Kobe (Release 5.0) where they have changed the structure of the job storage so the routines used to update these values would have to be changed.
We are developing an application (Parameter Management for DataStage) that allows you to manage the parameters used from a global aspect and update the parameters based on the server, project, and job that the parameter resides in. What this does is updates both the job design defaults as well as the runtime defaults for each job which eliminates the requirements for a recompile. The application is communicating with DataStage using the dll functions from DataStage. This means that from one release of DataStage to the next the underlying changes to DataStage do not effect the application.
This product is nearing the Beta stage which Patricia from AB will be part of that Beta.
This interface allows you to analyze a project to identify all the parameters in all of the jobs and determine what the default values should be based on the server/project that they are running in.
Another little security issue to keep in mind that anytime the DSRunjob is performed where there are parameters being passed those parameter values are visible to the operating system. So by making sure the default runtime values are updated and correct means that the job can be ran without parameters specified and they will not be exposed.
This product will be further integrated with Version Control for DataStage that we developed so that when using this to mi-grate the jobs to various projects the default parameter values will be updated at the same time. This will be a future enhancement.
We do not have any literature to speak of at this time for Parameter Management (first comes the product then the lit) but if anyone has any specific questions you can email me at the address below. ______________________________
Rick Schirm
VP Operations
2201 Harwood Rd.
Bedford, TX 76021
mailto:rick.schirm@performart.com
http://www.performart.com
Phone 817-318-6278
______________________________
______________________________