Can someone help me in understanging from which environments are those variables read into the DataStage process (it's login, the dsenv or somewhere else ?)
ArieAR
Using Environment Variables
Moderators: chulett, rschirm, roy
ariear
I think when you go outside of DataStage to solve a problem then you make your jobs overly complicated. Most of the time this is not needed. You can access environment variables created in dsenv or in other places but do you need to. This means as part of admin you need to maintain dsenv when you upgrade and do backups or whatever. If you include a shell script as part of your ETL then it makes it more complicated to see what is going wrong when soemthing breaks. I have seen developers shell out, run sql plus then get the result instead of doing a lookup. How much more complicated is that? Was it necessary? Did you gain performance? Stay within the tool. DataStage works great. You may gain 30 seconds performance and cost some developer 3 hours trying to figure out what you did.
One customer spent months wrapping Java around DataStage processes to see when or if a job finishes. The Java process would run in the backgraound just like the Director to see if the process would finish. It would page someone when it did not finish. These people waisted lots of time and money duplicating DataStage functionality.
If you need some external value then use environment variables. Try to keep them in dsenv. Limit how often you do this. Store temparary results in sequential files or hash files. Do not go ask a UNIX process to execute once per row when you could do it once per job. Keep it simple.
Sorry about the rant. That is my 2 cents worth.
Thanks Kim.
Kim Duke
DwNav - ETL Navigator
www.Duke-Consulting.com
I think when you go outside of DataStage to solve a problem then you make your jobs overly complicated. Most of the time this is not needed. You can access environment variables created in dsenv or in other places but do you need to. This means as part of admin you need to maintain dsenv when you upgrade and do backups or whatever. If you include a shell script as part of your ETL then it makes it more complicated to see what is going wrong when soemthing breaks. I have seen developers shell out, run sql plus then get the result instead of doing a lookup. How much more complicated is that? Was it necessary? Did you gain performance? Stay within the tool. DataStage works great. You may gain 30 seconds performance and cost some developer 3 hours trying to figure out what you did.
One customer spent months wrapping Java around DataStage processes to see when or if a job finishes. The Java process would run in the backgraound just like the Director to see if the process would finish. It would page someone when it did not finish. These people waisted lots of time and money duplicating DataStage functionality.
If you need some external value then use environment variables. Try to keep them in dsenv. Limit how often you do this. Store temparary results in sequential files or hash files. Do not go ask a UNIX process to execute once per row when you could do it once per job. Keep it simple.
Sorry about the rant. That is my 2 cents worth.
Thanks Kim.
Kim Duke
DwNav - ETL Navigator
www.Duke-Consulting.com
-
vmcburney
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
The .dsenv file will always be used sourced by DataStage jobs, which is why the installation notes recommend you put your additional DataStage related settings in there. Some login variables are also set but the .dsenv is run last and can overwrite login values.
Vincent McBurney
Data Integration Services
www.intramatix.com
Vincent McBurney
Data Integration Services
www.intramatix.com
-
vmcburney
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
It's an old post but I found the answer in the DataStage Administrator manual for version 6.0:
"DataStage derives the environment in which to run a job as follows:
1. All server process, job processes, and client connection helper
processes are ultimately child processes of dsrpcd and inherit its environment settings. The environment for dsrpcd is derived from the
/etc/profile and dshome/dsenv scripts. (Note that client connections DO NOT pick up any per-user environment settings from their
$HOME/.profile script)."
Vincent McBurney
Data Integration Services
www.intramatix.com
"DataStage derives the environment in which to run a job as follows:
1. All server process, job processes, and client connection helper
processes are ultimately child processes of dsrpcd and inherit its environment settings. The environment for dsrpcd is derived from the
/etc/profile and dshome/dsenv scripts. (Note that client connections DO NOT pick up any per-user environment settings from their
$HOME/.profile script)."
Vincent McBurney
Data Integration Services
www.intramatix.com