Using Environment Variables

[danjm]

Hi Folks,
I'm trying to use environment variables with DataStage and I'm having a little difficulty that I hope someone can help me with.
Background: I have a 'test' server job that simply copies one sequential file to another using a transformer stage. I want to use two environment variables to represent the path for the source file and the target file. The idea is that when I move the job to another server (IE: development to Acceptance Test) it will use the environment variable definitions on the new server to access and write to different locations.
What I've done:
1. Used DS Administrator to create environment variables in the project.
2. Created the simple job in that project using parameters
3. Compiled and run the job
4. Used DS Administrator to modify the environment variables in the project.
5. Re-compile and run the job
Results:
1. Job compiles and runs fine first time.
2. Job compiles and runs fine after changing the environment variables but is still using the old values.
QUESTION:: How do I make the job 'reload' the current values of the environment variables dynamically at run time??
Any ideas are most welcome, thanks...
If you need more info to help, please let me know and I'll email you whatever you need.

Merry Christmas to everyone... looking forward to a terrific New Year!!

[raju_chvr]

I have been waiting for Kenneth or Rau or Teej to answer this from the time you have posted. I guess they have already left for Christmas.

But they haven't till now. So, I know this may not solve your problem... but just one thought ... did u try restarting the server.

I know this may not be the solution ...


Merry Christmas Every one !!

[danjm]

Hey, I really hope that EVERYONE has 'taken off' for Xmas. I intend to myself in just a couple of hours. Thank you for your idea, but it doesn't affect the outcome. After changing the environment variables, I can access them in the job by dropping the old and re-importing the new definition. It's just the 'dyanmic' portion I'm after that's not working as I'd hoped it would (and I'm beginning to think maybe that this is not "doable" *pout* ). Anyway, thanks again. I won't be back online until next Monday so I don't want anyone to lose any sleep or Xmas party/family time over this one for sure. I'll see everyone next week. Take care, - Happy Holidays.

[chulett]

I have been waiting for Kenneth or Rau or Teej to answer this from the time you have posted.

If you guys would rather continue to wait...

What are you putting in for the value of the $Environment parameter when you run the job? There are a couple of options - a value entered at runtime will override the current value of the environment variable, or use $ENV to capture the current value and use it in the job. I suspect you may not be using the latter if you are recompiling the job after modifying the value in the Administrator.

AFAIK, you don't need to restart the server after working with Environment Variables in the Administrator. They are set dynamically when a job runs.

[danjm]

I'm using 2 environment variables, "Input_Data" and "Output_Data" created using the Administrator as "User Defined" environment variables.
At runtime I pass "$ENV" to both of the job variables that use 'them'. The message that I get from the Director is:
Project:Tutorial (EDM-ETL-4)
Job name:TestJobParams
Event #:395
Timestamp:12/23/2003 1:37:47 PM
Event type:Fatal
User:LAN\Dan.Marshall
Message:
TestJobParams: Missing Parameters $Input_Data [$ENV], $Output_Data[$ENV].
My understanding is that $ENV should cause the job parameter to pick up the current value of the environment variable???
I'm sure there is just some concept here that I'm "missing".. but it continues to elude me..

[chulett]

Doesn't sound like you are missing any of the concepts and you seem to be doing things right. Probably going to turn out to be something simple or a bug.

I just built a small test job and got the same message that you did about missing parameters. It doesn't seem to be setting the 'default' value as setup in the Administrator, and ends up blank in the job when I used $ENV. Even overriding it didn't seem to work as I got other error messages then. I'd report it as a bug to Ascential and see if they think it should be working or not as documented.

[kcbland]

I have been waiting for Kenneth or Rau or Teej to answer this from the time you have posted. I guess they have already left for Christmas.

No, we are always here. There's no big sign to tell you when we're lurking, other than to watch the front page to see who's on. If I'm available to answer questions, you will see my name on the home page. I work about 18 hours out of the day, half of it from home. So, whenever a post happens a little bell rings in my office and I can come see what's landed.

Anyway, the Inner Circle team sometimes lets others field questions just so there's perspective and discussion. Sometimes the question is best answered by pointing to previous answers, so I'll respond to those quickly. Othertimes, as in this post's case, I'll let the discussion go around a bit before I toss in my opinion. Don't want to stifle discussion. I'll contribute later...

[danjm]

Thank you Craig. And thank you Kenneth. ..and everyone else.
I will look at this again when I return to work on Monday and perhaps we can talk some more... Merry Xmas everyone. See you again soon. :D

[Teej]

The idea is that when I move the job to another server (IE: development to Acceptance Test) it will use the environment variable definitions on the new server to access and write to different locations.

Unfortunately, this is not possible with any version of DataStage. The concept of a Project-level environment parameters is to influence the engine behind DataStage (whether they are Server or Parallel) for each project. So if you want the engine to report back to you more frequently, then you use the parameters provided.

However, I believe it was Ascential's mindset that no matter where you want to run, you would prefers that your settings be set to YOUR value as defined in the job parameters. It is not the Project's responsibility to say, "Hey, the values are different here." What if you have something like $APT_NO_SHARED_MEM set to on, and the project you moved to have it set to off, which WILL break your job. Not good, huh? I know I mangled the name of that parameter, but the one I am thinking of DOES break certain jobs if used.

So in a nutshell, do not use project level parameters for anything that is company-based. Pathnames, usernames, passwords, et cetera, should never be defined there. Migrating between projects does not automatically update the parameters' values for a good reason.

I made a request to Ascential back in February to provide a Parameters folder, akin to Table Definition folder among others, to contain company-based parameters that gets automatically updated between projects (but can be overridden by request from the user.) If you agree with this idea, PLEASE FOR THE LOVE OF YOUR DIETY, submit a Crit P2 enhancement request asking for this. The more people that asks for it, the higher the priority it would be for Ascential to implement it. Maybe not 7.1, but 7.2 for sure if quite a few people make the same request.

Until that is implemented, rely on user-requested parameter values to save the day.

-T.J.

P.S. Yes, this is an issue very dear with me, especially after a LOT of 'debates' with my peers at work...

[kcbland]

Great suggestion Teej. Years ago I had submitted the suggestion that a new folder concept be introduced into DS Manager. This folder would be a "Parameter Block". Rather than define individual parameters per job, you would point to a block. Now, whenever you add more parameters to a job, they will dynamically appear in the job.

Continuing, you could then migrate parameter blocks, with or without default values, between environments. You could pre-configure the parameter block ahead of time in the right environment. This way, a global change could easily be instituted.

Since that went nowhere, I went with plan B which was an external parameter file. I also developed utilities to blast a prototype job parameter list from that job to all jobs of your choice. This meant all your jobs could have the same parameter set. My job control was enhanced to the point that at runtime you gave it the parameter file to use, and it would take care of setting values at runtime.

What this means is that jobs move from dev, to uat, to prod without any changes, recompilation, or default parameter blasting. You could setup different configurations for a given environment, depending on your needs.

There's a lot to be learned from how the product is actually deployed in the real world to meet "world class" objectives. There's a lot of "non-world class" data warehouses slapped together. Most large corporations run full SDLC projects, have security considerations, etc. User ids and passwords are a major issue, and is one of the subjects most companies have to tackle the day they want to start writing ETL. There's a lot of sad faces when you start laying out SDLC migration procedures and have to talk about how parameters are implemented.

[ray.wurlod]

Must be time to give Parameter Manager a plug.

While it doesn't directly address the OP's question, PM does provide for management of parameters between environments, in a sense offering the capability of having "global" parameters that can be loaded into jobs with different (project-wide) default values.

[gh_amitava]

Hi,,

Set those 2 variables in dsenv file.

Execute dsenv and restart the server.

Hope it will work.

Regards
Amitava Ghosh

Hi Folks,
I'm trying to use environment variables with DataStage and I'm having a little difficulty that I hope someone can help me with.
Background: I have a 'test' server job that simply copies one sequential file to another using a transformer stage. I want to use two environment variables to represent the path for the source file and the target file. The idea is that when I move the job to another server (IE: development to Acceptance Test) it will use the environment variable definitions on the new server to access and write to different locations.
What I've done:
1. Used DS Administrator to create environment variables in the project.
2. Created the simple job in that project using parameters
3. Compiled and run the job
4. Used DS Administrator to modify the environment variables in the project.
5. Re-compile and run the job
Results:
1. Job compiles and runs fine first time.
2. Job compiles and runs fine after changing the environment variables but is still using the old values.
QUESTION:: How do I make the job 'reload' the current values of the environment variables dynamically at run time??
Any ideas are most welcome, thanks...
If you need more info to help, please let me know and I'll email you whatever you need.

Merry Christmas to everyone... looking forward to a terrific New Year!!

[kcbland]

Hi,,

Set those 2 variables in dsenv file.

Execute dsenv and restart the server.

Hope it will work.

Regards
Amitava Ghosh

There are system wide environment variables that are put into dsenv, then there are project specific environment variables. The user wants project specific environment variables.

[Teej]

Set those 2 variables in dsenv file.

Execute dsenv and restart the server.

There are system wide environment variables that are put into dsenv, then there are project specific environment variables. The user wants project specific environment variables.

And the job-specific parameter using these values will NOT be updated between systems even with this 'trick', which is what the OP also wants.

-T.J.

P.S. Everyone, have a happy and SAFE holidays!

[danjm]

Thank you to everyone for your input on this matter. I very much appreciate the time and effort.
I'm understanding the 'bottom line' to be that DS does not use 'operating system environment variables' at all then (IE: it does not read and use their valuest at runtime) which leaves me to wonder what their purpose in the Administrator really is??? If I have to edit the job to use modified environment variables then there is no advantage over simply modifying string values in the job! I've still modified the job so I have no guarantee of tested repeatability at runtime. Once you edit ANYthing, you've possibly altered something else unintentionally.
I will give Ascential a call to be sure that I'm not missing/misunderstanding anything on this issue and I'll report back if they tell me anything different. Perhaps I'm stuck with using source and target variables as job parameters that are provided in the runtime command (which still allows an opportunity for error, although I suppose scripting and batching the commands can minimize this somewhat).
Anyway, thanks again to everyone for your time and input. Have a great day!!