Using of '~' (Unix Home directory) instead of hardcoding

seell · Post by **seell** » Fri Nov 14, 2003 11:09 pm

Can I use the Unix home directory ('~') instead of hardcoding the specific directory in the job parameter ?

If yes, how could this be done ?

Thank you
SLL

chulett · Post by **chulett** » Sat Nov 15, 2003 12:06 am

Of course. Just use it in pathname parameters in the same manner you would in any other Unix application. By itself it would refer to the home directory of the current user running the job or use it with a specific user name ("~dsadm/scripts" for example, instead of "~/scripts") to indicate that particular users home directory.

Does that help?

scboyce · Post by **scboyce** » Sun Nov 16, 2003 7:13 am

Another alternative is to use the $HOME environment variable which is standard variable created at login time under most UNIX environments.

seell · Post by **seell** » Sun Nov 16, 2003 6:48 pm

hi Craig

This is one of the parameters of a DataStage job where I would prompt the user for Userid/password/path/

I have tested by placing the "~" in the job parameter but the job failed validataion : the home directory concerned is /u01/ascential/
Failed to open DimRace.csv in directory ~/DataStage/Projects/Data/dsdvstud
STATUS = -1. or Failed to open "DimRace.csv in directory "~/DataStage/Projects/Data/dsdvstud
STATUS = -1.
----
I have also tested using $HOME; Same error

Failed to open "DimRace.csv in directory "~/DataStage/Projects/Data/dsdvstud
STATUS = -1.

Thank you for all your help.
Regards
SLL

kcbland · Post by **kcbland** » Sun Nov 16, 2003 8:51 pm

DataStage jobs run under the user and environment of the engine daemon as evidenced by "/u01/ascential/" as the home, not the user invoking the job. You cannot have the path go to the user starting the job without including the userid in the path as Craig has described.

This also means that the engine would have to have write permissions to this user's home. In addition, most people never configure home file systems to be large, so I'd guess you may run into capacity issues at some point.

Usage of a "~" is highly suspect, and because a job parameter is variable and can be programmatically set at runtime via job control, you should take the choice of a fully qualified path. IMO.

seell · Post by **seell** » Sun Nov 16, 2003 8:55 pm

hi
Are you saying that I need to include the userid dsadm which is the userid I am using anyway to run the job ?

I tried to put in the userid as well :

: DSD.SEQOpen Failed to open DimCtrytest.csv in directory ~dsadm/DataStage/Projects/Data/dsdvstud
STATUS = -1.

Regards
SLL

kcbland · Post by **kcbland** » Sun Nov 16, 2003 9:02 pm

First of all, DataStage will not create a path if it doesn't exist.

Second, usage of "~" in a path means put the user home directory in place of the "~". So, it's basically a token.

Do some experimentation of the environment. Create a BATCH job that executes an "env" statement:

Code: Select all

      UNIXcmd = "env"
      Call DSExecute("UNIX", UNIXcmd, ScreenOutput, SystemReturnCode)
      Call DSLogInfo("Environment: ":@AM:ScreenOutput, "Msg")

      UNIXcmd = "ulimit -a"
      Call DSExecute("UNIX", UNIXcmd, ScreenOutput, SystemReturnCode)
      Call DSLogInfo("Ulimit: ":@AM:ScreenOutput, "Msg")

      UNIXcmd = "umask"
      Call DSExecute("UNIX", UNIXcmd, ScreenOutput, SystemReturnCode)
      Call DSLogInfo("Umask: ":@AM:ScreenOutput, "Msg")

Look at the log file and see the CDW and the PWD. Also see the HOME directory. If you want to see where a job is looking at when you will use "~", then do this:

Code: Select all

      UNIXcmd = "ls -l ~dsadm"
      Call DSExecute("UNIX", UNIXcmd, ScreenOutput, SystemReturnCode)
      Call DSLogInfo("Tilde test: ":@AM:ScreenOutput, "Msg")

seell · Post by **seell** » Sun Nov 16, 2003 9:40 pm

I have found that :

PWD=/u01/ascential/DataStage/DSEngine
HOME=/u01/ascential

Tilde test: ~dsadm: No such file or directory

What the user wanted is to avoid hardcoding in the job paramters and the path required is /u01/ascential/DataStage/Projects/Data/dsdvstud/

so the more appropriate would be the HOME variable ???

Thank you for your patience
Regards

chulett · Post by **chulett** » Mon Nov 17, 2003 8:14 am

Something is wrong if "~dsadm" resolves to "No such file or directory"... or are you saying that that token used in your filename path resolves to that? If you are confident that the complete (absolute) resolved path is correct then you may have a permissions issue. I'd suggest testing this path from outside of DataStage at the command line for verification.

Your other results are... interesting as well, depending on exactly how you got them. I'll have to check, but I could have sworn "pwd" would have returned the project your job was running in as relative paths used in a job should be relative to the current project.

Keep in mind that putting absolute paths in job parameters is not "hard coding" as they are easily changed if the location needs to be changed. That's the whole point of using parameters instead of truly hard-coding path or filenames. Now, if you mean they should be relative paths instead of absolute paths, then something more consistant that the current users home directory should be used in my opinion. Ken pointed this out as well... can you guarentee that these jobs will always be run by the same user? If your path is going to be relative to where DataStage is installed (which is a whole 'nother topic of conversation as that is not the best place to be staging data), then one suggestion would be to use the DataStage "home" $DSHOME (or perhaps $dshome, check which one is appropriate on your system) even if you do have to back up one in the process. So perhaps?

$dshome/../Projects/Data/dsdvstud/

Something else to consider would be to use the Administrator to establish a new environmental parameter that points directly to your staging area and then use that in your jobs.

I'm sure Ken will come back with more thoughts as well, once he wakes up from his New Baby Nap.

kduke · Post by **kduke** » Mon Nov 17, 2003 10:13 am

Craig

~dsadm is a Korn shell concept. Some UNIX machines like Sun have totally replaced sh with ksh and ~user will always works. Old style UNIX machines do not understand ~user. Also a DataStage job does not login therefore does not have a default shell like ksh. If a process does not have a default shell then $HOME may not be defined either. The csh can also mess up these types of variables.

The dsenv file can solve some of these problems. Why not have SEQDIR defined there?

Kim.

kcbland · Post by **kcbland** » Mon Nov 17, 2003 10:32 am

Kim probably hit on the problem. Since dsadm doesn't have to be a real user, sometimes the HOME directory isn't defined.

I want to repeat again, that the user invoking a job does not influence the environment of a job one whit. So, if someone designs adhoc jobs that need to write to the location specific to the person invoking the job from Director, you have to step back and design an infrastructure to support this. Putting things in dsenv is usually a bad choice, because it is systemic for all projects and jobs.

If you want the most elegant solution, I have two very similar approaches. Create a configuration file of paths and variables that you want your jobs to use. Use ksh appropriate syntax so that shell scripts can source the file into the environment. You can have a single job control interface to your jobstream either read the parameter file, or source it into the job environment at runtime. Now, instead of dealing with paths, you have a source code-ish file (perfect for version control software I might add) that means jobs/scripts/etc can use to interact. If you need a job stream to run one time with a different set of paths from the next time, this is easily managed.

chulett · Post by **chulett** » Mon Nov 17, 2003 10:45 am

Ahh... sorry. I've been living in the Korn for so long, it seems I've forgotten that other shells exist.

Not to start another Holy War or anything, but while I agree with Ken that the most elegant solution is to build something akin to his 'configuration file' of paths and variables and a system to leverage that... it may not always be feasible given resource, expertise and/or time constraints.

Instead of using dsenv to define environment variables, wouldn't the "new" approach of defining them via the Administrator be "better"?

kcbland · Post by **kcbland** » Mon Nov 17, 2003 10:52 am

It's great having to type in parameters via Director at runtime. Screw up one directory path and you get to do it again, after all the cards fall.

Why not have a single parameter in a controlling process, that gives all of the information the jobstream will require? Where the information originates, table or file, doesn't matter. Where it ends up, file or environment doesn't matter. It's a fairly simple concept. When I see customers manually invoking jobs, typing in parameters, introducing user error everywhere, I just want to take up another line of work. Isn't it all supposed to be better than that???

Find a solution, just stop keying things in by hand every run.

chulett · Post by **chulett** » Mon Nov 17, 2003 11:02 am

Hmm... not really sure how we made the leap from not using a parameter file to running jobs manually and keying in parameters by hand every time. Certainly wasn't advocating that.

kcbland · Post by **kcbland** » Mon Nov 17, 2003 11:08 am

Not you Craig...the original poster. Sorry, sleep deprivation. 6 day old baby keeps you up aaaall night.

seell wrote: What the user wanted is to avoid hardcoding in the job paramters and the path required is /u01/ascential/DataStage/Projects/Data/dsdvstud/