Creating directories

[PhilHibbs]

How can I get my DataStage job to create a directory? Do I have to do this in a Before Job script? I want to get rid of my external scripts that pre-create all the directories used in my jobs, and get every job to create the directories it needs for the files it creates. Any suggestions?

[ray.wurlod]

You can do it with a simple before-job command, but a script (or routine) would give you better control (for example checking whether the directory already exists).

For the simple method just use a set of operating system commands in the Input Values field for ExecSH as the before-job subroutine, for example

Code: Select all

mkdir #DirPath1# ; mkdir #DirPath2# ; mkdir #DirPath3#

[chulett]

Right, aren't you going to need a script for this? Unless the directories never exist, you'll need to check first and only create if they don't, otherwise you'll generate errors.

[PhilHibbs]

Right, aren't you going to need a script for this? Unless the directories never exist, you'll need to check first and only create if they don't, otherwise you'll generate errors.

You can avoid any errors with -p

[chulett]

Hmmm... thought that just created intermediate directories in the path but you're correct - it also suppresses errors if the directories already exist. Learned something new today!

[PhilHibbs]

I've been thinking about doing something frightfully clever in this area for some time.

I already have two tools that scan a dsx file and build up a list of input and output files (inserting the default parameter values if necessary - we typically don't override parameter values, they are just a convenient place to put constant values). One of these tools also generates "mkdir" statements to create all of the directories that are referenced by any sequential or hashed file stages.

I could use this data to feed back into the job design, stuffing the results of the scan back into the dsx file in the Before Job area. That would require periodic re-runs of the scan/stuff/import process.

Alternatively, I could load the file onto the server and use it as the driver of the process, so every job would have a Before Job routine that opens up the file, searches for its own job name, and creates the directories that it finds in the file. It could go further than this and check that all of the input files are present, but I don't know if there's anything that it could do about it at that stage - is it feasible for a Before Job routine to deliberately abort the job? Or, for it to wait for one or more files to be created? I'll take a look at DSWaitForFile.

[shaimil]

Sounds like you got yourself a couple of useful scripts there.

The only thing I would be wary of when letting the jobs create their own directories is ensuring that the job has not mis-spelt the directory names, but easy to bug fix.

[ray.wurlod]

is it feasible for a Before Job routine to deliberately abort the job?

That's precisely what the ErrorCode argument is for.