Creating folders dynamically on Unix via DataStage

[Nic]

I need to create directories on a Solaris box dynamically based on the values of a field. The number of values and the values themselves change every run (they are 9 digit codes). Once these are created I need to place the relevant XML files into the relevant folders.
I have managed to achieve this latter part by adding "%@" to the path name in the XML writer, which recognises the correct path if it exists.
However I have been unable to create the folders using DS, so far I have been creating them manually using the mkdir command directly in unix.
I am trying to figure out if I can use the ExecSH command but I don't know how to write a command that would make directories by reading a field from a table.
Any suggestions would be greatly appreciated.
Thanks

[kcbland]

Possible solutions:

1. Write a job to spool output to a text file. Write a ksh to parse and create the appropriate directories.
2. Write a job to spool output to a text file. Write DS Batch job to read the text file, parse it, and issue Unix commands via calls to DSExecute to create the approriate directories.
3. Write a DS Batch job to execute a command line database (isql, dbaccess, sqplus, etc) query to spool the results, parse it, and issue Unix commands via calls to DSExecute to create the approriate directories.

[rleishman]

4. Write a job that transforms the directory names one per row to a dummy Sequential File stage (the file name is not important - you won't even use it), and place a filter on the Sequential Stage of

Code: Select all

xargs mkdir -p

[Nic]

Thank you so much for this, I have been struggling with it for some time, but your solution is simple and very effective!
Thanks very much again!
Nic

4. Write a job that transforms the directory names one per row to a dummy Sequential File stage (the file name is not important - you won't even use it), and place a filter on the Sequential Stage of

Code: Select all

xargs mkdir -p

[rleishman]

Nic,

Do me a favour and create an Annotation in your Job that describes what you are doing, and repeat it in the stage notes for the SEQ file stage. The solution is really a mis-use of the tool, so this might help the next guy who looks at it.

[Nic]

I will add the notes and description now, however I am still struggling with it as I have found it works perfectly but creates the directories in the default folder but they should be created somewhere else and I don't seem to be able to direct them elsewhere. Any suggestions with this please?
Thanks very much for your help, it is much appreciated!

Nic,

Do me a favour and create an Annotation in your Job that describes what you are doing, and repeat it in the stage notes for the SEQ file stage. The solution is really a mis-use of the tool, so this might help the next guy who looks at it.

[rleishman]

Try a filter command of:

(cd /path/path; xargs mkdir -p)

[Nic]

Unfortunately it didn't work. I tried it with a pipe too, tried it as a Before/After Job subroutine too. I am not giving up though so I'll keep trying until I resolve this.
This is the error I got just FYI:

CPHLookupCensusforParts..CPHOnly.LnkToCPHOnly: ds_seqopen() - Error in filter command "/usr/bin/cd /wfa1/DataStage/Projects/WFAPilot/Data ; xargs mkdir -p" -
/usr/bin/cd[8]: cd: bad argument count

Thanks for your help.

Nic

Try a filter command of:

(cd /path/path; xargs mkdir -p)

[rleishman]

Hmmm. Did you put the path name /usr/bin in front of the 'cd'? cd is a shell builtin, not an executable file or script in /usr/bin. Check in /usr/bin, is there actually a file there called 'cd'? Try running the command you pasted above from the Unix command line and see what happens - it certainly won't change your current working directory.

If you tire of trying to solve what's happening with 'cd', try the following in the filter command:

Code: Select all

sed 's/^/#$PATHVAR#/' | xargs mkdir -p

$PATHVAR is an Environment variable containing the name of the path - it must have a trailing '/'. eg. /home/rleishman/.
If you are hard-coding the path , you will have to escape each of the slashes:

Code: Select all

sed 's/^/\/home\/rleishman\//' | xargs mkdir -p

[Nic]

Finally figured it out (didn't take this long - I took a break from it )!
No, I did not put in /usr/bin - it was DataStage that put that into the error message it gave.
Anyway, I created a flat file with all the folder names in Stage 1 where the field (which is to be the folder names) has to be concatenated with the hardcoded path as it is being created. The next stage uses a Before Job Subroutine to read that file with cat file| xargs mkdir -p
This seems to be working fine creating the files in a specific location.
Thanks for all your help again, I learnt a lot about xargs.

Hmmm. Did you put the path name /usr/bin in front of the 'cd'? cd is a shell builtin, not an executable file or script in /usr/bin. Check in /usr/bin, is there actually a file there called 'cd'? Try running the command you pasted above from the Unix command line and see what happens - it certainly won't change your current working directory.

If you tire of trying to solve what's happening with 'cd', try the following in the filter command:

Code: Select all

sed 's/^/#$PATHVAR#/' | xargs mkdir -p

$PATHVAR is an Environment variable containing the name of the path - it must have a trailing '/'. eg. /home/rleishman/.
If you are hard-coding the path , you will have to escape each of the slashes:

Code: Select all

sed 's/^/\/home\/rleishman\//' | xargs mkdir -p

[kduke]

This was posted in my user group:

Code: Select all

* ------------------------------------------------------------ 
* KgdMakeDir(DirName) 
* Decription: This routine will make all the dirs in DirName 
* Written by: Kim Duke 
* Notes: 
* ------------------------------------------------------------ 
* $INCLUDE DSINCLUDE JOBCONTROL.H 
      Ans = @FALSE 
      If System(91) = 0 Then 
         Shell = "UNIX" 
         Sep = '/' 
         OtherSep = '\' 
      End Else 
         Shell = "DOS" 
         Sep = '\' 
         OtherSep = '/' 
      End 
      convert OtherSep to Sep in DirName 
      print "Shell = ":Shell 
      print "Sep = ":Sep 
* ------------------------------------------------------------ 
      NoOfDirs = dcount(DirName, Sep) 
      for i=1 to NoOfDirs 
         tmpDir = field(DirName, Sep, 1, i) 
         OpenPath tmpDir To FileVariable On Error 
            cmd = "mkdir ":tmpDir 
            print cmd 
* ------------------------------------------------------------ 
            Call DSExecute(Shell, cmd, output, ErrorCode) 
            If ErrorCode > 0 Then 
               Ans = @FALSE 
               print "ErrorCode = ":ErrorCode 
               goto TheEnd 
            End Else 
               Ans = @TRUE 
            End 
         End Then 
            Ans = @TRUE 
            Close FileVariable 
         End Else 
            cmd = "mkdir ":tmpDir 
            print cmd 
* ------------------------------------------------------------ 
            Call DSExecute(Shell, cmd, output, ErrorCode) 
            If ErrorCode > 0 Then 
               Ans = @FALSE 
               print "ErrorCode = ":ErrorCode 
               goto TheEnd 
            End Else 
               Ans = @TRUE 
            End 
         End 
      next i 
* ------------------------------------------------------------ 
TheEnd: 
      print "Ans = ":Ans