Page 1 of 1
Creating folders dynamically on Unix via DataStage
Posted: Mon Sep 26, 2005 1:38 pm
by Nic
I need to create directories on a Solaris box dynamically based on the values of a field. The number of values and the values themselves change every run (they are 9 digit codes). Once these are created I need to place the relevant XML files into the relevant folders.
I have managed to achieve this latter part by adding "%@" to the path name in the XML writer, which recognises the correct path
if it exists.
However I have been unable to create the folders using DS, so far I have been creating them manually using the mkdir command directly in unix.
I am trying to figure out if I can use the ExecSH command but I don't know
how to write a command that would make directories by reading a field from a table.
Any suggestions would be greatly appreciated.
Thanks
![Wink :wink:](./images/smilies/icon_wink.gif)
Posted: Mon Sep 26, 2005 2:37 pm
by kcbland
Possible solutions:
1. Write a job to spool output to a text file. Write a ksh to parse and create the appropriate directories.
2. Write a job to spool output to a text file. Write DS Batch job to read the text file, parse it, and issue Unix commands via calls to DSExecute to create the approriate directories.
3. Write a DS Batch job to execute a command line database (isql, dbaccess, sqplus, etc) query to spool the results, parse it, and issue Unix commands via calls to DSExecute to create the approriate directories.
Posted: Mon Sep 26, 2005 4:22 pm
by rleishman
4. Write a job that transforms the directory names one per row to a dummy Sequential File stage (the file name is not important - you won't even use it), and place a filter on the Sequential Stage of
It works!
Posted: Tue Sep 27, 2005 1:02 pm
by Nic
Thank you so much for this, I have been struggling with it for some time, but your solution is simple and very effective!
Thanks very much again!
Nic
rleishman wrote:4. Write a job that transforms the directory names one per row to a dummy Sequential File stage (the file name is not important - you won't even use it), and place a filter on the Sequential Stage of
Posted: Tue Sep 27, 2005 5:18 pm
by rleishman
Nic,
Do me a favour and create an Annotation in your Job that describes what you are doing, and repeat it in the stage notes for the SEQ file stage. The solution is really a mis-use of the tool, so this might help the next guy who looks at it.
xargs
Posted: Wed Sep 28, 2005 11:02 am
by Nic
I will add the notes and description now, however I am still struggling with it as I have found it works perfectly but creates the directories in the default folder but they should be created somewhere else and I don't seem to be able to direct them elsewhere. Any suggestions with this please?
Thanks very much for your help, it is much appreciated!
rleishman wrote:Nic,
Do me a favour and create an Annotation in your Job that describes what you are doing, and repeat it in the stage notes for the SEQ file stage. The solution is really a mis-use of the tool, so this might help the next guy who looks at it.
Posted: Wed Sep 28, 2005 4:52 pm
by rleishman
Try a filter command of:
(cd /path/path; xargs mkdir -p)
Posted: Thu Sep 29, 2005 1:09 am
by Nic
Unfortunately it didn't work. I tried it with a pipe too, tried it as a Before/After Job subroutine too. I am not giving up though so I'll keep trying until I resolve this.
This is the error I got just FYI:
CPHLookupCensusforParts..CPHOnly.LnkToCPHOnly: ds_seqopen() - Error in filter command "/usr/bin/cd /wfa1/DataStage/Projects/WFAPilot/Data ; xargs mkdir -p" -
/usr/bin/cd[8]: cd: bad argument count
Thanks for your help.
Nic
rleishman wrote:Try a filter command of:
(cd /path/path; xargs mkdir -p)
Posted: Thu Sep 29, 2005 2:08 am
by rleishman
Hmmm. Did you put the path name /usr/bin in front of the 'cd'? cd is a shell builtin, not an executable file or script in /usr/bin. Check in /usr/bin, is there actually a file there called 'cd'? Try running the command you pasted above from the Unix command line and see what happens - it certainly won't change your current working directory.
If you tire of trying to solve what's happening with 'cd', try the following in the filter command:
Code: Select all
sed 's/^/#$PATHVAR#/' | xargs mkdir -p
$PATHVAR is an Environment variable containing the name of the path - it must have a trailing '/'. eg. /home/rleishman/.
If you are hard-coding the path
![Sad :(](./images/smilies/icon_sad.gif)
, you will have to escape each of the slashes:
Code: Select all
sed 's/^/\/home\/rleishman\//' | xargs mkdir -p
Figured it out
Posted: Tue Oct 04, 2005 8:53 am
by Nic
Finally figured it out (didn't take this long - I took a break from it
![Cool 8)](./images/smilies/icon_cool.gif)
)!
No, I did not put in /usr/bin - it was DataStage that put that into the error message it gave.
Anyway, I created a flat file with all the folder names in Stage 1 where the field (which is to be the folder names) has to be concatenated with the hardcoded path as it is being created. The next stage uses a Before Job Subroutine to read that file with cat file| xargs mkdir -p
This seems to be working fine creating the files in a specific location.
Thanks for all your help again, I learnt a lot about xargs.
rleishman wrote:Hmmm. Did you put the path name /usr/bin in front of the 'cd'? cd is a shell builtin, not an executable file or script in /usr/bin. Check in /usr/bin, is there actually a file there called 'cd'? Try running the command you pasted above from the Unix command line and see what happens - it certainly won't change your current working directory.
If you tire of trying to solve what's happening with 'cd', try the following in the filter command:
Code: Select all
sed 's/^/#$PATHVAR#/' | xargs mkdir -p
$PATHVAR is an Environment variable containing the name of the path - it must have a trailing '/'. eg. /home/rleishman/.
If you are hard-coding the path
![Sad :(](./images/smilies/icon_sad.gif)
, you will have to escape each of the slashes:
Code: Select all
sed 's/^/\/home\/rleishman\//' | xargs mkdir -p
Posted: Tue Oct 04, 2005 4:35 pm
by kduke
This was posted in my user group:
Code: Select all
* ------------------------------------------------------------
* KgdMakeDir(DirName)
* Decription: This routine will make all the dirs in DirName
* Written by: Kim Duke
* Notes:
* ------------------------------------------------------------
* $INCLUDE DSINCLUDE JOBCONTROL.H
Ans = @FALSE
If System(91) = 0 Then
Shell = "UNIX"
Sep = '/'
OtherSep = '\'
End Else
Shell = "DOS"
Sep = '\'
OtherSep = '/'
End
convert OtherSep to Sep in DirName
print "Shell = ":Shell
print "Sep = ":Sep
* ------------------------------------------------------------
NoOfDirs = dcount(DirName, Sep)
for i=1 to NoOfDirs
tmpDir = field(DirName, Sep, 1, i)
OpenPath tmpDir To FileVariable On Error
cmd = "mkdir ":tmpDir
print cmd
* ------------------------------------------------------------
Call DSExecute(Shell, cmd, output, ErrorCode)
If ErrorCode > 0 Then
Ans = @FALSE
print "ErrorCode = ":ErrorCode
goto TheEnd
End Else
Ans = @TRUE
End
End Then
Ans = @TRUE
Close FileVariable
End Else
cmd = "mkdir ":tmpDir
print cmd
* ------------------------------------------------------------
Call DSExecute(Shell, cmd, output, ErrorCode)
If ErrorCode > 0 Then
Ans = @FALSE
print "ErrorCode = ":ErrorCode
goto TheEnd
End Else
Ans = @TRUE
End
End
next i
* ------------------------------------------------------------
TheEnd:
print "Ans = ":Ans