Creating folders dynamically on Unix via DataStage
Moderators: chulett, rschirm, roy
Creating folders dynamically on Unix via DataStage
I need to create directories on a Solaris box dynamically based on the values of a field. The number of values and the values themselves change every run (they are 9 digit codes). Once these are created I need to place the relevant XML files into the relevant folders.
I have managed to achieve this latter part by adding "%@" to the path name in the XML writer, which recognises the correct path if it exists.
However I have been unable to create the folders using DS, so far I have been creating them manually using the mkdir command directly in unix.
I am trying to figure out if I can use the ExecSH command but I don't know how to write a command that would make directories by reading a field from a table.
Any suggestions would be greatly appreciated.
Thanks
I have managed to achieve this latter part by adding "%@" to the path name in the XML writer, which recognises the correct path if it exists.
However I have been unable to create the folders using DS, so far I have been creating them manually using the mkdir command directly in unix.
I am trying to figure out if I can use the ExecSH command but I don't know how to write a command that would make directories by reading a field from a table.
Any suggestions would be greatly appreciated.
Thanks
Possible solutions:
1. Write a job to spool output to a text file. Write a ksh to parse and create the appropriate directories.
2. Write a job to spool output to a text file. Write DS Batch job to read the text file, parse it, and issue Unix commands via calls to DSExecute to create the approriate directories.
3. Write a DS Batch job to execute a command line database (isql, dbaccess, sqplus, etc) query to spool the results, parse it, and issue Unix commands via calls to DSExecute to create the approriate directories.
1. Write a job to spool output to a text file. Write a ksh to parse and create the appropriate directories.
2. Write a job to spool output to a text file. Write DS Batch job to read the text file, parse it, and issue Unix commands via calls to DSExecute to create the approriate directories.
3. Write a DS Batch job to execute a command line database (isql, dbaccess, sqplus, etc) query to spool the results, parse it, and issue Unix commands via calls to DSExecute to create the approriate directories.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Premium Member
- Posts: 252
- Joined: Mon Sep 19, 2005 10:28 pm
- Location: Melbourne, Australia
- Contact:
4. Write a job that transforms the directory names one per row to a dummy Sequential File stage (the file name is not important - you won't even use it), and place a filter on the Sequential Stage of
Code: Select all
xargs mkdir -p
Ross Leishman
It works!
Thank you so much for this, I have been struggling with it for some time, but your solution is simple and very effective!
Thanks very much again!
Nic
Thanks very much again!
Nic
rleishman wrote:4. Write a job that transforms the directory names one per row to a dummy Sequential File stage (the file name is not important - you won't even use it), and place a filter on the Sequential Stage of
Code: Select all
xargs mkdir -p
xargs
I will add the notes and description now, however I am still struggling with it as I have found it works perfectly but creates the directories in the default folder but they should be created somewhere else and I don't seem to be able to direct them elsewhere. Any suggestions with this please?
Thanks very much for your help, it is much appreciated!
Thanks very much for your help, it is much appreciated!
rleishman wrote:Nic,
Do me a favour and create an Annotation in your Job that describes what you are doing, and repeat it in the stage notes for the SEQ file stage. The solution is really a mis-use of the tool, so this might help the next guy who looks at it.
Unfortunately it didn't work. I tried it with a pipe too, tried it as a Before/After Job subroutine too. I am not giving up though so I'll keep trying until I resolve this.
This is the error I got just FYI:
CPHLookupCensusforParts..CPHOnly.LnkToCPHOnly: ds_seqopen() - Error in filter command "/usr/bin/cd /wfa1/DataStage/Projects/WFAPilot/Data ; xargs mkdir -p" -
/usr/bin/cd[8]: cd: bad argument count
Thanks for your help.
Nic
This is the error I got just FYI:
CPHLookupCensusforParts..CPHOnly.LnkToCPHOnly: ds_seqopen() - Error in filter command "/usr/bin/cd /wfa1/DataStage/Projects/WFAPilot/Data ; xargs mkdir -p" -
/usr/bin/cd[8]: cd: bad argument count
Thanks for your help.
Nic
rleishman wrote:Try a filter command of:
(cd /path/path; xargs mkdir -p)
-
- Premium Member
- Posts: 252
- Joined: Mon Sep 19, 2005 10:28 pm
- Location: Melbourne, Australia
- Contact:
Hmmm. Did you put the path name /usr/bin in front of the 'cd'? cd is a shell builtin, not an executable file or script in /usr/bin. Check in /usr/bin, is there actually a file there called 'cd'? Try running the command you pasted above from the Unix command line and see what happens - it certainly won't change your current working directory.
If you tire of trying to solve what's happening with 'cd', try the following in the filter command:
$PATHVAR is an Environment variable containing the name of the path - it must have a trailing '/'. eg. /home/rleishman/.
If you are hard-coding the path , you will have to escape each of the slashes:
If you tire of trying to solve what's happening with 'cd', try the following in the filter command:
Code: Select all
sed 's/^/#$PATHVAR#/' | xargs mkdir -p
If you are hard-coding the path , you will have to escape each of the slashes:
Code: Select all
sed 's/^/\/home\/rleishman\//' | xargs mkdir -p
Ross Leishman
Figured it out
Finally figured it out (didn't take this long - I took a break from it )!
No, I did not put in /usr/bin - it was DataStage that put that into the error message it gave.
Anyway, I created a flat file with all the folder names in Stage 1 where the field (which is to be the folder names) has to be concatenated with the hardcoded path as it is being created. The next stage uses a Before Job Subroutine to read that file with cat file| xargs mkdir -p
This seems to be working fine creating the files in a specific location.
Thanks for all your help again, I learnt a lot about xargs.
No, I did not put in /usr/bin - it was DataStage that put that into the error message it gave.
Anyway, I created a flat file with all the folder names in Stage 1 where the field (which is to be the folder names) has to be concatenated with the hardcoded path as it is being created. The next stage uses a Before Job Subroutine to read that file with cat file| xargs mkdir -p
This seems to be working fine creating the files in a specific location.
Thanks for all your help again, I learnt a lot about xargs.
rleishman wrote:Hmmm. Did you put the path name /usr/bin in front of the 'cd'? cd is a shell builtin, not an executable file or script in /usr/bin. Check in /usr/bin, is there actually a file there called 'cd'? Try running the command you pasted above from the Unix command line and see what happens - it certainly won't change your current working directory.
If you tire of trying to solve what's happening with 'cd', try the following in the filter command:
$PATHVAR is an Environment variable containing the name of the path - it must have a trailing '/'. eg. /home/rleishman/.Code: Select all
sed 's/^/#$PATHVAR#/' | xargs mkdir -p
If you are hard-coding the path , you will have to escape each of the slashes:
Code: Select all
sed 's/^/\/home\/rleishman\//' | xargs mkdir -p
This was posted in my user group:
Code: Select all
* ------------------------------------------------------------
* KgdMakeDir(DirName)
* Decription: This routine will make all the dirs in DirName
* Written by: Kim Duke
* Notes:
* ------------------------------------------------------------
* $INCLUDE DSINCLUDE JOBCONTROL.H
Ans = @FALSE
If System(91) = 0 Then
Shell = "UNIX"
Sep = '/'
OtherSep = '\'
End Else
Shell = "DOS"
Sep = '\'
OtherSep = '/'
End
convert OtherSep to Sep in DirName
print "Shell = ":Shell
print "Sep = ":Sep
* ------------------------------------------------------------
NoOfDirs = dcount(DirName, Sep)
for i=1 to NoOfDirs
tmpDir = field(DirName, Sep, 1, i)
OpenPath tmpDir To FileVariable On Error
cmd = "mkdir ":tmpDir
print cmd
* ------------------------------------------------------------
Call DSExecute(Shell, cmd, output, ErrorCode)
If ErrorCode > 0 Then
Ans = @FALSE
print "ErrorCode = ":ErrorCode
goto TheEnd
End Else
Ans = @TRUE
End
End Then
Ans = @TRUE
Close FileVariable
End Else
cmd = "mkdir ":tmpDir
print cmd
* ------------------------------------------------------------
Call DSExecute(Shell, cmd, output, ErrorCode)
If ErrorCode > 0 Then
Ans = @FALSE
print "ErrorCode = ":ErrorCode
goto TheEnd
End Else
Ans = @TRUE
End
End
next i
* ------------------------------------------------------------
TheEnd:
print "Ans = ":Ans
Mamu Kim