I need a DSX-Cutter
Moderators: chulett, rschirm, roy
I don't know what the permissions mean, but if you're running this on a PC, I used Active Perl and it works fine. The unix side of me says your umask setting is wrong and the .pl script can't write the files to those directories.
It's a shame we're cluttering up the original post, next time you should start a thread instead of hijacking existing threads.
Anyway, my guess is now that you've opened up DUMP2 and below, does re-running the script work now? How about spooling to screen some messages within the script to figure out when/where/why your setup doesn't work?
It's a shame we're cluttering up the original post, next time you should start a thread instead of hijacking existing threads.
Anyway, my guess is now that you've opened up DUMP2 and below, does re-running the script work now? How about spooling to screen some messages within the script to figure out when/where/why your setup doesn't work?
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
First off.. sorry about the bad 'thread' etiquette.
Anyway.. yup.. it worked right in ActivePerl.
I'll debug the script and make sure it works in a POSIX env like Linux and Cygwin. I'm sure if everything was set right on my end.. it would work.. but the script should error out if the env is setup wrong.. which I'll fix and post back to this thread.
again. thanks!
Anyway.. yup.. it worked right in ActivePerl.
I'll debug the script and make sure it works in a POSIX env like Linux and Cygwin. I'm sure if everything was set right on my end.. it would work.. but the script should error out if the env is setup wrong.. which I'll fix and post back to this thread.
again. thanks!
If you are on Windows then why not use a routine:
Code: Select all
* -----------------------------------------------------------------
* KgdParseDSX(Path, FileName, JobsToSkip, CatOnlySw, DebugSw)
* Decription: Parse DSX File.
* Written by: Kim Duke
* -----------------------------------------------------------------
* Notes: ./KimD/Backups/20040809 Kim20040809.dsx
* -----------------------------------------------------------------
Ans = 'Done'
* -----------------------------------------------------------------
DEFFUN KgdMakeDir(DirName) CALLING "DSU.KgdMakeDir"
DEFFUN KgdGetSlash(NotUsed) CALLING "DSU.KgdGetSlash"
* -----------------------------------------------------------------
* initialize standard variables
* -----------------------------------------------------------------
* DebugSw = @false
* CatOnlySw = @true
JobName = ''
RoutineName = ''
Category = ''
Slash = KgdGetSlash('Na')
* -----------------------------------------------------------------
openpath Path to DirPtr else
ErrMsg = "Error: Bad Path = ":Path
goto ErrRtn
end
* -----------------------------------------------------------------
read AllDsxRec from DirPtr, FileName then
* -----------------------------------------------------------------
* create directories for jobs like FileName_Jobs
* -----------------------------------------------------------------
* OutDir1 = Path :Slash: ServerName :Slash: ProjectName
* if KgdMakeDir(OutDir1) then null
OutDir1 = Path :Slash: field(FileName, ".", 1) :"_"
JobDir = OutDir1 : 'Jobs'
if KgdMakeDir(JobDir) then null
RtnDir = OutDir1 : 'Routines'
if KgdMakeDir(RtnDir) then null
* -----------------------------------------------------------------
openpath JobDir to JobDirPtr else
ErrMsg = "Error: Bad JobDir = ":JobDir
goto ErrRtn
end
print 'Clearing ... ':JobDir
CLEARFILE JobDirPtr
openpath RtnDir to RtnDirPtr else
ErrMsg = "Error: Bad RtnDir = ":RtnDir
goto ErrRtn
end
print 'Clearing ... ':RtnDir
CLEARFILE RtnDirPtr
* -----------------------------------------------------------------
HeaderRec = ''
InHeader = @false
NewDsx = ''
InJob = @false
InRtn = @false
NoLines = dcount(AllDsxRec, @FM)
* -----------------------------------------------------------------
for j=1 to NoLines
Line = AllDsxRec<j>
if DebugSw then print Line
FirstWord = field(trim(Line), ' ', 1)
FirstUp = upcase(FirstWord)
SecondWord = field(trim(Line), ' ', 2)
SecondUp = upcase(SecondWord)
* -----------------------------------------------------------------
begin case
case FirstUp = 'BEGIN'
begin case
case SecondUp = 'HEADER'
InHeader = @true
HeaderRec = Line
case SecondUp = 'DSJOB'
InJob = @true
NewDsx = Line
JobName = field(AllDsxRec<j+1>, '"', 2)
print 'JobName =':JobName
case SecondUp = 'DSROUTINES'
InRtn = @true
NewDsx = ''
RoutineName = field(AllDsxRec<j+2>, '"', 2)
case SecondUp = 'DSRECORD' and InRtn
if NewDsx <> '' then
* -----------------------------------------------------------------
* write out routine (add DSROUTINES)
* -----------------------------------------------------------------
NewDsxRec = RtnHeaderRec
NewDsxRec := @FM : NewDsx
NewDsxRec := @FM : 'END DSROUTINES'
print 'Routine = ':RoutineName
write NewDsxRec on RtnDirPtr, RoutineName:'.dsx' else
ErrMsg = "Error: Unable to write to ":RtnDir
goto ErrRtn
end
if DebugSw then goto TheEnd
NewDsx = Line
RoutineName = field(AllDsxRec<j+1>, '"', 2)
end else
NewDsx = Line
end
case @true
NewDsx := @FM : Line
end case
case FirstUp = 'END'
begin case
case SecondUp = 'HEADER'
InHeader = @false
HeaderRec := @FM : Line
RtnHeaderRec = HeaderRec
RtnHeaderRec := @FM : 'BEGIN DSROUTINES'
* -----------------------------------------------------------------
* write out header
* -----------------------------------------------------------------
NewDsxRec = HeaderRec
write NewDsxRec on DirPtr, 'KgdHeader.dsx' else
ErrMsg = "Error: Unable to write to ":Path
goto ErrRtn
end
case SecondUp = 'DSJOB'
InJob = @false
NewDsx := @FM : Line
* -----------------------------------------------------------------
* write out job
* -----------------------------------------------------------------
NewDsxRec = HeaderRec
NewDsxRec := @FM : NewDsx
convert ':' to '_' in JobName
if not(CatOnlySw) and index(JobsToSkip,JobName[1,1],1)=0 then
print 'Job = ':JobName
write NewDsxRec on JobDirPtr, JobName:'.dsx' else
ErrMsg = "Error: Unable to write to ":JobDir
goto ErrRtn
end
end
CatId = Category :".dsx"
read CatRec from JobDirPtr, CatId else
CatRec = HeaderRec
end
CatRec := @FM : NewDsx
write CatRec on JobDirPtr, CatId else
ErrMsg = "Error: Unable to write to ":JobDir:', ':CatId
goto ErrRtn
end
if DebugSw then goto TheEnd
case SecondUp = 'DSROUTINES'
InRtn = @false
* NewDsx := @FM : Line
* -----------------------------------------------------------------
* write out routine
* -----------------------------------------------------------------
NewDsxRec = RtnHeaderRec
NewDsxRec := @FM : NewDsx
NewDsxRec := @FM : 'END DSROUTINES'
print 'Routine = ':RoutineName
write NewDsxRec on RtnDirPtr, RoutineName:'.dsx' else
ErrMsg = "Error: Unable to write to ":RtnDir
goto ErrRtn
end
if DebugSw then goto TheEnd
case @true
NewDsx := @FM : Line
end case
case @true
begin case
case InHeader
HeaderRec := @FM : Line
begin case
case FirstWord = 'ServerName'
ServerName = field(Line, '"', 2)
print 'ServerName =':ServerName
case FirstWord = 'ToolInstanceID'
ProjectName = field(Line, '"', 2)
print 'ProjectName =':ProjectName
case FirstWord = 'Date'
ExportDate = field(Line, '"', 2)
convert "-" to "" in ExportDate
print 'ExportDate =':ExportDate
end case
case @true
NewDsx := @FM : Line
if FirstWord = 'Category' then
Category = field(Line, '"', 2)
convert "\" to "_" in Category
print 'Category =':Category
end
end case
end case
* -----------------------------------------------------------------
next j
Ans = 'Double click me'
end else
ErrMsg = 'Error: Bad FileName = ':FileName
goto ErrRtn
end
goto TheEnd
* -----------------------------------------------------------------
ErrRtn:
print
print ErrMsg
print
* -----------------------------------------------------------------
TheEnd:
print Ans
Mamu Kim
Most of the SCM scripting I plan on doing will be under Linux, so it seems the Perl script would be my best bet..
However.. I'll keep your routine in my back pocket .. if anything I use it as a sanity check to the Perl script output.
However.. I'll keep your routine in my back pocket .. if anything I use it as a sanity check to the Perl script output.
kduke wrote:If you are on Windows then why not use a routine:
Code: Select all
* ----------------------------------------------------------------- * KgdParseDSX(Path, FileName, JobsToSkip, CatOnlySw, DebugSw) * Decription: Parse DSX File. * Written by: Kim Duke * ----------------------------------------------------------------- * Notes: ./KimD/Backups/20040809 Kim20040809.dsx * ----------------------------------------------------------------- Ans = 'Done' * ----------------------------------------------------------------- DEFFUN KgdMakeDir(DirName) CALLING "DSU.KgdMakeDir" DEFFUN KgdGetSlash(NotUsed) CALLING "DSU.KgdGetSlash" * ----------------------------------------------------------------- * initialize standard variables * ----------------------------------------------------------------- * DebugSw = @false * CatOnlySw = @true JobName = '' RoutineName = '' Category = '' Slash = KgdGetSlash('Na') * ----------------------------------------------------------------- openpath Path to DirPtr else ErrMsg = "Error: Bad Path = ":Path goto ErrRtn end * ----------------------------------------------------------------- read AllDsxRec from DirPtr, FileName then * ----------------------------------------------------------------- * create directories for jobs like FileName_Jobs * ----------------------------------------------------------------- * OutDir1 = Path :Slash: ServerName :Slash: ProjectName * if KgdMakeDir(OutDir1) then null OutDir1 = Path :Slash: field(FileName, ".", 1) :"_" JobDir = OutDir1 : 'Jobs' if KgdMakeDir(JobDir) then null RtnDir = OutDir1 : 'Routines' if KgdMakeDir(RtnDir) then null * ----------------------------------------------------------------- openpath JobDir to JobDirPtr else ErrMsg = "Error: Bad JobDir = ":JobDir goto ErrRtn end print 'Clearing ... ':JobDir CLEARFILE JobDirPtr openpath RtnDir to RtnDirPtr else ErrMsg = "Error: Bad RtnDir = ":RtnDir goto ErrRtn end print 'Clearing ... ':RtnDir CLEARFILE RtnDirPtr * ----------------------------------------------------------------- HeaderRec = '' InHeader = @false NewDsx = '' InJob = @false InRtn = @false NoLines = dcount(AllDsxRec, @FM) * ----------------------------------------------------------------- for j=1 to NoLines Line = AllDsxRec<j> if DebugSw then print Line FirstWord = field(trim(Line), ' ', 1) FirstUp = upcase(FirstWord) SecondWord = field(trim(Line), ' ', 2) SecondUp = upcase(SecondWord) * ----------------------------------------------------------------- begin case case FirstUp = 'BEGIN' begin case case SecondUp = 'HEADER' InHeader = @true HeaderRec = Line case SecondUp = 'DSJOB' InJob = @true NewDsx = Line JobName = field(AllDsxRec<j+1>, '"', 2) print 'JobName =':JobName case SecondUp = 'DSROUTINES' InRtn = @true NewDsx = '' RoutineName = field(AllDsxRec<j+2>, '"', 2) case SecondUp = 'DSRECORD' and InRtn if NewDsx <> '' then * ----------------------------------------------------------------- * write out routine (add DSROUTINES) * ----------------------------------------------------------------- NewDsxRec = RtnHeaderRec NewDsxRec := @FM : NewDsx NewDsxRec := @FM : 'END DSROUTINES' print 'Routine = ':RoutineName write NewDsxRec on RtnDirPtr, RoutineName:'.dsx' else ErrMsg = "Error: Unable to write to ":RtnDir goto ErrRtn end if DebugSw then goto TheEnd NewDsx = Line RoutineName = field(AllDsxRec<j+1>, '"', 2) end else NewDsx = Line end case @true NewDsx := @FM : Line end case case FirstUp = 'END' begin case case SecondUp = 'HEADER' InHeader = @false HeaderRec := @FM : Line RtnHeaderRec = HeaderRec RtnHeaderRec := @FM : 'BEGIN DSROUTINES' * ----------------------------------------------------------------- * write out header * ----------------------------------------------------------------- NewDsxRec = HeaderRec write NewDsxRec on DirPtr, 'KgdHeader.dsx' else ErrMsg = "Error: Unable to write to ":Path goto ErrRtn end case SecondUp = 'DSJOB' InJob = @false NewDsx := @FM : Line * ----------------------------------------------------------------- * write out job * ----------------------------------------------------------------- NewDsxRec = HeaderRec NewDsxRec := @FM : NewDsx convert ':' to '_' in JobName if not(CatOnlySw) and index(JobsToSkip,JobName[1,1],1)=0 then print 'Job = ':JobName write NewDsxRec on JobDirPtr, JobName:'.dsx' else ErrMsg = "Error: Unable to write to ":JobDir goto ErrRtn end end CatId = Category :".dsx" read CatRec from JobDirPtr, CatId else CatRec = HeaderRec end CatRec := @FM : NewDsx write CatRec on JobDirPtr, CatId else ErrMsg = "Error: Unable to write to ":JobDir:', ':CatId goto ErrRtn end if DebugSw then goto TheEnd case SecondUp = 'DSROUTINES' InRtn = @false * NewDsx := @FM : Line * ----------------------------------------------------------------- * write out routine * ----------------------------------------------------------------- NewDsxRec = RtnHeaderRec NewDsxRec := @FM : NewDsx NewDsxRec := @FM : 'END DSROUTINES' print 'Routine = ':RoutineName write NewDsxRec on RtnDirPtr, RoutineName:'.dsx' else ErrMsg = "Error: Unable to write to ":RtnDir goto ErrRtn end if DebugSw then goto TheEnd case @true NewDsx := @FM : Line end case case @true begin case case InHeader HeaderRec := @FM : Line begin case case FirstWord = 'ServerName' ServerName = field(Line, '"', 2) print 'ServerName =':ServerName case FirstWord = 'ToolInstanceID' ProjectName = field(Line, '"', 2) print 'ProjectName =':ProjectName case FirstWord = 'Date' ExportDate = field(Line, '"', 2) convert "-" to "" in ExportDate print 'ExportDate =':ExportDate end case case @true NewDsx := @FM : Line if FirstWord = 'Category' then Category = field(Line, '"', 2) convert "" to "_" in Category print 'Category =':Category end end case end case * ----------------------------------------------------------------- next j Ans = 'Double click me' end else ErrMsg = 'Error: Bad FileName = ':FileName goto ErrRtn end goto TheEnd * ----------------------------------------------------------------- ErrRtn: print print ErrMsg print * ----------------------------------------------------------------- TheEnd: print Ans
ParsDSX.pl should work without mods in CYGWIN environment...
Hi guys. It's been a while. I have since moved on and have not used DataSatage for a few years. I have moved over to the dark side and have been using Informatica Power Center for the last three years.
As I recall, I had both ActivePerl (http://www.activestate.com) and CYGWIN (http://www.cygwin.com) installed on my PC at the time I wrote ParseDSX. While I can't specifically remember running ParseDSX on CYGWIN, I surley must have. But, to try to address this latest issue, I was going to try today. But, ha, I don't happen to have a DSX file sitting around for me to play with. If someone wants to send me one, that would be great. You can send it to my personal address scboyce@tampabay.rr.com.
Testing aside, it should work. I did not use any fancy file I/O functions inside the script. Just standard open, print commands. So, at first blush, I would say that there is something odd going on in the CYGWIN environment on that particular machine. Perl is usually pretty good about portability between platforms especially for non O/S specific stuff.
As I recall, I had both ActivePerl (http://www.activestate.com) and CYGWIN (http://www.cygwin.com) installed on my PC at the time I wrote ParseDSX. While I can't specifically remember running ParseDSX on CYGWIN, I surley must have. But, to try to address this latest issue, I was going to try today. But, ha, I don't happen to have a DSX file sitting around for me to play with. If someone wants to send me one, that would be great. You can send it to my personal address scboyce@tampabay.rr.com.
Testing aside, it should work. I did not use any fancy file I/O functions inside the script. Just standard open, print commands. So, at first blush, I would say that there is something odd going on in the CYGWIN environment on that particular machine. Perl is usually pretty good about portability between platforms especially for non O/S specific stuff.
Re: ParsDSX.pl should work without mods in CYGWIN environmen
I unfortunately have not had a ton of time to debug the script but I agree.. at first glance I don't see anything funky going on.
Just for grins I ran the script under FreeBSD 6.0 and the results were similar.. Here is an example of the perms I got for the directories it created:
dr----x--x 2 root wheel 512 Apr 11 08:02 Harvester
Just for grins I ran the script under FreeBSD 6.0 and the results were similar.. Here is an example of the perms I got for the directories it created:
dr----x--x 2 root wheel 512 Apr 11 08:02 Harvester
-
- Charter Member
- Posts: 4
- Joined: Mon Sep 19, 2005 3:06 pm
For anyone interested in an sqr that performs the cut, consider this a template.
Code: Select all
!
! dsx_cutter.sqr
!
#define input_dsx_file '\\rtwpsm01p\ds_upd\patches\4064689_upd625274\UPD625274\ETL_ASCL\ACCT_TYPE_TBL_JOBS.dsx'
#define output_dsx_file_directory '\\rtwpsm01p\ds_upd\patches\4064689_upd625274\'
#define overwrite_dttm 'N' ! Y to overwrite
#define generic_header 'N' ! Y to use neutral header
begin-program
do Init
do Read-DSX
close 1
end-program
begin-procedure Read-DSX
do Init-Job-Flags
while 1 = 1
read 1 into $x:2500
!show 'read this >' $x '<'
if #end-file = 1
break
end-if
if instr($x, 'END DSJOB', 0) or instr($x, 'END DSSHAREDCONTAINER', 0) or instr($x, 'END DSROUTINES', 0) or instr($x, 'END DSEXECJOB', 0)
do Load-Line-Into-Array
do Init-Job-Flags
let $output_dsx_file = {output_dsx_file_directory} || $DSJOB_Identifier || '.dsx'
if instr($x, 'END DSEXECJOB', 0)
let $output_dsx_file = {output_dsx_file_directory} || $DSJOB_Identifier || '_exe.dsx'
end-if
open $output_dsx_file as 2 for-writing record=2500:vary
if {generic_header} = 'Y'
do Write-Header
else
do Write-DSXHeaderArray
end-if
do dump-DSXJobArray
clear-array name=DSXJobArray
close 2
end-if
if instr($x, 'BEGIN DSJOB', 0) or instr($x, 'BEGIN DSSHAREDCONTAINER', 0) or instr($x, 'BEGIN DSEXECJOB', 0)
add 1 to #DSJOB_Flag
do Load-Line-Into-Array
read 1 into $x:200
let $GCBDQ_Input_String = $x
do Get-Content-Between-Double-Quotes
let $DSJOB_Identifier = $GCBDQ_Output_String
else
if instr($x, 'BEGIN DSROUTINES', 0)
add 1 to #DSJOB_Flag
do Load-Line-Into-Array
read 1 into $x:200
do Load-Line-Into-Array
read 1 into $x:200
let $GCBDQ_Input_String = $x
do Get-Content-Between-Double-Quotes
let $DSJOB_Identifier = $GCBDQ_Output_String
end-if
end-if
if #DSJOB_Flag
do Load-Line-Into-Array
else
if instr($x, 'BEGIN HEADER', 0)
do Load-Header-Into-Array
else
if instr($x, 'END', 0)
show 'Job extracted: ' $DSJOB_Identifier
else
if Not instr($x, 'COMMENT', 0)
show 'unsupported line >' $x '<'
end-if
end-if
end-if
end-if
end-while
!close 2
end-procedure
begin-procedure Load-Line-Into-Array
if substr(ltrim($x,' '),1,12) = 'DateModified'
let $OD_Input_String = $x
do Overwrite-Date
let $x = $OD_Output_String
end-if
if substr(ltrim($x,' '),1,12) = 'TimeModified'
let $OT_Input_String = $x
do Overwrite-Time
let $x = $OT_Output_String
end-if
put $x into DSXJobArray(#DSXJobArray_index)
add 1 to #DSXJobArray_size
add 1 to #DSXJobArray_index
end-procedure
begin-procedure Load-Header-Into-Array
let #DSXHeaderArray_size = 0
let #DSXHeaderArray_index = 0
clear-array name=DSXHeaderArray
while 1 = 1
if substr(ltrim($x,' '),1,4) = 'Date'
let $OD_Input_String = $x
do Overwrite-Date
let $x = $OD_Output_String
end-if
if substr(ltrim($x,' '),1,4) = 'Time'
let $OT_Input_String = $x
do Overwrite-Time
let $x = $OT_Output_String
end-if
put $x into DSXHeaderArray(#DSXHeaderArray_index)
add 1 to #DSXHeaderArray_size
add 1 to #DSXHeaderArray_index
read 1 into $x:200
!show 'read this >' $x '<'
if instr($x, 'END HEADER', 0)
put $x into DSXHeaderArray(#DSXHeaderArray_index)
add 1 to #DSXHeaderArray_size
add 1 to #DSXHeaderArray_index
break
end-if
end-while
end-procedure
begin-procedure Init-Job-Flags
let #DSJOB_Flag = 0
let #DSRECORD_Flag = 0
let #DSRECORD_ROOT = 0
end-procedure
begin-procedure Get-Content-Between-Double-Quotes
let #length = length($GCBDQ_Input_String)
let #offset = instr($GCBDQ_Input_String, '"',0) + 1
let $GCBDQ_Output_String = substr($GCBDQ_Input_String,#offset,#length - #offset)
end-procedure
begin-procedure Overwrite-Date
if {overwrite_dttm} = 'Y'
let #length = length($OD_Input_String)
let #offset = instr($OD_Input_String, '"',0)
let $OD_Output_String = substr($OD_Input_String,0,#offset) || '2001-01-01"'
else
let $OD_Output_String = $OD_Input_String
end-if
end-procedure
begin-procedure Overwrite-Time
if {overwrite_dttm} = 'Y'
let #length = length($OT_Input_String)
let #offset = instr($OT_Input_String, '"',0)
let $OT_Output_String = substr($OT_Input_String,0,#offset) || '01.00.00"'
else
let $OT_Output_String = $OT_Input_String
end-if
end-procedure
begin-procedure Init
!while (1)
! input $A status=#stat batch-mode
! if #stat = 3
! break
! else
! show 'input >' $A
! end-if
!end-while
!stop
create-array name=DSXJobArray
size=50000
field=dsxline:char
let #DSXJobArray_size = 0
let #DSXJobArray_index = 0
create-array name=DSXHeaderArray
size=20
field=dsxline:char
let #DSXHeaderArray_size = 0
let #DSXHeaderArray_index = 0
open {input_dsx_file} as 1 for-reading record=2500:vary
end-procedure
begin-procedure dump-DSXJobArray
let #DSXJobArray_index = 0
while (#DSXJobArray_index < #DSXJobArray_size)
get $dsjob_wrk from DSXJobArray(#DSXJobArray_index)
write 2 from $dsjob_wrk
!show 'DSXJobArray >' $dsjob_wrk '<'
add 1 to #DSXJobArray_index
end-while
let #DSXJobArray_size = 0
let #DSXJobArray_index = 0
end-procedure
begin-procedure Write-DSXHeaderArray
let #DSXHeaderArray_index = 0
while (#DSXHeaderArray_index < #DSXHeaderArray_size)
get $dsjob_wrk from DSXHeaderArray(#DSXHeaderArray_index)
write 2 from $dsjob_wrk
add 1 to #DSXHeaderArray_index
end-while
end-procedure
begin-procedure Write-Header
write 2 from 'BEGIN HEADER'
write 2 from ' CharacterSet "ENGLISH"'
write 2 from ' ExportingTool "Ascential DataStage Export"'
write 2 from ' ToolVersion "4"'
write 2 from ' ServerName "server"'
write 2 from ' ToolInstanceID "projectname"'
write 2 from ' MDISVersion "1.0"'
write 2 from ' Date "2001-01-01"'
write 2 from ' Time "01.00.00"'
write 2 from ' ServerVersion "7.5"'
write 2 from 'END HEADER'
end-procedure[code!]
I have a fix for the bug. It turns out that I improperly used the mkdir function in the script. I was passing 777 for the MODE where I needed to be passing 0777. The bug seemed to only cause issues in the CYGWin environment. The script was working fine under RH Linux, Solaris, HPUX and DOS (ActiveState). So I must have not run it under CYGWin when I originally developed it. It's always what you don't test that will break. Go figure.
All I did was change the call to MakeDir() and pass 0777 instead of 777 if you have modified your own version.
If you want a copy of the updated master script, here it is...
All I did was change the call to MakeDir() and pass 0777 instead of 777 if you have modified your own version.
If you want a copy of the updated master script, here it is...
Code: Select all
#!/usr/bin/perl
##############################################################################
#
# Program: ParseDSX.pl
#
# Description: See ShowBlurb function below for details
#
# === Modification History ===================================================
# Date Author Comments
# ---------- --------------- -------------------------------------------------
# 07-18-2002 Steve Boyce Created.
# 08-21-2002 Steve Boyce Exporting routines now includes binary info.
# 08-28-2002 Steve Boyce Corrected bug relating to jobs and routines
# located in the root folder. They now get
# created in the correct location.
# 08-28-2002 Steve Boyce Changed default output directory to be the name
# of the dsx file being parsed without the
# extension.
# -s option now works.
# 10-04-2002 Steve Boyce Eliminated -c option. That now the default
# and only behavior.
# Default Parmameter metadata is now stripped out
# of all jobs except any jobs that have PROTOTYPE
# or Batch:UTIL in the name.
# Routines are unaffected.
# 11-12-2002 Steve Boyce Added Version dipsplay option.
# Corrected source code generation bug.
# 02-27-2003 Steve Boyce Added -x option to strip out ValidationStatus
# 03-21-2003 Steve Boyce Stripping out ValidationStatus is now default
# behavior.
# 04-29-2003 Steve Boyce Bumped version number
# 04-26-2006 Steve Boyce Fixed mkdir bug where directories were getting
# created with the wrong permissions. Bug only
# manifested itself when running under CYGWin.
#
##############################################################################
use Getopt::Std;
use File::Basename;
my $version="2.1.01";
##############################################################################
sub ShowBlurb
{
print <<ENDOFBLURB;
Syntax: ParseDSX.pl -h -l<ListFile> -o<OutputDir> -s -v -y <DSXFile>
Version: $version
Description: Extracts individual jobs and routines from a DataStage export
file.
Parameters: <DSXFile> Name of DataStage DSX file to parse. This file is
assumed to be generated from the DataStage export
process.
Options: -l job/routine list file. (future enhancement)
This file contains a list of jobs and routines to extract
from the <DSXFile>.
-o Explicitly specify <OutputDir> directory.
Default is the name of the parsed dsx file without the
extension in the current directory.
-s Extract job "Job Control Code" and routine "source"
code into "source files".
<job>.src and <routine>.src
These will appear in the same directory as the generated
dsx files in the <OutputDir> directory.
-v Display version information.
-y Force a "Yes" answer to overwrite existing <OutputDir>
directory prompt.
-h This help.
Notes: Job and routine names are case sensitive in DataStage. Extracted
jobs and routines are placed in file names constructed based on
job or routine names. Running this utility on the Windows
platform will ignore case and possibly consider some jobs and
routines duplicates when the UNIX platform will not.
It is a good practice to not rely on case as a differentiator
for file names.
ENDOFBLURB
}
##############################################################################
sub ShowVersion
{
print <<ENDOFBLURB;
ParseDSX.pl Version $version
ENDOFBLURB
}
##############################################################################
sub DieWith
{
my ($MessageLine) = @_;
print "$MessageLine\nType ParseDSX.pl -h for help.\n";
exit 1;
}
##############################################################################
sub OKToOverWriteOutputDir
{
my ($OutPutDirectory, $opt_y) = @_;
my $RetVal = 0;
if ( -e $OutPutDirectory ) {
if ( $opt_y ) {
print "*** Warning: <OutputDir> directory ($OutPutDirectory) already exists. Using anyway.\n";
$RetVal = 1;
}
else {
print "*** Warning: <OutputDir> directory ($OutPutDirectory) already exists.\n";
print "Proceed anyway? [y|n] ";
$Ans = <STDIN>;
chomp($Ans) if ($Ans);
if ( "$Ans" eq "Y" || "$Ans" eq "y" ) {
$RetVal = 1;
}
else {
DieWith("Aborting.");
}
}
}
else {
if ( MakeDir($OutPutDirectory, 0777) ) {
$RetVal = 1;
}
else {
DieWith("Error: Could not create ($OutPutDirectory) directory");
}
}
return $RetVal;
}
##############################################################################
sub LoadObjectList
{
my ($DSXListFile) = @_;
my %DSXObjectList = ();
if ( $DSXListFile ) {
if (open fhDSXListFile, "<".$DSXListFile) {
while (<fhDSXListFile>) {
chop;
#-- Push line onto array
$DSXObjectList{$_} = 1;
}
close fhDSXListFile;
}
else {
DieWith("Error: Can't open $DSXListFile");
}
while ( ($key,$value) = each %DSXObjectList ) {
print "$key=$value\n";
}
}
return %DSXObjectList;
}
##############################################################################
sub MakeDir
{
my ($FullDirPath, $Mode) = @_;
my @DirList = ();
my $PartialDirPath = "";
my $RetVal = 1;
$FullDirPath =~ tr/\\/\//;
@DirList = split(/\//,$FullDirPath);
foreach $Directory ( @DirList ) {
$PartialDirPath = $PartialDirPath . $Directory. "/" ;
if ( ! (length($PartialDirPath) == 3 && substr($PartialDirPath, 1, 2) eq ":/") ) {
if ( ! -e $PartialDirPath ) {
if ( ! mkdir($PartialDirPath, $Mode) ) {
$RetVal = 0;
}
}
}
}
return $RetVal;
}
##############################################################################
sub ParseQuotedString
{
my ($InputLine) = @_;
my $FirstQuotePos = 0;
my $SecondQuotePos = 0;
my $Length = 0;
$FirstQuotePos = index($InputLine, '"');
$SecondQuotePos = index($InputLine, '"', $FirstQuotePos+1);
$Length = $SecondQuotePos - $FirstQuotePos;
return substr($InputLine, $FirstQuotePos + 1, $Length - 1);
}
##############################################################################
sub MakeDuplicateName
{
my ($OriginalName) = @_;
my $NewName = "";
my $DupSuffix = 1;
$NewName = $OriginalName . "_dup" . "$DupSuffix";
while ( -e $NewName ) {
if ( $DupSuffix > 99 ) {
DieWith("Error: There seems to be more than 99 duplicate jobs or routines.\n");
}
$DupSuffix += 1;
$NewName = $OriginalName . "_dup" . "$DupSuffix";
}
return $NewName;
}
##############################################################################
sub WriteDSXHeader
{
my ($fhOutputFile) = @_;
print $fhOutputFile "BEGIN HEADER\n";
print $fhOutputFile " CharacterSet \"ENGLISH\"\n";
print $fhOutputFile " ExportingTool \"Ardent DataStage Export\"\n";
print $fhOutputFile " ToolVersion \"3\"\n";
print $fhOutputFile " ServerName \"$cStandardServerName\"\n";
print $fhOutputFile " ToolInstanceID \"$cStandardToolInstanceID\"\n";
print $fhOutputFile " MDISVersion \"1.0\"\n";
print $fhOutputFile " Date \"$cStandardDate\"\n";
print $fhOutputFile " Time \"$cStandardTime\"\n";
print $fhOutputFile "END HEADER\n";
}
##############################################################################
sub WriteDSXObjectFile
{
my ($ObjectType, $tmpDSXObjectHolder, $OutPutDirectory, $DSXObjectName, $DSXCategoryName) = @_;
my $x = 0;
my $TranslatedDSXObjectName = "";
my $TranslatedCategorytName = "";
my $OutputFileName = "";
my $OutputLine = "";
my $WriteLine = 1;
$TranslatedDSXObjectName = $DSXObjectName;
$TranslatedDSXObjectName =~ tr/:/_/;
$TranslatedDSXObjectName =~ tr/ /_/;
if ($ObjectType eq "JOB") {
$OutPutDirectory = $OutPutDirectory . "/jobs";
if ( ! -e $OutPutDirectory ) {
if ( ! MakeDir($OutPutDirectory, 0777) ) {
DieWith("Error: Could not create directory: $OutPutDirectory");
}
}
}
else {
$OutPutDirectory = $OutPutDirectory . "/routines";
if ( ! -e $OutPutDirectory ) {
if ( ! MakeDir($OutPutDirectory, 0777) ) {
DieWith("Error: Could not create directory: $OutPutDirectory");
}
}
}
if ($DSXCategoryName) {
$TranslatedCategoryName = $DSXCategoryName;
$TranslatedCategoryName =~ tr/ /_/;
$TranslatedCategoryName =~ tr/\\/\//s;
$OutPutDirectory = $OutPutDirectory . "/" . $TranslatedCategoryName;
if ( ! -e $OutPutDirectory ) {
if ( ! MakeDir($OutPutDirectory, 0777) ) {
DieWith("Error: Could not create directory: $OutPutDirectory");
}
}
}
$OutputFileName = $OutPutDirectory . "/" . $TranslatedDSXObjectName . ".dsx";
print "Writing File: $OutputFileName...\n";
if ( -e $OutputFileName ) {
print "*** WARNING: Job/Routine output DSX file ($OutputFileName) already exists. Creating duplicate.\n";
$OutputFileName = MakeDuplicateName($OutputFileName);
}
if (open (fhOutputFile, ">$OutputFileName")) {
WriteDSXHeader(\*fhOutputFile);
if ($ObjectType eq "ROUTINE") {
print fhOutputFile "BEGIN DSROUTINES\n";
}
while ( $$tmpDSXObjectHolder[$x] ) {
$OutputLine = $$tmpDSXObjectHolder[$x];
$WriteLine = 1;
#-- Filter ValidationStatus metadata out
#-- This metadata seems to be intermitent with no value added.
if ($OutputLine =~ /^.*ValidationStatus /) {
$WriteLine = 0;
}
if ($WriteLine) {
#-- Normalize dates and times
if ($OutputLine =~ /^ {3,6}DateModified /) {
$OutputLine =~ s/\".{10}\"/\"$cStandardDate\"/;
}
else {
if ($OutputLine =~ /^ {3,6}TimeModified /) {
$OutputLine =~ s/\".{8}\"/\"$cStandardTime\"/;
}
}
#-- Send the line to the output file
print fhOutputFile "$OutputLine";
}
$x = $x + 1;
}
if ($ObjectType eq "ROUTINE") {
print fhOutputFile "END DSROUTINES\n";
}
close fhOutputFile;
}
}
##############################################################################
sub WriteDSXSourceFile
{
my ($ObjectType, $tmpDSXObjectSourceHolder, $OutPutDirectory, $DSXObjectName, $DSXCategoryName) = @_;
my $x = 0;
my $TranslatedDSXSourceName = "";
my $TranslatedCategorytName = "";
my $OutputFileName = "";
my $OutputLine = "";
$TranslatedDSXSourceName = $DSXObjectName;
$TranslatedDSXSourceName =~ tr/:/_/;
$TranslatedDSXSourceName =~ tr/ /_/;
if ($ObjectType eq "JOB") {
$OutPutDirectory = $OutPutDirectory . "/jobs";
$SourceKeyword = "JobControlCode";
}
else {
$OutPutDirectory = $OutPutDirectory . "/routines";
$SourceKeyword = "Source";
}
if ($DSXCategoryName) {
$TranslatedCategoryName = $DSXCategoryName;
$TranslatedCategoryName =~ tr/ /_/;
$TranslatedCategoryName =~ tr/\\/\//s;
$OutPutDirectory = $OutPutDirectory . "/" . $TranslatedCategoryName;
}
$OutputFileName = $OutPutDirectory . "/" . $TranslatedDSXSourceName . ".src";
if ( -e $OutputFileName ) {
$OutputFileName = MakeDuplicateName($OutputFileName);
}
#-- Convert single line encoded source code to properly formated code
#-- Chop off trailing 6 spaces after every CR-LF (really leading 6 spaces)
#-- Convert "symbolic CR-LF to real CR-LF
$tmpDSXObjectSourceHolder =~ s/\\\(D\)\\\(A\)/\n/g;
#-- Chop off leading keyword - either Source " or JobControlCode "
$tmpDSXObjectSourceHolder =~ s/^ *$SourceKeyword "//;
#-- Chop off trailing quote
$tmpDSXObjectSourceHolder =~ s/\" *$//;
#-- Replace all \" with "
$tmpDSXObjectSourceHolder =~ s/\\\"/\"/g;
#-- replace all \\ with \
$tmpDSXObjectSourceHolder =~ s/\\\\/\\/g;
if (open (fhOutputFile, ">$OutputFileName")) {
print fhOutputFile $tmpDSXObjectSourceHolder;
close fhOutputFile;
}
}
##############################################################################
sub OKToStripDefaultValue
{
my ($DSXObjectName, $DSParameterName) = @_;
my $RetVal = 0;
if (! ($DSXObjectName =~ /Batch::UTIL/) ) {
if (! ($DSXObjectName =~ /PROTOTYPE/) ) {
if (! ($DSParameterName eq "JobName" or $DSParameterName eq "PartitionNumber" or $DSParameterName eq "PartitionCount" ) ) {
$RetVal = 1;
}
}
}
return $RetVal
}
##############################################################################
sub ParseDSXObjects
{
my ($DSXFileName, $Greppize, $OutPutDirectory, $DSXObjectList) = @_;
my @tmpDSXObjectHolder = ();
my $tmpDSXObjectSourceHolder = "";
my $DSXObjectName = "";
my $DSXCategoryName = "";
my $InDSJobBlock = 0;;
my $InDSRoutineBlock = 0;
my $InDSRecordBlock = 0;
my $InDSSubRecordBlock = 0;
my $InDSUBinaryBlock = 0;
my $DSParameterName = "";
if (open fhDSXFileName, "<".$DSXFileName) {
while (<fhDSXFileName>) {
if ($InDSJobBlock) {
push(@tmpDSXObjectHolder, $_);
if ($_ =~ /^END DSJOB/) {
$InDSJobBlock = 0;
WriteDSXObjectFile("JOB", \@tmpDSXObjectHolder, $OutPutDirectory,
$DSXObjectName, $DSXCategoryName);
if ( $tmpDSXObjectSourceHolder ) {
WriteDSXSourceFile("JOB", $tmpDSXObjectSourceHolder, $OutPutDirectory,
$DSXObjectName, $DSXCategoryName);
}
}
else {
if ($InDSRecordBlock) {
if ($_ =~ /^ END DSRECORD/) {
$InDSRecordBlock = 0;
}
else {
if ($InDSSubRecordBlock) {
if ($_ =~ /^ END DSSUBRECORD/) {
$InDSSubRecordBlock = 0;
}
else {
if ($_ =~ /^ Name/) {
$DSParameterName = ParseQuotedString($_);
}
if ($_ =~ /^ Default/) {
if (OKToStripDefaultValue($DSXObjectName, $DSParameterName)) {
pop(@tmpDSXObjectHolder);
}
}
}
}
else {
if ($_ =~ /^ BEGIN DSSUBRECORD/) {
$InDSSubRecordBlock = 1;
}
else {
if ($_ =~ /^ Category /) {
$DSXCategoryName = ParseQuotedString($_);
}
if ($Greppize) {
if ($_ =~ /^ JobControlCode /) {
$tmpDSXObjectSourceHolder = $_;
}
}
}
}
}
}
else {
if ($_ =~ /^ BEGIN DSRECORD/) {
$InDSRecordBlock = 1;
}
else {
if ($_ =~ /^ Identifier /) {
$DSXObjectName = ParseQuotedString($_);
}
}
}
}
}
else {
if ($InDSRoutineBlock) {
if ($_ =~ /^END DSROUTINES/) {
$InDSRoutineBlock = 0;
}
else {
if ($InDSRecordBlock) {
push(@tmpDSXObjectHolder, $_);
if ($_ =~ /^ END DSRECORD/) {
$InDSRecordBlock = 0;
}
else {
if ($_ =~ /^ Identifier /) {
$DSXObjectName = ParseQuotedString($_);
}
else {
if ($_ =~ /^ Category /) {
$DSXCategoryName = ParseQuotedString($_);
}
if ($Greppize) {
if ($_ =~ /^ Source /) {
$tmpDSXObjectSourceHolder = $_;
}
}
}
}
}
else {
if ($InDSUBinaryBlock) {
push(@tmpDSXObjectHolder, $_);
if ($_ =~ /^ END DSUBINARY/) {
$InDSUBinaryBlock = 0;
WriteDSXObjectFile("ROUTINE", \@tmpDSXObjectHolder, $OutPutDirectory,
$DSXObjectName, $DSXCategoryName);
if ( $tmpDSXObjectSourceHolder ) {
WriteDSXSourceFile("ROUTINE", $tmpDSXObjectSourceHolder, $OutPutDirectory,
$DSXObjectName, $DSXCategoryName);
}
}
else {
if ($_ =~ /^ COMMENT Record is empty/) {
print "*** WARNING: Routine ($DSXObjectName) is missing compiled executable.\n";
}
}
}
else {
if ($_ =~ /^ BEGIN DSRECORD/) {
$InDSRecordBlock = 1;
@tmpDSXObjectHolder = ();
push(@tmpDSXObjectHolder, $_);
$tmpDSXObjectSourceHolder = "";
$DSXCategoryName = "";
}
else {
if ($_ =~ /^ BEGIN DSUBINARY/) {
$InDSUBinaryBlock = 1;
push(@tmpDSXObjectHolder, $_);
}
}
}
}
}
}
else {
if ($_ =~ /^BEGIN DSJOB/) {
$InDSJobBlock = 1;
@tmpDSXObjectHolder = ();
push(@tmpDSXObjectHolder, $_);
$tmpDSXObjectSourceHolder = "";
$DSXCategoryName = "";
}
else {
if ($_ =~ /^BEGIN DSROUTINES/) {
$InDSRoutineBlock = 1;
}
}
}
}
}
close (fhDSXFileName);
}
}
##############################################################################
# Main
#-- Global variables (constants)
$cStandardDate = "2001-01-01";
$cStandardTime = "01.00.00";
$cStandardServerName = "ServerName";
$cStandardToolInstanceID = "ToolInstanceID";
#-- Local variables
my %DSXObjectList = ();
my $NumArgs = 0;
my $DSXFileName = "";
my $OutPutDirectory = "";
my $Ans = "";
if (getopts('hl:o:svy')) {
if ( $opt_h ) {
ShowBlurb();
exit 2;
}
if ( $opt_v ) {
ShowVersion();
exit 2;
}
$NumArgs = scalar(@ARGV);
if ( $NumArgs == 1 ) {
$DSXFileName = $ARGV[0];
if ( -r $DSXFileName ) {
if ( $opt_o ) {
$OutPutDirectory = $opt_o;
}
else {
$OutPutDirectory = basename($DSXFileName, ".dsx");
}
if ( OKToOverWriteOutputDir($OutPutDirectory, $opt_y) ) {
%DSXObjectList = LoadObjectList($opt_l);
ParseDSXObjects($DSXFileName, $opt_s, $OutPutDirectory, \@DSXObjectList);
}
}
else {
DieWith("Error: Unable to read file ($DSXFileName).");
}
}
else {
DieWith("Error: Invalid filespec.");
}
}
else {
DieWith("Error: Invalid options.");
}
This worked great!
I tried the new version.. and it worked great.. thanks!
I was wondering why the script doesn't handle other object like shared containers, table definitions, stuff like that?
thanks!
I was wondering why the script doesn't handle other object like shared containers, table definitions, stuff like that?
thanks!
Re: This worked great!
Because that's how I had him write it. Table definitions are completely recreatable and DS is not the system of record. MetaStage, ERWin, etc is the place for that.Cowmix wrote:I was wondering why the script doesn't handle other object like shared containers, table definitions, stuff like that?
Feel free to enhance it and share! :D
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Re: This worked great!
If anyone else sees the merits of having shared containers 'cut out' too.. I'll gladly give it shot.
Re: I need a DSX-Cutter
Hi all,
so does this DSX cutter work for version 7.1, i.e., jobs designed with DS version 7.1 server edition and exported into a dsx file?
so does this DSX cutter work for version 7.1, i.e., jobs designed with DS version 7.1 server edition and exported into a dsx file?
Thanks,
Chad
__________________________________________________________________
"There are three kinds of people in this world; Ones who know how to count and the others who don't know how to count !"
Chad
__________________________________________________________________
"There are three kinds of people in this world; Ones who know how to count and the others who don't know how to count !"