Hello everyone,
I am writing an application that should read data from csv files for further processing. The problem is that these csv files may or may not contain a header record, depending on the source.
I know how to eliminate the header (if present) using UNIX (sed, for example does it quite elegantly). But I would like to do it using a DataStage server routine. The routine needs to check the existence of a certain string (e.g. "Country_Code") whose presence would indicate that its a header record, and then delete that record from the input file, and leave the file untouched if the string doesn't exist in it.
Any help much appreciated!
Thanks,
kaus
server routine for eliminating headers
Moderators: chulett, rschirm, roy
While you certainly could write a routine to do this, why not stick with 'sed'? It would certainly be... simpler. Heck, you could even leverage that in the Filter option of the stage. Otherwise, you'll need to investigate all of the BASIC functions for handling sequential files:
OPENSEQ
READSEQ
WRITESEQ
CLOSESEQ
WEOFSEQ
SEEK
Etc. The BASIC pdf manual will have examples while more useful ones would be found by searching the forums here for those functions.
OPENSEQ
READSEQ
WRITESEQ
CLOSESEQ
WEOFSEQ
SEEK
Etc. The BASIC pdf manual will have examples while more useful ones would be found by searching the forums here for those functions.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
How about just encapsulating this command as the filter command in your Sequential File stage?
Code: Select all
grep -v Country_Code #SourceFile#
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Then do it in a Routine.
Process the new file (filename_noheader) in your Sequential File stage.
Code: Select all
Shell = "UNIX"
Command = "grep -v Country_Code " : argSourceFilePath : " > " : argSourceFilePath : "_noheader"
Call DSExecute(Shell, Command, Result, ExitStatus)
If ExitStatus <> 0
Then
Call DSLogWarn("Error (" : ExitStatus : ") removing header.", "MyRoutine")
End
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.