TO Check If a DOS filepath exists using Basic.
Moderators: chulett, rschirm, roy
TO Check If a DOS filepath exists using Basic.
Hey,
I havent used a whole lot of basic and hence I ask. One of my columns in the SQL server is a file path. e.g (\\ pathname\edxt\data\folder\abc.tiff) I want to check if this file path and document actually exists on the system before I transfer the data from my transformer.How can I check for that? If it does'nt exist I need to write that row to an error file.
Any suggestions would be helpful.
Thanx guys
I havent used a whole lot of basic and hence I ask. One of my columns in the SQL server is a file path. e.g (\\ pathname\edxt\data\folder\abc.tiff) I want to check if this file path and document actually exists on the system before I transfer the data from my transformer.How can I check for that? If it does'nt exist I need to write that row to an error file.
Any suggestions would be helpful.
Thanx guys
Do it in job control before your run your job. Simply use a DOS batch script using the dir command and check for the results of the file. Use the DSExecute API (see your manual) and either call it using a custom BATCH job control, or use it in a DS user function/routine.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Create a Routine that attempts to open the file as if it were a record in a table, relying on the fact that DataStage BASIC can treat the folder as a table. For example:
Code: Select all
FUNCTION FileExists(ThePathname)
* Returns 1 (= true) if file exists, 0 (= false) otherwise.
OpenPath(ThePathname) To FileVariable
On Error
Ans = @FALSE
End
Then
Ans = @TRUE
Close FileVariable
End
Else
Ans = @FALSE
End
RETURN(Ans)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Writing a routine worked!!!
Hey ,
I wrote a transform and it worked.So thanx for your prompt response.Thanks a lot Ray.I am going to try your response as well.
This is what I wrote in the Transform
FUNCTION KsDirExists(Arg1)
Ans = 10
openseq Arg1 to testfile Then Ans = 'opened'
Else Ans = 'not Opened'
closeseq testfile
RETURN(Ans)
I wrote a transform and it worked.So thanx for your prompt response.Thanks a lot Ray.I am going to try your response as well.
This is what I wrote in the Transform
FUNCTION KsDirExists(Arg1)
Ans = 10
openseq Arg1 to testfile Then Ans = 'opened'
Else Ans = 'not Opened'
closeseq testfile
RETURN(Ans)
You need to be careful if you are doing a lot of repetitive calls to a DS function doing OPENSEQ statements looking for the same file. DS has issues and will false-positive hit sometimes. The most solid solution is to use the OS's own tools for file existence (dir on NT and ls or test on UNIX).
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Arrr, who needs performance?!!
(He's right, of course.)
(He's right, of course.)
Code: Select all
FUNCTION FileExists(Pathname)
If System(91)
Then
Command = "DIR " : Pathname
Shell = "DOS"
End
Else
Command = "ls " : Pathname
Shell = "UNIX"
End
Call DSExecute(Shell, Command, Output, Code)
Ans = (Code = 0)
RETURN(Ans)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Filepath
The Basic Manual says not to use this as a transform but basically I need to check this for 60,000 rows. The file path is a column in my Sql server table. So if i write a routine and call it before my transformer stage executes I need to write the ones that do not exist to a sequential file and not tranfer them to my target database.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Probably the fastest approach is to do it in BASIC as I outlined earlier, with the opened directory held open by having its file variable declared to be in COMMON. The modified code is:
Code: Select all
FUNCTION FileExists(ThePathname)
$INCLUDE UNIVERSE.INCLUDE FILEINFO.H
COMMON /FileExists/FileVariable
* Returns 1 (= true) if file exists, 0 (= false) otherwise.
* Open the file's directory if it is not already open.
If Not(FileInfo(FileVariable, FINFO$IS.FILEVAR))
Then
Call !GET.PATHNAME(ThePathname, DirPath, BaseName, Code)
OpenPath DirPath To FileVariable
On Error
Call DSLogWarn("Unable to open directory " : DirPath, RoutineName)
End
Else
Call DSLogWarn("Unable to open directory " : DirPath, RoutineName)
End
End
Ans = @NULL
If FileInfo(FileVariable, FINFO$IS.FILEVAR)
Then
ReadV FileName From FileVariable, BaseName, 0
Then Ans = 1
Else Ans = 0
End
RETURN(Ans)
Last edited by ray.wurlod on Tue May 11, 2004 8:46 pm, edited 1 time in total.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Ray,
I did not completely understand your code.
But I tried using it and compiling it and it gave me the following:
WARNING: Variable 'FINFO$IS.FILEVAR' never assigned a value.
Compilation Complete.
is what I wrote and it works for files that reside on the ascential server but not across the network.
I did not completely understand your code.
But I tried using it and compiling it and it gave me the following:
WARNING: Variable 'FINFO$IS.FILEVAR' never assigned a value.
Compilation Complete.
Code: Select all
openseq Arg1 to testfile Then Ans = 0
Else Ans = 1
closeseq testfile
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Sorry about that. I've edited the code adding one line, the $INCLUDE declaration. This loads the constants used by the FileInfo() function, in particular FINFO$IS.FILEVAR.
It should be OK now.
It should be OK now.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Failed to compile test harness error
When I used the modified code it worked in the morning.Now when I try to run it, it gives me a " Failed to compile test harness error".Do you have any clue as to what that means.
Also in the morning it wasnt reading files stored on an other server.I mean it used to open files on the Ascential server but not on any other though I had the access rights to that server.
In any case it now completely does not open anything.Thanks a lot ray for your help.
Also in the morning it wasnt reading files stored on an other server.I mean it used to open files on the Ascential server but not on any other though I had the access rights to that server.
In any case it now completely does not open anything.Thanks a lot ray for your help.
[quote="kcbland"]You need to be careful if you are doing a lot of repetitive calls to a DS function doing OPENSEQ statements looking for the same file. DS has issues and will false-positive hit sometimes. The most solid solution is to use the OS's own tools for file existence (dir on NT and ls or test on UNIX).[/quote]
Ray here is my code review.
Conceptually using the openpseq or openseq opcode to check for existance of a file is incorrect, honest. Consider what baggage and potential probems you are carrying with you.
1) You are going to create a readu lock, most people would not be aware of this. This has numerous side affects.
2) You are going to open the file for writing, why?, apart form the overhead, room for some very (actually mind boggling) obscure failures.
Checking for existence should only check the directoru header the file is contained in. Creating solid code involves not doing things that you do not need to do, unless you are very aware of the consequences.
If you must use BASIC use a select, readlist and find.
I do not have the time to give a case study of this here, but Ray I would be pleased to do so over a quiet drop of liffy water with you, next time we are in the same city.
Of course the proposed code will work for most people in most circustances most of the time. I do not expect people to be aware of these obscurities, but only doing what you have do will provide protection.
See ya soon mate.
Ray here is my code review.
Conceptually using the openpseq or openseq opcode to check for existance of a file is incorrect, honest. Consider what baggage and potential probems you are carrying with you.
1) You are going to create a readu lock, most people would not be aware of this. This has numerous side affects.
2) You are going to open the file for writing, why?, apart form the overhead, room for some very (actually mind boggling) obscure failures.
Checking for existence should only check the directoru header the file is contained in. Creating solid code involves not doing things that you do not need to do, unless you are very aware of the consequences.
If you must use BASIC use a select, readlist and find.
I do not have the time to give a case study of this here, but Ray I would be pleased to do so over a quiet drop of liffy water with you, next time we are in the same city.
Of course the proposed code will work for most people in most circustances most of the time. I do not expect people to be aware of these obscurities, but only doing what you have do will provide protection.
See ya soon mate.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Thanks, Johno.
I was aware of the OpenSeq issues - it weren't me, guv! I used OpenPath (not OpenSeq), and opened the directory into COMMON. Others magically transmogrified my OpenPath into OpenSeq. No-one seems to have noticed.
I suspect, indeed, that even that was unnecessary, since !GET.PATHNAME checks for existence would have to check that out.
Otherwise, I was under the impression that ReadV with a third argument of 0 had smarts built in so that it did only do the directory scan (or apply the hashing algorithm or do a BSCAN, as appropriate).
The pint or two of Liffey water sounds like a good plan, though. Will be in your neck of the woods the last two weeks of June. The question is, will you?
Regards,
Ray
I was aware of the OpenSeq issues - it weren't me, guv! I used OpenPath (not OpenSeq), and opened the directory into COMMON. Others magically transmogrified my OpenPath into OpenSeq. No-one seems to have noticed.
I suspect, indeed, that even that was unnecessary, since !GET.PATHNAME checks for existence would have to check that out.
Otherwise, I was under the impression that ReadV with a third argument of 0 had smarts built in so that it did only do the directory scan (or apply the hashing algorithm or do a BSCAN, as appropriate).
The pint or two of Liffey water sounds like a good plan, though. Will be in your neck of the woods the last two weeks of June. The question is, will you?
Regards,
Ray
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Are you sure? OpenPath can open Type 1/19 files. What does it mean if OpenSeq opens one of these? I just tested it (OpenSeq with a Type 19 file); it took the Else clause with a Status() of 0, even though the Type 19 file I specified actually exists. This is not in accordance with what's in the manual!jwhyman wrote:openpath and openseq are undistinguishible uner the covers, notice I said openpseq and openseq.
INMAT() returned 0, and a STATUS statement took its ELSE clause. Something funny is happening in the state of Denmark!
OpenPath was quite happy with the same Type 19 file. STATUS() returns 19, INMAT() returns 1, and STATUS statement takes its THEN clause.
I can't accept that they're indistinguishable. And, in the public version of DataStage that I'm using, OpenPseq does not compile.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.