Reading Multiple Files on a File Server
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 133
- Joined: Tue Nov 23, 2004 11:24 pm
- Location: India
Reading Multiple Files on a File Server
Hi
I have a requirement where i need to run a particular job picking files which are kept in a file server. The datastage is on a different machine.The Number of files on the file server might vary day to day i can get one file today and 2 files tomorrow and so on. I need to run the job the number of times as the number of files on that server. I need to automate this.
In between is there a FTP stage which i can use for getting these files on file server to my datastage server? Or is a shell script al i have for getting these files?
But the main Problem is i need to first get a count of number of files on the file server to loop the shell script and the job...
Any ideas to do this will be really Helpful
Thanks
Pavan
I have a requirement where i need to run a particular job picking files which are kept in a file server. The datastage is on a different machine.The Number of files on the file server might vary day to day i can get one file today and 2 files tomorrow and so on. I need to run the job the number of times as the number of files on that server. I need to automate this.
In between is there a FTP stage which i can use for getting these files on file server to my datastage server? Or is a shell script al i have for getting these files?
But the main Problem is i need to first get a count of number of files on the file server to loop the shell script and the job...
Any ideas to do this will be really Helpful
Thanks
Pavan
-
- Premium Member
- Posts: 1255
- Joined: Wed Feb 02, 2005 11:54 am
- Location: United States of America
If your requirement was to get the count of files to loop in your script, then take Whale's code (But you need to be careful if you have subdirectories. This command gives you the count of files inside them too)
If you are looking for an alternate approach-
You might have to write an shell script to FTP all the the file/files from the file server to your datastage server.
You don't have to know the number of files, you can use wildcards to match your search criteria.
Note: Limitation with FTP Stage - it is not possible to use wildcards.
Now that you have all the files on the datastage server, as per your requirement, you can utilize the techinque of your choice (Loops, script,..,..) to read all these files.
If you are looking for an alternate approach-
You might have to write an shell script to FTP all the the file/files from the file server to your datastage server.
You don't have to know the number of files, you can use wildcards to match your search criteria.
Note: Limitation with FTP Stage - it is not possible to use wildcards.
Now that you have all the files on the datastage server, as per your requirement, you can utilize the techinque of your choice (Loops, script,..,..) to read all these files.
Narasimha Kade
Finding answers is simple, all you need to do is come up with the correct questions.
Finding answers is simple, all you need to do is come up with the correct questions.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
... or even a filter in the find command itself, both for level searched and for file name(s). If you can see the directory to use the find command, then you can see the directory to use a Sequential File stage, and should not need to be concerned about using FTP.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 133
- Joined: Tue Nov 23, 2004 11:24 pm
- Location: India
-
- Premium Member
- Posts: 1255
- Joined: Wed Feb 02, 2005 11:54 am
- Location: United States of America
-
- Premium Member
- Posts: 133
- Joined: Tue Nov 23, 2004 11:24 pm
- Location: India
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 133
- Joined: Tue Nov 23, 2004 11:24 pm
- Location: India
Hi Ray
The File Server is also windows and i have access to the windows file server from my datastage windows server. I can ping that file server from my datastage instance and i can access the directory and the file folder in which files are kept. Sorry if i am a bit cryptic i didn't get what you meant by "see" the file server.
Is there any commands like how Whale as suggested find . -type f | wc -l in windows environment for getting the count. Does the FTP commands work in my script for the windows environment
Thanks
Pavan
The File Server is also windows and i have access to the windows file server from my datastage windows server. I can ping that file server from my datastage instance and i can access the directory and the file folder in which files are kept. Sorry if i am a bit cryptic i didn't get what you meant by "see" the file server.
Is there any commands like how Whale as suggested find . -type f | wc -l in windows environment for getting the count. Does the FTP commands work in my script for the windows environment
Thanks
Pavan
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Can you "see" the directory (folder) on the file server as a shared folder? If so, you won't need FTP.
DOS does not have an equivalent of find, but DataStage does. It's called uvwalk and can traverse a tree structure if required. However, DOS also lacks a wc command, and uvwalk does not report the number of files found. So you may be better off executing a DIR command (with appropriate wildcard) and parsing out the "n File(s)" message near the end of the output.
DOS does not have an equivalent of find, but DataStage does. It's called uvwalk and can traverse a tree structure if required. However, DOS also lacks a wc command, and uvwalk does not report the number of files found. So you may be better off executing a DIR command (with appropriate wildcard) and parsing out the "n File(s)" message near the end of the output.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 133
- Joined: Tue Nov 23, 2004 11:24 pm
- Location: India
Ray,
Thanks for the reply. but i am waiting for an approval from my manager for Buying Premium membership. it is soo frustating that i am not able to see the answer and try this but as soon as i get it i will try the solution you suggested...
For a moment i thought i wont get a solution for this but DSXchange rocks as usual... :D
Thanks
Pavan
Thanks for the reply. but i am waiting for an approval from my manager for Buying Premium membership. it is soo frustating that i am not able to see the answer and try this but as soon as i get it i will try the solution you suggested...
For a moment i thought i wont get a solution for this but DSXchange rocks as usual... :D
Thanks
Pavan
-
- Premium Member
- Posts: 1255
- Joined: Wed Feb 02, 2005 11:54 am
- Location: United States of America