Reading Multiple Files on a File Server

Pavan_Yelugula · Post by **Pavan_Yelugula** » Tue Dec 26, 2006 4:00 pm

Hi
I have a requirement where i need to run a particular job picking files which are kept in a file server. The datastage is on a different machine.The Number of files on the file server might vary day to day i can get one file today and 2 files tomorrow and so on. I need to run the job the number of times as the number of files on that server. I need to automate this.
In between is there a FTP stage which i can use for getting these files on file server to my datastage server? Or is a shell script al i have for getting these files?
But the main Problem is i need to first get a count of number of files on the file server to loop the shell script and the job...
Any ideas to do this will be really Helpful

Thanks
Pavan

I_Server_Whale · Post by **I_Server_Whale** » Tue Dec 26, 2006 4:05 pm

You can use

Code: Select all


find . -type f | wc -l

to count the number of files.

Whale.

narasimha · Post by **narasimha** » Tue Dec 26, 2006 5:04 pm

If your requirement was to get the count of files to loop in your script, then take Whale's code (But you need to be careful if you have subdirectories. This command gives you the count of files inside them too)

If you are looking for an alternate approach-

You might have to write an shell script to FTP all the the file/files from the file server to your datastage server.
You don't have to know the number of files, you can use wildcards to match your search criteria.
Note: Limitation with FTP Stage - it is not possible to use wildcards.

Now that you have all the files on the datastage server, as per your requirement, you can utilize the techinque of your choice (Loops, script,..,..) to read all these files.

kumar_s · Post by **kumar_s** » Tue Dec 26, 2006 5:28 pm

To get the count and the list of file names with some pattern, you can use ls in that directory coupled with wc as well. Use sed or awk to fetch the file name is the required pattern.

ray.wurlod · Post by **ray.wurlod** » Tue Dec 26, 2006 6:04 pm

... or even a filter in the find command itself, both for level searched and for file name(s). If you can see the directory to use the find command, then you can see the directory to use a Sequential File stage, and should not need to be concerned about using FTP.

Pavan_Yelugula · Post by **Pavan_Yelugula** » Tue Dec 26, 2006 6:54 pm

Hi All
Thanks a lot for all your replies....

I_Server_Whale · Post by **I_Server_Whale** » Tue Dec 26, 2006 8:01 pm

Please mark the topic as resolved if you think the issue has been solved.

Pavan_Yelugula · Post by **Pavan_Yelugula** » Tue Jan 02, 2007 8:47 am

Hi All
I need to achieve the same functionality on another data stage Instance which is a Server Canvass and ported on Windows Environment.
Any Ideas on how to do this....

Thanks
Pavan

ray.wurlod · Post by **ray.wurlod** » Tue Jan 02, 2007 1:30 pm

Is the file server UNIX or Windows? How do you "see" the file server from the DataStage Windows-based server?

Pavan_Yelugula · Post by **Pavan_Yelugula** » Tue Jan 02, 2007 3:39 pm

Hi Ray
The File Server is also windows and i have access to the windows file server from my datastage windows server. I can ping that file server from my datastage instance and i can access the directory and the file folder in which files are kept. Sorry if i am a bit cryptic i didn't get what you meant by "see" the file server.

Is there any commands like how Whale as suggested find . -type f | wc -l in windows environment for getting the count. Does the FTP commands work in my script for the windows environment

Thanks
Pavan

ray.wurlod · Post by **ray.wurlod** » Tue Jan 02, 2007 5:21 pm

Can you "see" the directory (folder) on the file server as a shared folder? If so, you won't need FTP.

DOS does not have an equivalent of find, but DataStage does. It's called uvwalk and can traverse a tree structure if required. However, DOS also lacks a wc command, and uvwalk does not report the number of files found. So you may be better off executing a DIR command (with appropriate wildcard) and parsing out the "n File(s)" message near the end of the output.

Pavan_Yelugula · Post by **Pavan_Yelugula** » Wed Jan 03, 2007 11:56 am

Ray,
Thanks for the reply. but i am waiting for an approval from my manager for Buying Premium membership. it is soo frustating that i am not able to see the answer and try this

but as soon as i get it i will try the solution you suggested...
For a moment i thought i wont get a solution for this but DSXchange rocks as usual... :D

Thanks
Pavan

I_Server_Whale · Post by **I_Server_Whale** » Wed Jan 03, 2007 12:25 pm

Take a look at these GOOGLE results. They look promising.

Whale.