Here's my latest challenge, scan a folder and make a list of the files for a particular file type. For example a make a list of all the *.dbf files in a particular folder. Because once I have that list I'm going to loop through and make an ODBC connection to that file and extract the data that I want. I will never know how many files are going to be in the folder and all I need is the file name. Bonus if I can use the date modified of the file to determine if I should add it to the list since the last run time of my job.
I know there is a Folder Stage but I haven't ever used it but I believe it takes two columns: file name, file data. Can that stage do a recursive search if I also need to scan sub folders?
Another way I thought of doing it is writing a linux script that data stage calls and have the OS do the work of making a CSV of file names for data stage to use.
Does anyone have any insight or pointers?
How to make a list of files in a folder directory
Moderators: chulett, rschirm, roy
The Folder stage doesn't support sub-folders. Regardless, in my opinion it is best to stick with the UNIX command approach in a User Variables Activity stage feeding a Start/End Loop sequence job. Your modification time filter may need to be post-list as I don't recall a way to directly compare it to a supplied timestamp while generating the list, only if it is X days old... ala 'find' for example.
There may very well be a way, it's just not coming to me at this exact moment.![Wink :wink:](./images/smilies/icon_wink.gif)
There may very well be a way, it's just not coming to me at this exact moment.
![Wink :wink:](./images/smilies/icon_wink.gif)
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: