DSXchange

Here's my latest challenge, scan a folder and make a list of the files for a particular file type. For example a make a list of all the *.dbf files in a particular folder. Because once I have that list I'm going to loop through and make an ODBC connection to that file and extract the data that I want. I will never know how many files are going to be in the folder and all I need is the file name. Bonus if I can use the date modified of the file to determine if I should add it to the list since the last run time of my job.

I know there is a Folder Stage but I haven't ever used it but I believe it takes two columns: file name, file data. Can that stage do a recursive search if I also need to scan sub folders?

Another way I thought of doing it is writing a linux script that data stage calls and have the OS do the work of making a CSV of file names for data stage to use.

Does anyone have any insight or pointers?

The Folder stage doesn't support sub-folders. Regardless, in my opinion it is best to stick with the UNIX command approach in a User Variables Activity stage feeding a Start/End Loop sequence job. Your modification time filter may need to be post-list as I don't recall a way to directly compare it to a supplied timestamp while generating the list, only if it is X days old... ala 'find' for example.

There may very well be a way, it's just not coming to me at this exact moment.

How about using a Sequential File stage with a find command as the Filter? The find command probably has all the options you'll need.

How do I execute a Unix command from within a UserVariable stage? And the unix command I would call, is it just a combination of the ls command?

You would execute your ls command in a command activity, then parse and interpret the output of the ls command in a user activity stage.