retrieving a list of files in a remote ftp location
Moderators: chulett, rschirm, roy
retrieving a list of files in a remote ftp location
I have the need to retrieve a list of files names from a ftp server directory, then process some of the files found in it. I don't think that I can use a folder stage in a server job because of the need to enter a username/password. I don't think I can use a ftp stage either.
I am not familiar with parallel jobs to know if this is possible there either.
Does anyone have any suggestions?
I am not familiar with parallel jobs to know if this is possible there either.
Does anyone have any suggestions?
You're going to have to script this, as far as I know. Connect to the remote server via ftp and then issue a 'dir' command. Using DSExecute() from a routine to run the 'ftp script' will capture the filenames in a dynamic array which you can then loop through.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Craig is correct. I only want to retrieve the remote files that I need (i.e. not processed). There could be a lot of files in the ftp server directory, so I definitely don't want to get all of them.
Thank you for taking the time to respond. I guess I need to write a script to retrieve the file list.
Thank you for taking the time to respond. I guess I need to write a script to retrieve the file list.
Craig is correct. I only want to retrieve the remote files that I need (i.e. not processed). There could be a lot of files in the ftp server directory, so I definitely don't want to get all of them.
Thank you for taking the time to respond. I guess I need to write a script to retrieve the file list.
Thank you for taking the time to respond. I guess I need to write a script to retrieve the file list.
So, a snippet from a script for a job where I needed to do almost the exact same thing:
Wrapped in whatever level of error handling you feel you need. When the script is called from DSExecute() the output from the dir command will be captured in whatever you've called the Output dynamic array variable. You can then use functions like DCount to see how many files you've found and array notation and looping structures to iterate through the list.
Code: Select all
ftp -ni ${ftp_host} <<!!
user ${user} ${pwd}
cd ${destination}
dir ${pattern}
quit
!!
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Craig/Nagaraj,
Thanks for you help so far.
Part 2:
I have created my script and now have a list of files that I would like to process. I know that I can use a start loop stage in a sequencer job to loop and process all the files in my list. My question is how can I pass the file list to this stage?
Can I use a routine activity stage to call my file list script and pass the output to the startloop stage directly?
Thanks for you help so far.
Part 2:
I have created my script and now have a list of files that I would like to process. I know that I can use a start loop stage in a sequencer job to loop and process all the files in my list. My question is how can I pass the file list to this stage?
Can I use a routine activity stage to call my file list script and pass the output to the startloop stage directly?