Page 1 of 1
Regarding ftp stage,want to ftp files from remote server
Posted: Thu May 31, 2007 11:31 pm
by shaonli
Hi,
I need to design a job which will import all XML files in a directory of a remote server and put it into a specific path of a DS server.
Then a job will run which will process all the xml files.
How to import all the XML files(names are different) to DS server.
I looked into ftp stage.But I have not found the option to mention the path of DS server.
Please suggest.
Posted: Fri Jun 01, 2007 5:03 am
by ray.wurlod
Create a script on the DataStage server. If the file names conform to a pattern you can use MGET.
Don't use the FTP stage; it tries to transfer a line at a time, and the concept of "line" in XML documents is rather vague.
Posted: Fri Jun 01, 2007 6:39 am
by chulett
They are typically one very long line and thus not at all suitable for the FTP Stage. Stick with a scripted solution.
Posted: Fri Jun 01, 2007 8:50 am
by VCInDSX
chulett wrote:They are typically one very long line and thus not at all suitable for the FTP Stage. Stick with a scripted solution. ...
Hi Craig, Ray,
As I too am interested in an FTP job, to upload a compressed file from the DS Server to an FTP site, I was wondering how one should plan the error handling from the scripting job.
If a script at the OS level (.bat or .cmd in Windows, e.g.) is launched by an ExecuteSH or ExecuteCommand, how much information can be grabbed from the FTP script upon its completion, especially in case of different types of failures (FTP Permission, Invalid Accounts, interrupted transfer et al...)?
I would appreciate your insight from your odysseys as you both are very experienced in the various aspects of Datastage.
Thanks in advance,
Posted: Fri Jun 01, 2007 9:17 am
by chulett
Sorry, but I can't really speak to a .bat file solution except in generic terms. Scripts should be as robust as possible. Assume nothing. Check as much of the output as you possibly can.
From a UNIX standpoint, at the very least we capture the $? status and alarm when that is non-zero. For ftp, which we typically control in a 'here document' stype, we 'tee' off a copy of any output to a temp file. Post ftp one could grep the temp file for errors of any kind. Snippet:
Code: Select all
ftp -ni your.server.com <<!! | tee /tmp/xx$$.out
user userid password
asc
cd wherever
get whatever
quit
!!
Hope that helps.
Posted: Fri Jun 01, 2007 10:49 am
by VCInDSX
Hi Craig,
Thanks for the details. Appreciate it much.
I guess i need to work a bit more on handling these log files when things don't go well with the FTP.
Our DS job has to notify a support team (email notification) with enough details about the job's status after the FTP is executed.
I should, either provide the log information in a separate file and leave it to the support team to review them, or try to parse the error and then load them into the email and/or DS Log.
Need to think a little more which way to go....
Thanks for your time,