File FTP in Datastage vs UNIX
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 38
- Joined: Tue Jun 19, 2012 11:03 pm
- Location: India
File FTP in Datastage vs UNIX
Hi,
I want to FTP a file from Datastage UNIX server to a different server.
Will the Datastage FTP stage be better than UNIX FTP command?
Also, what kind of configuration is needed between the 2 servers to enable FTP?
I want to FTP a file from Datastage UNIX server to a different server.
Will the Datastage FTP stage be better than UNIX FTP command?
Also, what kind of configuration is needed between the 2 servers to enable FTP?
-
- Premium Member
- Posts: 38
- Joined: Tue Jun 19, 2012 11:03 pm
- Location: India
The main advantage to using an FTP stage is eliminating the need to land a physical file on the server. Performance is usually the trade-off, with disk space being less expensive that CPU components vs. an essential doubling of the I/O -- once to land the file, another to FTP it.
For Unix-to-Unix, if you are using sshv2, the better choice in my experience is secure copy (scp) from a script.
For Unix-to-Unix, if you are using sshv2, the better choice in my experience is secure copy (scp) from a script.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
-
- Premium Member
- Posts: 38
- Joined: Tue Jun 19, 2012 11:03 pm
- Location: India
If you use DataStage FTP stage, the remote file (which is NOT on your server - it is somewhere else), is used as a data source and read one record at a time. It can then be processed (albeit slowly) and sent to its destination, which may be a database or file location on another server.
Hence - "it isn't landed"...
Though DataStage FTP is slow (per-record transfers) it will at least notify you if the connection fails. One thing that most people don't realize is that almost all "freebie" FTP commands on UNIX servers do not report error codes.(they always return "0" after execution). This means you might get a partial file and not know it.
That's why I usually design anything that uses UNIX FTP commands to validate the size of the file that was received, insuring "we got it all".
Hence - "it isn't landed"...
Though DataStage FTP is slow (per-record transfers) it will at least notify you if the connection fails. One thing that most people don't realize is that almost all "freebie" FTP commands on UNIX servers do not report error codes.(they always return "0" after execution). This means you might get a partial file and not know it.
That's why I usually design anything that uses UNIX FTP commands to validate the size of the file that was received, insuring "we got it all".
Yes, what Andy said, with this:
FTP "get" as first stage streams the transferred data directly to the output link of the stage. No file is created.
FTP "put" as final stage takes rows from the input link and transfers them directly to the destination server. No file is created on the local server.
FTP "get" as first stage streams the transferred data directly to the output link of the stage. No file is created.
FTP "put" as final stage takes rows from the input link and transfers them directly to the destination server. No file is created on the local server.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872