FTP using a datastage job

sumitgulati · Post by **sumitgulati** » Fri Dec 17, 2004 3:56 pm

Hi All,

In our project we have a lot of jobs that read data from a sequential file. These source files are provided to us by a different team through an email OR they post them to a Shared Drive in a Windows platform. We then need to FTP these files to the DataStage host system which is an AIX box. This whole process is manual and we want to automate it. Kindly suggest what is the best way to do that? I am not sure if there is any way to connect to the Shared Drive from the AIX box using a DataStage Job.

Thanks and Regards,
-Sumit

wdudek · Post by **wdudek** » Fri Dec 17, 2004 4:06 pm

If they can post the files to a windows server running ftp, you could use datastage and the command stage to ftp the files off of the windows server and back to the unix environment. We do the opposite in our environment going from datastage on nt to Unix. You could also set up and NFS drive on the AIX server and allow them to post the files there instead of the windows server.

sumitgulati · Post by **sumitgulati** » Fri Dec 17, 2004 4:37 pm

wdudek wrote:If they can post the files to a windows server running ftp, you could use datastage and the command stage to ftp the files off of the windows server and back to the unix environment. We do the opposite in our environment going from datastage on nt to Unix. You could also set up and NFS drive on the AIX server and allow them to post the files there instead of the windows server.

Thanks for your reply. I did not get the following things:

1) Could you also please clarify on NFS drive. I am quite sure what it is?

2) How do we connect to a remote machine (Windows system) from the DataStage host machine (AIX box) using a command stage. Isn't command stage meant to run a OS command on DS host system?

I also see an FTP stage in DataStage. The help says that we can connect to a remote machine using this stage only if the remote machine has an ftp server running. Can it help?

Thanks and Regards,
-Sumit

ketfos · Post by **ketfos** » Fri Dec 17, 2004 4:56 pm

Hi Sumit,

You can get detail info on NFS from their home page.
You will need to configure both the server and the client.
basically it allows allow machines to mount a disk partition on a remote machine as if it were on a local hard drive.
it will allow users fast access, sharing files on the network.
But it has potential to allow other unwanted users to access your drive. So security is very important.

at http://nfs.sourceforge.net.

Ketfos

sumitgulati · Post by **sumitgulati** » Fri Dec 17, 2004 5:16 pm

Thanks Ketfos. What I understand by going through it is that using NFS the Source File owners should be able to post the file directly to the DataStage host machine. Now this becomes a security issue because a lot people would then be accessing the AIX box.
I would still look forward for the suggestions on getting the FTP done using a DataStage job.

Thanks and Regards,
-Sumit

ketfos · Post by **ketfos** » Fri Dec 17, 2004 5:45 pm

Hi,

You can use the following script from windows

open <ServerName>
user
<username>
<password>
Specify Target Directory Path on AIX ssytem like mentioned in next line
cd /ardent/uv/DataStage/Projects/daily
send c:\mm.txt
quit

Save this in a file name ftptest.txt

Make a bat file ftpbat.bat
ftp -n -s:\ftptest.txt

Run this bat file from DOS prompt in windows.

Ketfos

sumitgulati · Post by **sumitgulati** » Fri Dec 17, 2004 5:58 pm

Hi Ketfos, "Open" does not seem to be a recognised command in DOS. Is it required for the Windows system to host an FTP server to be able to understand this command? OR is this command recognised only when issued using a bat file?

Thanks and Regards,
-Sumit

ketfos · Post by **ketfos** » Fri Dec 17, 2004 6:04 pm

Hi,

Correct me if I have understood different.
You are using a windows OS (NT/2000/98..)
The data file is on your local disk drive or in shared drive to which you have access.

Go to the source drive by using cmd command.
At DOS prompt change to the drive where the source file is.
Say it is in O Drive.
o: <return>
At this prompt type in
ftp -n -s:\ftptest.txt ---- I assume ftptest.txt is in o: drive.

Let me know fi you run into something else or send me your txt file.

Ketfos

ketfos · Post by **ketfos** » Fri Dec 17, 2004 6:12 pm

Hi,

Open is command in ftp to open the port for ftp.
I think you are trying it at DOS prompt and getting the error.
At dos prompt type ftp
Then try open command.

Ketfos

wdudek · Post by **wdudek** » Mon Dec 20, 2004 9:18 am

You could also do a similiar script from the Unx side so that datastage fires off the ftp command to get the fiels from the windows server.

wdudek · Post by **wdudek** » Mon Dec 20, 2004 9:19 am

Also in regards to the NFS drive, the users would only be accessing the parts of the unix server that you allow them to. Give them minimal permissions so that they cannot get at anything else.

PilotBaha · Post by **PilotBaha** » Mon Dec 20, 2004 10:38 am

Sumit,
since I am familiar with your project

I am not sure if this will work with the SOX rules. As far as I remember the ftps that you are planning to do are on the different network that you will be accessing. Am I correct in that? Then protecting the data during the transport becomes an issue..

sumitgulati · Post by **sumitgulati** » Mon Dec 20, 2004 10:57 am

Baha, the files will be in the same network.

You could also do a similiar script from the Unx side so that datastage fires off the ftp command to get the fiels from the windows server.

wdudek, from the Unix machine can I open an FTP connection to any Windows machine in the network OR it is something possible only if the Windows machine has an FTP server running.

Thanks and Regards,
-Sumit

PilotBaha · Post by **PilotBaha** » Mon Dec 20, 2004 11:02 am

Sumit,
two answer your question, you can open the ftp connection from the UNIX machine to the Windows one. If the files are within the same network, then it's good. That wasn't the case when I was working on the file transfers with the other subsidiary. I guess you know what I am talking about

Good Monday morning to you and enjoy the 70 degree weather

sumitgulati · Post by **sumitgulati** » Mon Dec 20, 2004 11:10 am

Thanks Baha.

-Sumit