data transfer issue - is NFS a good solution

wxqun · Post by **wxqun** » Thu Jul 20, 2006 9:30 pm

Usually, we use FTP to transfer a datafile from DataStage server to DB2 database server and Load data locally. Now we are thinking to build a NFS on DataStage server, DB2 database server will be the NFS client. So the DB2 Load utility can be run on the database server without FTP operation. However, the NFS performance is a little bit concerned.
Is there anyone have this experience with NFS? Is this a good solution? Please share some experience. Thanks.

kcbland · Post by **kcbland** » Thu Jul 20, 2006 9:55 pm

You're just simplifying the administration. For small files performance won't be that bad, but for big files you'll see it's just like FTP. For big files, compression and high-performance ftp or scp followed by local decompression can be faster, but it's more programming.

I don't mind the compress, scp, decompress, ssh in a nice compact script trick.

ray.wurlod · Post by **ray.wurlod** » Fri Jul 21, 2006 12:16 am

If it works, it's good. It may not be best, but it is good.

wxqun · Post by **wxqun** » Sat Jul 22, 2006 2:44 pm

Thanks for all of your responds.

Yes, the NFS can simplify the data loading process management.
However, my concern is, for those large volume table loading (e.g 4G), compare "Compress --> FTP --> decompress -->DB2 Load" , how does the DB2 Load performance if the input datafile is under the NFS which is on 100M ethernet?

For the small tables, we just use DS Sequential Insert, no bother.

Thanks.

ray.wurlod · Post by **ray.wurlod** » Sat Jul 22, 2006 6:47 pm

100M ethernet will be a bottleneck. Get a dedicated network segment, get higher speed components (even fibre optic). NICs for fibre aren't expensive these days.

kcbland · Post by **kcbland** » Sun Jul 23, 2006 3:58 pm

There's nothing like a good benchmark in your specific environment.

opdas · Post by **opdas** » Mon Jul 24, 2006 5:04 am

Hi,
In our project we are using the db2 load utility from the Datastage server itself to load Data to DB2 DB, its working fine. The only thing you need is DB2 client installed on Datastage server.
Why do you need to FTP datafile to DB2 Server ?

wxqun · Post by **wxqun** » Mon Jul 24, 2006 5:51 pm

We are using FTP just to bypass the network traffic during the load operation. In our workshop, FTP + Load locally is much faster than Load from Client.

No matter what, DB2 client has to be installed on DataStage server in order to connect to DB2 database server.

Anyway, I will do a test to see how NFS doing with our current network configuration. ( we are actually using Virtual I/O, maybe even worse than 100M ethernet. That's ashamed)

opdas · Post by **opdas** » Mon Jul 24, 2006 9:13 pm

I would prefer a FTP then a load rather than NFS. If for some reason the network goes down or something wrong happens to the NFS during loading , would have an adverse impact where as a file can be FTPed again if the nework goes down in between transfering.

wxqun · Post by **wxqun** » Tue Jul 25, 2006 10:56 am

That makes sense. Thanks for your suggestion.
My concern here is mainly focus on performance.