Using Multiple URIs in FTP Enterprise stage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
Meera
Premium Member
Premium Member
Posts: 21
Joined: Mon Nov 28, 2005 8:42 pm

Using Multiple URIs in FTP Enterprise stage

Post by Meera »

Hi,
I am using multiple URI's in FTP Enterprise Stage inorder to read different files located on a single remote server. The job works fine.

The question is , if 5 different files are being accessed using 5 URI's from single remote server, does this mean that the stage will try to establish connection to the remote server five times or is it establish connection once and then read the five files?

The URI's look like:
URI1: ftp://remoteservername.domain.com/'File1'
URI2: ftp://remoteservername.domain.com/'File2'
URI3: ftp://remoteservername.domain.com/'File3'
URI4: ftp://remoteservername.domain.com/'File4'
URI5: ftp://remoteservername.domain.com/'File5'

Password and Username are just entered once since the remote server is same.

Thanks
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Why not monitor the FTP server and find out?

My guess - and it's only a guess - would be parallel simultaneous connections.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
UweOdrich
Participant
Posts: 8
Joined: Fri Oct 27, 2006 7:44 am

Post by UweOdrich »

If you set your FTP Enterprise Stage in parallel Mode, then for each File is another partition used. Each partition does it's own ftp call and establishes its own ftp connection. All connections work in parallel. In your example you will get 5 connections to the ftp server and all work parallel.

The option "force parallism" in the ftp enterprise stage changes this behaviour a bit. If you set this option, the maximal count of parallel connentions is only the count of nodes in your APT_CONFIG_FILE. That means, if you have more files in your stage than nodes in your config file, then the stage limits the count of connections to the count of nodes.
Meera
Premium Member
Premium Member
Posts: 21
Joined: Mon Nov 28, 2005 8:42 pm

Post by Meera »

I made that changes to Force Parallelism to Yes. I tried running on 1 node and 2 node configurations, and it runs. But as Ray told, I will have to look into the remote server to see how many instances of conncetions are being made.

Thank You for the solutions
Post Reply