Nodes Configration strange warning

mgendy · Post by **mgendy** » Tue Oct 27, 2009 2:10 am

Hi all,

I faced a strang behavior in the configration file , which i need to understand

we are using datastage 7.5.2 on IBM AIX Server with 4 processors and 24 GB ram

i've made 8 nodes pointing to the same file system , i.e. the same node and copied it 8 times with diffrent name and i have some questions now :

1- There are warrning while checking configration file appears only on node1 the warning is

Warning: Config file checker on node node1 can only create 413 processes. Ascential Software recommends that ORCHESTRATE users be able to create at least 64 processes per logical node. This node is one of 8 on the physical node csdsp1. Therefore, ORCHESTRATE users should be able to create 512 processes on this node

although the 8 nodes refer to the same file system i.e. the datasets & scratch folders are the same for the 8 nodes , so i think that it is strang behavior to warn in one node only

2- does making 8 file systems , i.e. mount points and making each node point to different file system will improve the performance of parallel jobs ?

3- what is the difference between physical nodes and logical nodes , i really need to understand that

i tried to search dsxchange for these questions , i found really some usefull information but i need more illustrated answers for my questuions

i really appreciate your effort and support

ray.wurlod · Post by **ray.wurlod** » Tue Oct 27, 2009 9:36 am

Check the UNIX kernel limit on the number of processes any process can start. This is usually called something like NPROC.

mgendy · Post by **mgendy** » Wed Oct 28, 2009 5:02 am

ray.wurlod wrote:Check the UNIX kernel limit on the number of processes any process can start. This is usually called something like NPROC. ...

dear ray

thanks a lot for your interest , i already checkd the maximum number of processes , and i can change them , but the question is , why this warninng appears only for one node , although all the nodes are mounted to the same filesystem , and the sam limits.

kindly ray , if you could illustrate the difference between logical nodes and phisical nodes , it will be totaly appreciated

Gendy

ArndW · Post by **ArndW** » Wed Oct 28, 2009 5:58 am

Physical nodes (or, in our context, physical CPUs) is what hardware you have. The Datastage concept of logical nodes defines how manz concurrent or parallel
DataStage uses logical nodes for defining levels of parallelism. The physical layout of a machine and the number and type of logical nodes defined in an APT_CONFIG file are not related. The operating system takes care of mapping the DataStage processes onto the physically available CPUs and other resources.

While you can declare a 16 node APT_CONFIG file and run it on a 1 CPU system, it wouldn't be a very efficient use of resources. There is no simple and straightforward answer to the relationship between logical and physical nodes.

chulett · Post by **chulett** » Wed Oct 28, 2009 7:22 am

Completely agree with what Arnd posted, with one clarification in my mind.

In my context, a "physical node" would be the box, the physical server regardless of the number of CPUs on the box. You can further sub-divide a node into logical collections of resources, pools - scratch and resource disk spaces for instance, so that processes running on the same box can spread their loads across the different physical resources there.

mgendy · Post by **mgendy** » Wed Oct 28, 2009 8:10 am

unfortunatly i couldn't read these posts because i'm a basic member , but anyway many thanks for ArndW , chulett

i'll mark the topic as resolved , for premium users

Gendy