DSXchange

Posted: **Wed Apr 09, 2008 11:12 pm**

All our jobs are running on single node.
Below are the contains of our default.apt file.
$APT_CONFIG_FILE=/tools/Ascential/DataStage/Configurations/default.apt

Code: Select all

dsadmin@etl01> more default.apt
{
        node "node1"
        {
                fastname "etl01"
                pools ""
                resource disk "/data/dw/pxresource" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
}

When i searched in the above path i have found out following files:

Code: Select all

/tools/Ascential/DataStage/Configurations

dsadmin@etl01> ls -ltr
total 20
-rwxrwxr-x   1 dsadm    dstage      3367 Jul 21  2005 tnode8.apt
-rwxrwxr-x   1 dsadm    dstage      1668 Jul 21  2005 tnode4.apt
-rwxrwxr-x   1 dsadm    dstage       862 Jul 21  2005 node4.apt
-rwxrwxr-x   1 dsadm    dstage       397 Jul 21  2005 node2.apt
-rwxrwxr-x   1 dsadm    dstage       166 Jul 21  2005 node1.apt
-rwxrwxr-x   1 dsadm    dstage       166 Jul 21  2005 default.apt

If i change the path of configuration file ($APT_CONFIG_FILE) to any other '.apt' file name will my job will run on more nodes?
If yes then which of the following .apt files i have to use to run our same jobs on four node.

Code: Select all

dsadmin@etl01> more tnode4.apt
{
        node "node1"
        {
                fastname "etl01"
                pools "" 
                resource disk "/data/dw/pxresource/N1" {pools ""}
                resource disk "/data/dw/pxresource/N2" {pools ""}
                resource disk "/data/dw/pxresource/N3" {pools ""}
                resource disk "/data/dw/pxresource/N4" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
 
node "node2"
        {
                fastname "etl01"
                pools "" 
                resource disk "/data/dw/pxresource/N2" {pools ""}
                resource disk "/data/dw/pxresource/N3" {pools ""}
                resource disk "/data/dw/pxresource/N4" {pools ""}
                resource disk "/data/dw/pxresource/N1" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
 
node "node3"
        {
                fastname "etl01"
                pools ""
                resource disk "/data/dw/pxresource/N3" {pools ""}
                resource disk "/data/dw/pxresource/N4" {pools ""}
                resource disk "/data/dw/pxresource/N1" {pools ""}
                resource disk "/data/dw/pxresource/N2" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
node "node4"
        {
                fastname "etl01"
                pools ""
                resource disk "/data/dw/pxresource/N4" {pools ""}
                resource disk "/data/dw/pxresource/N1" {pools ""}
                resource disk "/data/dw/pxresource/N2" {pools ""}
                resource disk "/data/dw/pxresource/N3" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
 
}

or

Code: Select all

dsadmin@etl01> more node4.apt
{
        node "node1"
        {
                fastname "etl01"
                pools ""
                resource disk "/data/dw/pxresource" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
 
node "node2"
        {
                fastname "etl01"
                pools ""
                resource disk "/data/dw/pxresource" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
 
node "node3"
        {
                fastname "etl01"
                pools ""
                resource disk "/data/dw/pxresource" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
node "node4"
        {
                fastname "etl01"
                pools ""
                resource disk "/data/dw/pxresource" {pools ""}
                resource scratchdisk "/data/dw/pxscratch" {pools ""}
        }
 
}

Posted: **Wed Apr 09, 2008 11:27 pm**

Both are 4 node .apt files.
Both will cause your job to run on 4 nodes.
Both have all four nodes on the same SMP machine ETL01.
The differences I see are that the first appears to have four resource subdirectories for use by each node. If N1-N4 represent separate disks then each node will first use a distinct disk and when/if it gets full start using the others in order. This, presumably, is why they are staggered as they are.
The second .apt will have all four nodes use the same resource and scratch disks as the single node .apt. The scratch area is the same in all 3 .apts.

Posted: **Thu Apr 10, 2008 12:31 am**

In case of huge volume of data
the first one will have a bit more advantage than the second one.

Posted: **Thu Apr 10, 2008 12:41 am**

Thanks for the reply!

Means at the run time if i change the value of the environment variable $APT_CONFIG_FILE from '/tools/Ascential/DataStage/Configurations/default.apt' to '/tools/Ascential/DataStage/Configurations/node4.apt', my job will run on four nodes (i have to take care of the partitioning as well )?

Will there be any impact on other jobs in the same project if i run only one job on four nodes while all other jobs are running in single node(default.apt)?

Posted: **Thu Apr 10, 2008 12:51 am**

Yes, you need to define a parameter $APT_CONFIG_FILE and at runtime you can select different configuration file.
if your jobs has any stage which usage sorting, there there are chances that the memory space may not be enough, and jobs will run little slowly

Posted: **Thu Apr 10, 2008 3:14 pm**

Do the same thing with scratch disk that you have done with disk.

Using only one directory for scratch space across all four nodes introduces a bottleneck. Use four (or more) and the same "round robin" mechanism for allocating them. Make sure you have plenty of free space for scratch.

Posted: **Fri Apr 11, 2008 5:00 am**

Thanks all for the valuable input!

DSXchange

Changing values of '.apt' in Configuration file

Changing values of '.apt' in Configuration file