CPU and APT file

[wuruima]

The machine I am using, has 16 CPU
------
System configuration: lcpu=16 mem=27648MB ent=1.94
------

In the DS apt file, I set 4 nodes. like below text.
How could I make use of the 16 CPUs? Set 16 nodes? To make the job faster.

Code: Select all

{
        node "node1"
        {
                fastname "HKxxxxx"
                pools ""
                resource disk "/a/DataSets81" {pools ""}
                resource scratchdisk "/a/Scratch81" {pools ""}
        }
        node "node2"
        {
                fastname "HKxxxxx"
                pools ""
                resource disk "/a/DataSets81" {pools ""}
                resource scratchdisk "/a/Scratch81" {pools ""}
        }
        node "node3"
        {
                fastname "HKxxxxx"
                pools ""
                resource disk "/a/DataSets81" {pools ""}
                resource scratchdisk "/a/Scratch81" {pools ""}
        }
        node "node4"
        {
                fastname "HKxxxxx"
                pools ""
                resource disk "/a/DataSets81" {pools ""}
                resource scratchdisk "/a/Scratch81" {pools ""}
        }
}

[chulett]

Node != CPU. The former is a logical concept, the latter a physical object. The O/S decides how many actual CPUs to use, not you and not DataStage.

[qt_ky]

Note also that lcpu = logical cpu (not physical).

Your server may have 1 or more physical processors, each with 1 or more (likely more) processor cores, each with 1 or more logical cpu. An IBM POWER8 processor for instance can be assigned 1, 2, 4, or 8 logical cpu per core via the SMT setting.

You allocation of cores is probably much less than 16. The ent=1.94 means that your LPAR is entitled to use 1.94 processor cores at any given time. It is likely you can exceed the entitlement number as well.

Are you on AIX? If so, run the lparstat -i command to see more details (min/max values, etc.).

All that aside, you can increase or decrease the number of logical nodes your jobs use while running to find one that is optimal (at least optimal for a given job...).

[chulett]

Ack... noticed the "l" and meant to mention that as well, then got sidetracked. Thanks.

Rather than simply throw more nodes at it to 'make the job faster', my first pass would be to ensure the current job design was as optimal as possible for the task at hand. Once there, throw more workers at it - if needed.

[PaulVL]

In terms of making your jobs run faster...

This is all of course assuming that your job has a CPU constraint rather than an IO constraint.

Your environment will also be affected by the quantity of concurrent jobs.

Have you determined what is causing your job slowness?

Higher degrees of parallelism does not always equate to a faster job.
Job design is always #1, job IO to/from your data sources is typically #2.

Have you determined that it's actually a CPU issue?

[ray.wurlod]

What do you mean by "faster"?

Do not believe anything expressed as rows/sec. Search DSXchange for reasons.