Hello,
I have a datastage job which take 3 hours to process 30 million records and using a 4 node config file and this is a generic config file shared by all the jobs.
I want to make a new Config files specially for this job to optimise to process.
Wht information will be required to make an efficient config file -
I have made a list as per my understanding and is as below
1.Find the number of CPUs (phyical can Logical)
2.Find the memory installed on this server
3.if the CPU is Core Duo then define the Nodes in Config file as per that information.
....
...
....
Could someone please help me in writing the good config files for my job.
Writing configuration file
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 39
- Joined: Sun Apr 15, 2007 11:30 pm
Writing configuration file
"Books are as useful to a stupid person as a mirror is useful to a Blind person."
-
- Charter Member
- Posts: 193
- Joined: Tue Sep 05, 2006 8:01 pm
- Location: Australia
A different config file is only really useful is your jobs have been designed properly to make use of parallel processing. Best to start by looking at the job and see what you can do to improve things. Next you can just test your results with different config files ,say start with a 2 node config file and progressively increase that. See if your performance improves.
-
- Participant
- Posts: 39
- Joined: Sun Apr 15, 2007 11:30 pm
No.of CPU's used can be known by the Infrastructure admin.Generally no.of cpu's ~ no.of nodes.To improve performance,depending on resources available you can increase the node size in config file at the job level.Before doing this,check dump score of the job if any unnecessary sortings are present and see whether upstream or downstream is slow to change the settings.
Kiran Vaduguri
As soon as the fear approaches near, attack and destroy it.
As soon as the fear approaches near, attack and destroy it.