Page 1 of 1

Performance DEV Vs QA Vs PRD

Posted: Mon Jan 16, 2012 1:07 am
by naren6876
Hi All,

My jobs are running pretty good in DEV and QA, but not in PRD.

One of my job is running under 15 min in DEV and QA and the same job is taking 90 min in Production.

Please let me know , what could be the main reason?. Please shed some light on it.


Thanks,
Naren.

Posted: Mon Jan 16, 2012 6:07 am
by v2kmadhav
There could be loads of reasons why this could happen.
Is it just this job that performs badly?
Do you have any other jobs that have a similar logic that run as expected. Does this job take 90mins irrespective of when you run it (peak/offpeak ?)

Could You please tell us more about your job and its design please.

Posted: Mon Jan 16, 2012 7:52 am
by qt_ky
Does production have more data?

Posted: Mon Jan 16, 2012 8:51 am
by PaulVL
2 words: Solar Flares!

Validate that at the time of execution (found within your datastage logs) if there were any Solar Flares going on. While you are in the log, write down how much data your job processed, and the time of day.

Consult your astrological maps, and compare it against the first letter of each of the other processes running in PROD at the time your job ran. Don't overlook any process names that may have been running on the Production Database servers you ran against as well. Specifically, Cancer sign related jobs hold a bad oman.

Speak with your networking guy and ask him how his day went, casually mention if the solar flares woke him up during that timeframe and if others complained too. But if he was borned under a Full Moon, then nevermind.

Posted: Mon Jan 16, 2012 2:57 pm
by ray.wurlod
Describe the hardware of the three systems. Describe the workload on the three systems - not just the DataStage workload, the total workload. Describe the connectivity to databases of the three systems. Describe the workload on those database servers in the three environments.

Posted: Mon Jan 16, 2012 7:29 pm
by vmcburney
Solar flares - ridiculous! It's much more likely to be the position and fullness of the moon.

I would consider installing a version of ISALite on each environment, running the diagnostic report and then comparing the results. It might show up differences between the environments. If you are on version 8 you should find ISALite on your installation disk or you can download it straight from IBM. ISALite (Information Server Support Assistant) is designed to help diagnose problems. It's likely to end up being something like a network bottleneck or a slow disk I/O or a database bottleneck or an undersized production environment (shared CPUs in a virtual environment) or any of a dozen other reasons.

PaulVL wrote:2 words: Solar Flares!

Posted: Wed Jan 18, 2012 4:27 pm
by naren6876
Thanks you All.

I have asked my Admin to give all the Environment info (Configuration on each environment).

I Will update you, once i have them.

Thanks,
Naren

Posted: Wed Jan 18, 2012 5:14 pm
by SURA
Please do consider, if the table is having index? Data volume already loaded into the table etc.

DS User

Posted: Thu Jan 26, 2012 11:55 pm
by naren6876
Hi All,

I did the

echo "`hostname` has `kstat cpu_info | grep core_id | uniq | wc -l` cores"

on DEV ,QA and PRD and the results are below

dev - 64
qa - 64
prd - 8

Thanks,
Naren

Posted: Thu Jan 26, 2012 11:59 pm
by qt_ky
Seriously?

Compare the memory also.

Ask your admin and IT people why... Why is prod 1/8 the size of dev and QA?

Posted: Fri Jan 27, 2012 11:01 am
by PaulVL
Maybe they are GRID in prod and that's just the size of the head node.

But ya... 8 is a tad small given the size of DEV and QA.

Posted: Fri Jan 27, 2012 6:41 pm
by qt_ky
If so, the grid must be having issues... Or they are hiding the real production server elsewhere!