Performance DEV Vs QA Vs PRD

naren6876 · Post by **naren6876** » Mon Jan 16, 2012 1:07 am

Hi All,

My jobs are running pretty good in DEV and QA, but not in PRD.

One of my job is running under 15 min in DEV and QA and the same job is taking 90 min in Production.

Please let me know , what could be the main reason?. Please shed some light on it.

Thanks,
Naren.

v2kmadhav · Post by **v2kmadhav** » Mon Jan 16, 2012 6:07 am

There could be loads of reasons why this could happen.
Is it just this job that performs badly?
Do you have any other jobs that have a similar logic that run as expected. Does this job take 90mins irrespective of when you run it (peak/offpeak ?)

Could You please tell us more about your job and its design please.

qt_ky · Post by **qt_ky** » Mon Jan 16, 2012 7:52 am

Does production have more data?

PaulVL · Post by **PaulVL** » Mon Jan 16, 2012 8:51 am

2 words: Solar Flares!

Validate that at the time of execution (found within your datastage logs) if there were any Solar Flares going on. While you are in the log, write down how much data your job processed, and the time of day.

Consult your astrological maps, and compare it against the first letter of each of the other processes running in PROD at the time your job ran. Don't overlook any process names that may have been running on the Production Database servers you ran against as well. Specifically, Cancer sign related jobs hold a bad oman.

Speak with your networking guy and ask him how his day went, casually mention if the solar flares woke him up during that timeframe and if others complained too. But if he was borned under a Full Moon, then nevermind.

ray.wurlod · Post by **ray.wurlod** » Mon Jan 16, 2012 2:57 pm

Describe the hardware of the three systems. Describe the workload on the three systems - not just the DataStage workload, the total workload. Describe the connectivity to databases of the three systems. Describe the workload on those database servers in the three environments.

vmcburney · Post by **vmcburney** » Mon Jan 16, 2012 7:29 pm

Solar flares - ridiculous! It's much more likely to be the position and fullness of the moon.

I would consider installing a version of ISALite on each environment, running the diagnostic report and then comparing the results. It might show up differences between the environments. If you are on version 8 you should find ISALite on your installation disk or you can download it straight from IBM. ISALite (Information Server Support Assistant) is designed to help diagnose problems. It's likely to end up being something like a network bottleneck or a slow disk I/O or a database bottleneck or an undersized production environment (shared CPUs in a virtual environment) or any of a dozen other reasons.

PaulVL wrote:2 words: Solar Flares!

naren6876 · Post by **naren6876** » Wed Jan 18, 2012 4:27 pm

Thanks you All.

I have asked my Admin to give all the Environment info (Configuration on each environment).

I Will update you, once i have them.

Thanks,
Naren

SURA · Post by **SURA** » Wed Jan 18, 2012 5:14 pm

Please do consider, if the table is having index? Data volume already loaded into the table etc.

DS User

naren6876 · Post by **naren6876** » Thu Jan 26, 2012 11:55 pm

Hi All,

I did the

echo "`hostname` has `kstat cpu_info | grep core_id | uniq | wc -l` cores"

on DEV ,QA and PRD and the results are below

dev - 64
qa - 64
prd - 8

Thanks,
Naren

qt_ky · Post by **qt_ky** » Thu Jan 26, 2012 11:59 pm

Seriously?

Compare the memory also.

Ask your admin and IT people why... Why is prod 1/8 the size of dev and QA?

PaulVL · Post by **PaulVL** » Fri Jan 27, 2012 11:01 am

Maybe they are GRID in prod and that's just the size of the head node.

But ya... 8 is a tad small given the size of DEV and QA.

qt_ky · Post by **qt_ky** » Fri Jan 27, 2012 6:41 pm

If so, the grid must be having issues... Or they are hiding the real production server elsewhere!