Datastage Capacity planning ...HELP

colorblue · Post by **colorblue** » Thu Sep 29, 2005 4:10 pm

Hi,

I have the situation like this.

I have one datastage server V7.5 EE already installed and running in sun solaris with oracle 10g as db. I have a oracle server grid with Terabytes of space and 28CPU's with both oracle server (datawarehouse) and DS Server installed. We have 4 new datamarts coming up and all going to get created in the same server. My question is
1) Do i need to install 4 datastage servers (each 1 for mart) or 1 server will be sufficient with 4 separate projects created?
2) Datawarehouse will run every 1hr and all the 4 datamart loads will run parallely twice in a day. what are the performance issues that i may face? (Data volume is huge with millions)
3) Is it possible to evenly distribute the CPU's for all the datastage projects??(ex. 4 datamart*4 cpu=16 cpu's). How can i do that and who (unix or dsadmin) can do that??
4) Any other things that i need to take care of??
5) Best possible approaches keeping this scenario?

Any inputs/suggestions will be really helpful.

Thanks in Adv

vmcburney · Post by **vmcburney** » Thu Sep 29, 2005 6:07 pm

1) You can do everything on the one DataStage server (and it will certainly save you a lot in licensing fees!). If you are concerned about scalability and whether your poor little server can handle the volume of four marts then consider upgrading to enterprise edition, increasing the CPUs and RAM or using a cluster. One enterprise license will be better value then two server licenses.
2) Enterprise Edition to handle the volume. Consider materialised database views for your end users so that when your marts are being reloaded their queries can still access the old data. Do delta loads and not full loads so your twice daily loads are small in volume. Put your data marts on another server if you are doing intraday loads.
3) Enterprise Edition.
4) Do your tax returns, don't forget family birthdays, make sure you have a will, brush and floss, be nice to people.
5) Enterprise Edition.

vmcburney · Post by **vmcburney** » Thu Sep 29, 2005 6:21 pm

Just noticed that even though you posted in the server forum you already have an enterprise license. Because your DataStage and Oracle are using the same server you may need to scale your DataStage nodes down during the day so you don't pull resources away from online users and then ramp it up at night to process at full volumes. One good thing about EE is that you can just throw more RAM at it for higher volumes.

What is an Oracle server grid? Is that a server grid with Oracle on it? Are you using grid management software?

tcj · Post by **tcj** » Thu Sep 29, 2005 6:33 pm

vmcburney wrote: 2) Enterprise Edition to handle the volume. Consider materialised database views for your end users so that when your marts are being reloaded their queries can still access the old data. Do delta loads and not full loads so your twice daily loads are small in volume. Put your data marts on another server if you are doing intraday loads.

Intesting idea about using the materialised views. One would assume that the refresh of the materialised views would be faster than the actual loading?

vmcburney wrote: 4) Do your tax returns, don't forget family birthdays, make sure you have a will, brush and floss, be nice to people.

colorblue · Post by **colorblue** » Mon Oct 03, 2005 10:17 am

Hi,
Thanks for the reply.

I have Datastage V7.5 Extended edition and not enterprise edition. What is the difference between these two?
We have a robust server and it will be extended when it's really required.

Thanks

vmcburney · Post by **vmcburney** » Mon Oct 03, 2005 6:11 pm

I don't know about an Extended Edition of DataStage but the Enterprise Edition comes with parallel jobs and MetaStage.

colorblue · Post by **colorblue** » Wed Oct 05, 2005 1:57 pm

[quote="vmcburney"]I don't know about an Extended Edition of DataStage but the Enterprise Edition comes with parallel jobs and MetaStage.[/quote]

I would like to really know the diff between extended and enterprise edition. If anyone provides some input on this, it will be really helpful.

Thanks

ray.wurlod · Post by **ray.wurlod** » Wed Oct 05, 2005 5:06 pm

I'm not aware of anything called "extended" edition. Nor is Vincent. If you can tell us what it is, we can advise how it might differ from Enterprise edition.

vmcburney · Post by **vmcburney** » Wed Oct 05, 2005 7:29 pm

So you only have server jobs, no parallel jobs. Goes back to my initial reply, if you have a lot of CPUs or multiple servers then parallel jobs provide easier scalability. If you are staying with server edition consider running all four data mart loads at once. This should keep all your CPUs busy and avoid bottlenecks. The risk of server edition is you have a long running job utilising a small amount of CPU that other jobs are dependent on.

One of the benefits of keeping all jobs on the one server is that you will able to share hash files.

ray.wurlod · Post by **ray.wurlod** » Wed Oct 05, 2005 8:24 pm

chulett wrote:It is listed on Ascential's Products website

So what does DSEE stand for now?!!

Mumble... marketing... mumble... mumble!

Is/will there be a DSEEE (extended enterprise edition)? Might that not be confused with DB2EEE?

chulett · Post by **chulett** » Thu Oct 06, 2005 3:29 pm

Apparently, DataStage Enterprise Edition as DataStage Extended Edition is abbreviated as 'DSXE'.

vmcburney · Post by **vmcburney** » Thu Oct 06, 2005 5:22 pm

It gets more confusing. XE comes with something call the web services client pack, SOA comes with web services. XE can be taken as an SOA version, to give you web services to go with your web services client pack????

DSXchange

Datastage Capacity planning ...HELP

Datastage Capacity planning ...HELP

Re: Datastage Capacity planning ...HELP