Datastage Capacity planning ...HELP
Moderators: chulett, rschirm, roy
Datastage Capacity planning ...HELP
Hi,
I have the situation like this.
I have one datastage server V7.5 EE already installed and running in sun solaris with oracle 10g as db. I have a oracle server grid with Terabytes of space and 28CPU's with both oracle server (datawarehouse) and DS Server installed. We have 4 new datamarts coming up and all going to get created in the same server. My question is
1) Do i need to install 4 datastage servers (each 1 for mart) or 1 server will be sufficient with 4 separate projects created?
2) Datawarehouse will run every 1hr and all the 4 datamart loads will run parallely twice in a day. what are the performance issues that i may face? (Data volume is huge with millions)
3) Is it possible to evenly distribute the CPU's for all the datastage projects??(ex. 4 datamart*4 cpu=16 cpu's). How can i do that and who (unix or dsadmin) can do that??
4) Any other things that i need to take care of??
5) Best possible approaches keeping this scenario?
Any inputs/suggestions will be really helpful.
Thanks in Adv
I have the situation like this.
I have one datastage server V7.5 EE already installed and running in sun solaris with oracle 10g as db. I have a oracle server grid with Terabytes of space and 28CPU's with both oracle server (datawarehouse) and DS Server installed. We have 4 new datamarts coming up and all going to get created in the same server. My question is
1) Do i need to install 4 datastage servers (each 1 for mart) or 1 server will be sufficient with 4 separate projects created?
2) Datawarehouse will run every 1hr and all the 4 datamart loads will run parallely twice in a day. what are the performance issues that i may face? (Data volume is huge with millions)
3) Is it possible to evenly distribute the CPU's for all the datastage projects??(ex. 4 datamart*4 cpu=16 cpu's). How can i do that and who (unix or dsadmin) can do that??
4) Any other things that i need to take care of??
5) Best possible approaches keeping this scenario?
Any inputs/suggestions will be really helpful.
Thanks in Adv
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
1) You can do everything on the one DataStage server (and it will certainly save you a lot in licensing fees!). If you are concerned about scalability and whether your poor little server can handle the volume of four marts then consider upgrading to enterprise edition, increasing the CPUs and RAM or using a cluster. One enterprise license will be better value then two server licenses.
2) Enterprise Edition to handle the volume. Consider materialised database views for your end users so that when your marts are being reloaded their queries can still access the old data. Do delta loads and not full loads so your twice daily loads are small in volume. Put your data marts on another server if you are doing intraday loads.
3) Enterprise Edition.
4) Do your tax returns, don't forget family birthdays, make sure you have a will, brush and floss, be nice to people.
5) Enterprise Edition.
2) Enterprise Edition to handle the volume. Consider materialised database views for your end users so that when your marts are being reloaded their queries can still access the old data. Do delta loads and not full loads so your twice daily loads are small in volume. Put your data marts on another server if you are doing intraday loads.
3) Enterprise Edition.
4) Do your tax returns, don't forget family birthdays, make sure you have a will, brush and floss, be nice to people.
5) Enterprise Edition.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
Just noticed that even though you posted in the server forum you already have an enterprise license. Because your DataStage and Oracle are using the same server you may need to scale your DataStage nodes down during the day so you don't pull resources away from online users and then ramp it up at night to process at full volumes. One good thing about EE is that you can just throw more RAM at it for higher volumes.
What is an Oracle server grid? Is that a server grid with Oracle on it? Are you using grid management software?
What is an Oracle server grid? Is that a server grid with Oracle on it? Are you using grid management software?
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Intesting idea about using the materialised views. One would assume that the refresh of the materialised views would be faster than the actual loading?vmcburney wrote: 2) Enterprise Edition to handle the volume. Consider materialised database views for your end users so that when your marts are being reloaded their queries can still access the old data. Do delta loads and not full loads so your twice daily loads are small in volume. Put your data marts on another server if you are doing intraday loads.
vmcburney wrote: 4) Do your tax returns, don't forget family birthdays, make sure you have a will, brush and floss, be nice to people.
![Laughing :lol:](./images/smilies/icon_lol.gif)
Re: Datastage Capacity planning ...HELP
Hi,
Thanks for the reply.
I have Datastage V7.5 Extended edition and not enterprise edition. What is the difference between these two?
We have a robust server and it will be extended when it's really required.
Thanks
Thanks for the reply.
I have Datastage V7.5 Extended edition and not enterprise edition. What is the difference between these two?
We have a robust server and it will be extended when it's really required.
Thanks
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
I don't know about an Extended Edition of DataStage but the Enterprise Edition comes with parallel jobs and MetaStage.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I'm not aware of anything called "extended" edition. Nor is Vincent. If you can tell us what it is, we can advise how it might differ from Enterprise edition.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
So you only have server jobs, no parallel jobs. Goes back to my initial reply, if you have a lot of CPUs or multiple servers then parallel jobs provide easier scalability. If you are staying with server edition consider running all four data mart loads at once. This should keep all your CPUs busy and avoid bottlenecks. The risk of server edition is you have a long running job utilising a small amount of CPU that other jobs are dependent on.
One of the benefits of keeping all jobs on the one server is that you will able to share hash files.
One of the benefits of keeping all jobs on the one server is that you will able to share hash files.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
So what does DSEE stand for now?!!chulett wrote:It is listed on Ascential's Products website
![Rolling Eyes :roll:](./images/smilies/icon_rolleyes.gif)
Mumble... marketing... mumble... mumble!
Is/will there be a DSEEE (extended enterprise edition)? Might that not be confused with DB2EEE?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
It gets more confusing. XE comes with something call the web services client pack, SOA comes with web services. XE can be taken as an SOA version, to give you web services to go with your web services client pack????
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn