Just wanted to share what we have decided to go with -- any thoughts on this is much appreicated.
1) Currentley we have 2 HN's and 6 CN's 8 Core each Physcial Server with GTK and LSF 9.1.3 on 8.7
2) HN2 is acting like NFS Server were SAN is shared to rest of the complete cluster. Also HN2 also has non DATASTAGE Tools like IA/BG/MWB.
3) Running on HP UX with HP Servcice Gaurd acting as a Failover.
We have decided to go with following architecture with 11.5
1) Complete VM's on latest Blade with Intel X5 V3 1 HN 10 core and 4 CN with 18 Core each VM where each of CN VM is located on a different Physcial HOST.
2) Total 5 Physcial blades will have one VM each 1HN and 4 CN on each with RHEL.
3) DB will XMETA on a different Soalris VM's
4) The above all is completey DATASTAGE ALONE.
5) For IGC TOOLS we will create totally different VM's with only those tools they will share the same physcial host as HEADNODE VM and they will also be VM on RHEL.
Storage wise we debatting should we go with NAS or SAN. NAS is being suggested there is a need for NFS we can use NAS Applicance for that.
Also we are not builing any active passive as Our folks from VMWARE Team suggesting if a HN VM goes down they will fire up a new one within a very short time.
Also we have 15k jobs running every time just to get an idea of the load.
-
Questions - 1) NAS or SAN in this architecture is well suited
2) IS VMWARE Failover reailable from Infosphere Grid context
3) What are the best practices for directlories Isolcation projects/datasets to get better I/O should they all be in a different physcial drives.. etc/
Really appreciate any thoughts overall.
Thanks
Sri
Migration 8.7 Grid Physcial HP UX to 11.5 Grid VM's RHEL.
Moderators: chulett, rschirm, roy
You don't "fail over" compute nodes, just the Head Node and Domain (WAS) if that is on a separate box.
I prefer NAS mounts to share that data across hosts. Clustered file systems become a pain if the controlling host takes a nose dive. File system integrity tends to kick in and the mounts get auto unmounted from the other boxes. Had that happen one to many times. I will never recommend it again.
IF a VM Compute node gets assigned to a new physical host, your Platform LSF software will get jacked up. You will have to do a badmin reconfig I believe to resolve.
I would recommend caping your grid submittion to hosts with less than 85% cpu utilization. That is defined in your queue settings.
my preferred setting is this:
RES_REQ = "span[ptile=1] rusage[ut=0.1:duration=1]"
ut = 0.85
JOB_ACCEPT_INTERVAL = 1
helps create an even round robin job distribution while capping the box at 85% (then it will not target the box for another job until it's below that level).
As far as best practice for project allocation...
I prefer to put each project on its own mount. That way they can't corrupt the other guy. Never put a project under the default install path. $DSHOME mount should never contain customer data or jobs. If they fill it up, they run the risk of corrupting projects. very bad day for the admins.
scratch should be local san for speed reasons. IBM will also recommend that.
TMPDIR should also be set to something (scratch path is ok), because otherwise it will default to /tmp and that ain't good either.
Your datastage admins will recommend your best course of action in terms of setting up the application and mount space required.
They are probably already grumbling about having to put them on VMs but got outvoted because who ever listens to admins anyways.
I prefer NAS mounts to share that data across hosts. Clustered file systems become a pain if the controlling host takes a nose dive. File system integrity tends to kick in and the mounts get auto unmounted from the other boxes. Had that happen one to many times. I will never recommend it again.
IF a VM Compute node gets assigned to a new physical host, your Platform LSF software will get jacked up. You will have to do a badmin reconfig I believe to resolve.
I would recommend caping your grid submittion to hosts with less than 85% cpu utilization. That is defined in your queue settings.
my preferred setting is this:
RES_REQ = "span[ptile=1] rusage[ut=0.1:duration=1]"
ut = 0.85
JOB_ACCEPT_INTERVAL = 1
helps create an even round robin job distribution while capping the box at 85% (then it will not target the box for another job until it's below that level).
As far as best practice for project allocation...
I prefer to put each project on its own mount. That way they can't corrupt the other guy. Never put a project under the default install path. $DSHOME mount should never contain customer data or jobs. If they fill it up, they run the risk of corrupting projects. very bad day for the admins.
scratch should be local san for speed reasons. IBM will also recommend that.
TMPDIR should also be set to something (scratch path is ok), because otherwise it will default to /tmp and that ain't good either.
Your datastage admins will recommend your best course of action in terms of setting up the application and mount space required.
They are probably already grumbling about having to put them on VMs but got outvoted because who ever listens to admins anyways.
Thanks Paul.
Some comments - Based on your feedback
1) There is no Failover at all for HN or CN they would just fireup on a different host if one of them goes down. (HN has Engine and WAS).
2) We are going with NAS rather than SAN as we dont have any GPFS or GFS we would use NAS with NFS.
3) if we had to badmin reconfig when one compute node goes down then do we need stop all jobs in that case till the compute node comes up on a different host.
4) What if HN goes down what will happen to all SSH Setups when the HN comes back in a Different Physical hoST.
5) Why do you suggest to cap and also will jobs wait by any chance with the above setting. we have many queues we tried this before but it didnt work may be syntax is wrong.
I am the lead admin you are right who listens to us
Thanks
Sri
Some comments - Based on your feedback
1) There is no Failover at all for HN or CN they would just fireup on a different host if one of them goes down. (HN has Engine and WAS).
2) We are going with NAS rather than SAN as we dont have any GPFS or GFS we would use NAS with NFS.
3) if we had to badmin reconfig when one compute node goes down then do we need stop all jobs in that case till the compute node comes up on a different host.
4) What if HN goes down what will happen to all SSH Setups when the HN comes back in a Different Physical hoST.
5) Why do you suggest to cap and also will jobs wait by any chance with the above setting. we have many queues we tried this before but it didnt work may be syntax is wrong.
I am the lead admin you are right who listens to us
Thanks
Sri
Praveen
Grid or not...
If a head node goes down, all jobs abort.
If a compute node goes down, all jobs running on the host abort.
If a compute node get reassigned to a different physical host, it's IP address might/will change. That causes a problem within Platform LSF. It caches IP address, not hostname. So a badmin reconfig is to refresh that internal cache.
SSH keys are hostname based, unless someone reached out to a host via IP address, then that IP address is used in the "knownhosts" file and that will cause a prompt when you try to use the ssh/sftp connection. Spinning up the virtual again will not impact the ssh keys since you are using the same hostname.
I suggest to cap job submission to the grid compute nodes because you do not want to oversaturate a compute node with to much CPU work. It's a grid and it should load balance... but sending work to a box that is already full is suicide. So... wait till it empties below 85% (rough number pulled out of a hat).
85% will also accommodate existing jobs to stretch during normal execution. As it's not a consistent CPU consumption during the execution of a particular job.
If a head node goes down, all jobs abort.
If a compute node goes down, all jobs running on the host abort.
If a compute node get reassigned to a different physical host, it's IP address might/will change. That causes a problem within Platform LSF. It caches IP address, not hostname. So a badmin reconfig is to refresh that internal cache.
SSH keys are hostname based, unless someone reached out to a host via IP address, then that IP address is used in the "knownhosts" file and that will cause a prompt when you try to use the ssh/sftp connection. Spinning up the virtual again will not impact the ssh keys since you are using the same hostname.
I suggest to cap job submission to the grid compute nodes because you do not want to oversaturate a compute node with to much CPU work. It's a grid and it should load balance... but sending work to a box that is already full is suicide. So... wait till it empties below 85% (rough number pulled out of a hat).
85% will also accommodate existing jobs to stretch during normal execution. As it's not a consistent CPU consumption during the execution of a particular job.