Server and client sizing

Devyani Sarkar · Post by **Devyani Sarkar** » Thu Oct 26, 2006 12:04 am

Hi,
Generally what should be the indicative server and client configuration assuming that we have around 1 TB data flowing through?
Are there any high level sizing guidelines for Ascential Server?
Thanks,

ArndW · Post by **ArndW** » Thu Oct 26, 2006 12:30 am

The client configuration, at least the minimums, are outlined by IBM/Ascential. Sizing the server is a bit tougher. I can flow 1Tb through my notebook with no problems, except it will take quite a bit longer than on a top-end Solaris box. So part of the sizing questions must be times and not just data amounts. Is the data being processed heavily (i.e. a lot "T" from the ETL equation) or just being moved? What are the sources and the database(s)? Are they local or remote?

Many factor play into sizing and capacity planning, making correct judgement a matter of experience (and luck).

kumar_s · Post by **kumar_s** » Thu Oct 26, 2006 12:31 am

Are lookin at the version of Datastage or the size of the server?
Assuming you are after the server size and capacity, the vauge answer would be, "it depends on your jobs desing that you use". That true. More the stage, more the resource used in Datastage. I have seen some site, where, ETL is used just to trigger Oracle Stored procedure, and Extract and load directly into other server. Which may not ponentially utilize the Datastage resource. Say if you have some lookup through hashed file need to be done, and the whole file need to be palce in Datastage server (atleast for better performance).

Devyani Sarkar · Post by **Devyani Sarkar** » Thu Oct 26, 2006 1:12 am

Thanks ArndW & kumar_w,
The requirement is that the expected data volume handled to be of 1TB and the tranformation and load will be carried out on Datastage Designer. What will be your suggestion based on your experience, we can get the leads from this ?

kumar_s · Post by **kumar_s** » Thu Oct 26, 2006 1:19 am

Since the input you have, just gives the size inoformation, I again assume that you are after Disk Size.
As mentioned, it all depends on the level of transformation involed. What is your source/targer? Is it generally a daily run/weekly/Fortnightly/monthly run? Based on your high level desing, how many days you require the staging files/tables to be retained. Is it in the same server or different?
IF you dont have any of this information, just go ahead with double or triple of your input (assuming that you are not going to process all the 1TB at the same time).

ArndW · Post by **ArndW** » Thu Oct 26, 2006 1:59 am

Ok, I recommend using an old Intel 486 processor (or a Pentium if you feel like splurging) and the minimum amount of memory that the IBM/Ascential datastage guide suggests, but you could probably make do with 512Mb. This will be your cheapest solution.

(do you now understand that this is why you've been asked twice about performance This DS Server configuration will work, but your load might take over a week to complete).

This is like me asking you what sort of a car I should purchase and tell you that I have 60 boxes in my garage that I need to move.

Do you have operational constraints? Is the company a UNIX or a Windows shop The list of questions can go on quite a bit, but without any information about the general parameters you will not be able to get a good answer from anyone. Except perhaps a hardware salesman - he/she will sell you a box with pleasure.

Devyani Sarkar · Post by **Devyani Sarkar** » Thu Oct 26, 2006 4:24 am

Server will be hosted on Unix Box, and acceptable processing time should be about 1-2 hour for daily run for full load. Datastage server should be able to process about 1TB Data in 1 hour or 2 hours maximum, this includes read from sources transform and load the presentation layer.

ArndW · Post by **ArndW** » Thu Oct 26, 2006 5:35 am

Devyani,

great (that rules out me being able to sell you my old PC, though).

What UNIX hardware platform is currently used in-house; usually companies prefer homogenous hardware?

Any UNIX system can process 1TB from a sequential file to a sequential file in that time period. But if your database isn't on your DS server and you have a gigabit ntework card you will only be able to transfer just under 3Tb per hour (if nothing else is going on and everything is perfect)

chulett · Post by **chulett** » Thu Oct 26, 2006 7:38 am

All these terabytes flying around in '1 or 2 hours maximum' using the Server product? Interesting...

ray.wurlod · Post by **ray.wurlod** » Thu Oct 26, 2006 9:06 am

NVLLVS ANXIETAS

If you have multiple CPUs (or even sufficient spare CPU capacity) you can create multiple instances each of which processes a subset of the rows - partition parallelism, in short.

chulett · Post by **chulett** » Thu Oct 26, 2006 9:50 am

ray.wurlod wrote:NVLLVS ANXIETAS

Cvte...

Obviously, you can do pretty much anything with a big enough X - lever, server, whatever. Just wondering how practical it would be, that's all.