Hardware requirements

[davidthree]

Hi All

I'm trying to pin down some hardware requirements for a DataStage server to run an overnight Data Migration job. We're using DataStage 5.2.2 on Solaris, and will probably run 8 parallel jobs over 8 CPUs.

The job uses several hash files, all pre-loaded to memory, but none larger than 500 kB. There is one transformer, within which there are some complex calculations. Array size on the source will be set between 100 and 500 rows.

Can anybody tell me the amount of memory (disk space and RAM) that I will need?

Ie.

Disk space: I can easily see the size of the project & hash files within our dev environment, but how much temp. space would an ideal production environment include?

RAM: How much RAM will I need? Enough to run the server, plus the size of the hash files, plus the size of 100-500 rows of data (as above)? Or are there other factors influencing memory usage?

Thanks guys

David

[Teej]

If you are working on an 8 CPU box, and you are worried about RAM...

It really depends on how much source data you have. If those are the hash files, then I doubt you would even hit 1 gigabyte in total use, and I would expect a 8 cpu SERVER to have at least 8 gb.

Heck, I have a 4 cpu box running 8 gig, and throwing around terrabytes worth of data (Feed output). Of course, we prefers to use the 8 cpu with 16 gig, but hey, it works fine (only slower) on the 4 cpu box. It's a PX job, but I suspect that server jobs are not that much more demanding than PX in term of RAM usage.

-T.J.

[kcbland]

RAM is almost never the issue, if you have a somewhat standard configuration, ie. 1 GB of RAM per cpu. Since Server jobs are extremely memory friendly, and hash caches only use what is necessary, it is difficult to predict the total memory usage because it is data dependent.

Using standard Unix tools (memstat, top, glance, iostat, etc) you can measure your utilization during runtime to guess at resource utilization. In my experience, RAM is not the problem, it's usually a deficiency in cpus and then in disks. You want to make sure that your disk arrays support the type of i/o activity DS is going to do to them. If you're using Clarions versus EMC's, you will notice a difference. Especially when you have multiple job instances reading/writing on the same filesystems.

Your worries will be more inline with getting maximum throughput off your disks, then making sure you are using every cpu cycle you can get your hands on.

[davidthree]

Thanks for those, they were very helpful. One more question however.

Kenneth - you mention

the type of i/o activity DS is going to do to them

Could you clarify the sort of activity we should expect? Would the following encompass it?

writing log files
reading / writing inter-job hash files
one-time read of pre-loaded hash files
use of temp space

Also, could you give me an indication of what use DS makes of the temp space?

Cheers

David

[kcbland]

Please clarify what you mean by "temp" space. Are you talking about /tmp, or something else? DataStage Server has two configurable paths for temporary work files. In the uvconfig file is a uvtemp variable, and in the dsenv file is a DS_MMAPPATH variable. As with any piece of software, you would want to isolate applications on their own filesystems, so as to reduce the ability of one application chewing up disk space to crash another application. That's a lesson in exposure. You should avoid defaulting to /tmp.

Now, DataStage's memory footprint is very light for jobs. The more you hash files and caching the more memory utilization. Then, multiply that by job instantiation and you're talking a lot more utilization. But again, all memory allocation will be ragged and depends on the data currently processing.

As for disk i/o, if you have a job that is reading and writing the same hash file without disk caching, then instantiating this job X 8 will cause bottlenecking on the disks around that hash file on certain configurations. There's too many variables here, are you on EMC's, or Clarions, how many controllers, are the controllers sharing disks with other applications like Oracle, blah blah blah. I like simple rules of thumb:

Try to isolate your disks and controllers from other applications so that you have dedicated resources that aren't going to get pinched when you're not looking.

Don't worry so much about log files and the internals of the DataStage repository. There's really not a lot to do about optimization. See the above quote, it can go a long way.

I spent a month at one client helping them rundown why DataStage jobs absolutely died, we're talking dropped to less that 1 row /second, whenever Oracle queries were ran on that server. It turns out somewhere there was a controller and a disk shared by the file system that DataStage's work filesystem resided. If a certain query was ran that used a certain temp tablespace, the controller overloaded and any DataStage jobs using hash lookups ground to a snails pace. When the query finished the jobs would speed up again. They were mis-diagnosing the problem, because they saw the jobs were not using much cpu, they assumed the jobs weren't getting cpu time because of load balancing or other. I had to show them that when the instance was taken down, every job that did purely isolated to the server work (jobs that read/write and reference only hash and sequential files) would run at 100% cpu utilization. I got those jobs running, and when at full steam had them start the instance. Once that instance was up, they did a simple select count(1) on a certain table. We saw the job cpu utilization drop from 100% to almost 0 during that query run. My total point is isolation. Don't sweat the other stuff.