Maximum No of jobs in a Project

pankajg · Post by **pankajg** » Mon Oct 30, 2006 5:05 am

Can someone share if there is a limitation on the maximum number of jobs that can be supported by a single project in DataStage.

Also incase there is No limit on the number of jobs, I would like to know what are the factor that could limit the no of jobs in a project?

I was suggested by IBM support that it would be nice to have 1000-1200 jobs in each project.

Waiting to hear from people.

Thanks in advance.

regards
P

Kirtikumar · Post by **Kirtikumar** » Mon Oct 30, 2006 5:51 am

In one of our project, the number of jobs where more than 1000 and the problem we faced was the turn around time when u open the job or see the logs or try to access the repository thro' DS manager was very high.

chulett · Post by **chulett** » Mon Oct 30, 2006 5:58 am

Exactly. I've seen Projects with three times that number of jobs or more. They can become unwieldy and slow to open in the client tools, depending on how they are organized - but proper use of categories can help alot in that regard.

I believe you'll find a physical limit on an AIX server, but that's an AIX thing only and involves a limit on the number of entries in a directory from what I recall. Other than that, it's pretty much limited by your threshold of pain.

ray.wurlod · Post by **ray.wurlod** » Mon Oct 30, 2006 7:42 am

There is a limit imposed at the operating system level, but no practical limit imposed within DataStage itself.

The operating system limit is the number of subdirectories in a directory; each job requires six or seven subdirectories in the project directory. This is often an issue in the Version Control project, where multiple versions of jobs are being stored.

Since they're never edited severely in the Version Control project, there is a workaround that involves changing the dynamic hashed files to static hashed files (these are not implemented as subdirectories). But not such a good idea in an active project, methinks.

kcbland · Post by **kcbland** » Mon Oct 30, 2006 8:48 am

The most obvious issue is with releases 7.5 down. The internal repository stores all job design information in a single hashed file called DS_JOBOBJECTS. When you have too much design information, this file exceeds the default 32BIT limitation of 2.2GB.

You will corrupt your repository if this happens, so it's generally recommended to limit the amout of design information in a given project. You could have the DeathStar job design methods that quickly run you to the limit for just a few hundred jobs, or you could have a lot of small jobs letting you get thousands of jobs in the project.

Your solutions are either to divide your jobs across multiple projects (recommended) or adjust the UVCONFIG file to set hashed files to default to 64BIT, then create new projects which will then be 64BIT sized repository files.

Search this forum to learn about more of both solutions up and down sides.

ray.wurlod · Post by **ray.wurlod** » Mon Oct 30, 2006 9:23 am

Oh, yuk. No reason not simply to RESIZE the DS_JOBOBJECTS hashed file to 64-bit addressing. You don't want to make the default 64-bit for all hashed files!

kcbland · Post by **kcbland** » Mon Oct 30, 2006 10:35 am

I tend to avoid custom tweaks on a project level. Those are the things always forgotten during project setup. Then, you corrupt your production project during a simple job import because DS_JOBOBJECTS corrupts. I prefer something more along the lines of the engine install and configuration, probably done once every 2 years. It seems to be a more safe approach.

You really don't have many good choices, either limit your amount of jobs in a give project (good idea), manually resize each project when you create it (probably a good idea), or set the engine default (easy to do, but has many ramifications).

Note I recommended to the poster to do some research, where they could discover all of the existing discussion, rather than re-hash it all again on this post.

ray.wurlod · Post by **ray.wurlod** » Mon Oct 30, 2006 3:00 pm

Rehash!

pankajg · Post by **pankajg** » Mon Oct 30, 2006 9:59 pm

All the replies have been extremely informative.

Thanks for sharing all the information. Frankly, I never knew about the 32 bit size limit imposed on the DS_JOBOBJECT.

Thanks again

pankajg · Post by **pankajg** » Wed Nov 01, 2006 9:30 pm

Is the 32-bit default size a Datastage setting or is it at the OS level??

pankajg · Post by **pankajg** » Wed Nov 01, 2006 9:34 pm

pankajg wrote:Is the 32-bit default size a Datastage setting or is it at the OS level??

Stupid me.. Should have read this

adjust the UVCONFIG file to set hashed files to default to 64BIT, then create new projects which will then be 64BIT sized repository files.

believe 32 bit default setting is the datastage setting.