It depends to some extent upon what flavour of UNIX you are on, and on the limit (if any) on the number of subdirectories that can be created in a directory.
1000 jobs is probably OK - you can go up to somewhere near 4000. However, performance problems start kicking in in areas like job startup and job creation. They take longer than in a project with fewer jobs.
Another risk is the total volume of entries in job logs, which are all stored within the project directory - the danger is that of filling the disk that contains the project directory. Splitting the project into multiple projects on the same disk will not ameliorate this situation; splitting into multiple projects on separate disks definitely will.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
PhilHibbs wrote:And each job requires 6 directories, which gives the ~5000 job limit mentioned earlier.
What are those 6?
i remember RTSTATUS,RTCONFIG and RTLOG which is a hashed file.
Correct me if i am wrong.
And what will happen if we exceed the maximum limit?(While attempting to create the 5001th job if max limit is 5000)
Er, it's possible that I got it wrong and that it's only 5 directories. Or, there may be a sixth possibility that doesn't always get created for all jobs. I'll see if I can dig up the thread where I learned about all this.
PhilHibbs wrote:Er, it's possible that I got it wrong and that it's only 5 directories. Or, there may be a sixth possibility that doesn't always get created for all jobs. I'll see if I can dig up the thread where I learned about all this.
I am really curious to know in which scenario, the sixth will create?
if you get any link, please share with me.
Thanks
Good info Ray!! generally it varies from 5 to 8.
From my understanding RT_SCnnn will be created only if the job conatins any shared containers(SC).RT_QSnnn will be created only if datastage jobs are integrated with Qualitystage.
Correct me if my understanding is incorrect.
Could you elaborate little bit about DS_TEMPnnn?Under which circumstances DS_TEMPnnn will be created?
Thanks