Unable Create Copy or Save As Job

ray.wurlod · Post by **ray.wurlod** » Sun Mar 02, 2008 11:23 pm

Search would be a good idea.

There are very few reasons why an operating system file (in this case RT_CONFIG20857) can not be created. In summary they are (a) permissions, (b) disk full, (c) disk quota exceeded or (d) limit on number of subdirectories in a directory exceeded.

Do you really have more than 20,000 jobs in one project?

my_stm · Post by **my_stm** » Mon Mar 03, 2008 9:28 pm

ray.wurlod wrote:Search would be a good idea.

There are very few reasons why an operating system file (in this case RT_CONFIG20857) can not be created. In summary they are (a) permissions, (b) disk full, (c) disk quota exceeded or (d) limit on number of subdirectories in a directory exceeded.

Do you really have more than 20,000 jobs in one project?

Thank you for your replied. we are now doing research on this issues.

PhilHibbs · Post by **PhilHibbs** » Tue Jul 28, 2009 5:32 am

I am also having this problem, when it occurred the project directory contained 32766 subdirectories. Is there any way to lift this limit on AIX?

ArndW · Post by **ArndW** » Tue Jul 28, 2009 6:28 am

The 32766 limit is UNIX one; the inode/entry table for a directory is limited to this number of files. When this happened at a customer site with lots of jobs, I did a quickie fix of resizing all the RT_STATUSnnn files from dynamic to static files. The better solution is to "calve" a new project when your old one reaches this size.

PhilHibbs · Post by **PhilHibbs** » Tue Jul 28, 2009 6:34 am

ArndW wrote:The better solution is to "calve" a new project when your old one reaches this size.

We have a ton of old jobs that we can delete. I'll have to write a little script that checks the directory count in each project so I can see this coming up. Something that does this in each one...

Code: Select all

ls -l|grep ^d|wc -l

Or, I could just look at the parent directory, that's what the number after the flags is isn't it?

chulett · Post by **chulett** » Tue Jul 28, 2009 7:55 am

The number after the flags?

PhilHibbs · Post by **PhilHibbs** » Tue Jul 28, 2009 7:57 am

Code: Select all

phibbs@tsaix60 # ls -l
total 24840
drwxrwxr-x 24257 dsadm    dstage      3239936 28 Jul 14:31 C1PDM_Dev
drwxrwxr-x 20238 dsadm    dstage      1613824 18 Jul 02:48 C1PDM_Prod
drwxrwxr-x 7259 dsadm    dstage       643072 07 Apr 13:24 C1PDM_QA
drwxrwxr-x 8459 dsadm    dstage       557056 27 May 10:01 C1PDM_QCP4400
drwxrwxr-x 15296 dsadm    dstage      1695744 14 Jul 11:51 C1PDM_QCP4401
drwxrwxr-x 5778 dsadm    dstage       483328 16 Jul 06:13 C1PDM_QCP4402
drwxrwxr-x  107 dsadm    dstage         8192 18 Jul 02:48 C1PDM_QDV8231
drwxrwxr-x 17988 dsadm    dstage      1638400 28 Jul 14:05 C1PDM_QTS4331
drwxrwxr-x  263 dsadm    dstage        16384 09 Jul 09:50 C1PDM_QTS8331
drwxrwxr-x 1325 dsadm    dstage        73728 09 Jul 09:50 C1PDM_QTS8351
drwxrwxr-x 1595 dsadm    dstage       114688 28 Jul 13:16 C1PDM_QTS8352
drwxrwxr-x 17119 dsadm    dstage      1540096 27 Jul 17:30 C1PDM_QXR4331
drwxrwxr-x 5849 dsadm    dstage       462848 28 Jul 11:09 C1PDM_QXR4352
drwxrwxr-x 6443 dsadm    dstage       446464 27 Jul 21:12 C1PDM_Scratch
drwxrwxr-x   47 dsadm    dstage         4096 01 Jun 16:19 test
phibbs@tsaix60 #

chulett · Post by **chulett** » Tue Jul 28, 2009 8:10 am

So, for example in the first line the number 24257? I'm not sure off the top of my head what it is but the man page should let you know.

PhilHibbs · Post by **PhilHibbs** » Wed Jul 29, 2009 10:13 am

DataStage appears to create 6 directories for each job in a project. This limits a project to slightly over 5400 jobs. Does that seem reasonable? Well, we're going to blow that limit by a significant margin.

I'm going to have to create a new production project, which is kind of annoying as I have a lot of scripts that will break, and will be harder to use when I fix them as I'll have to pass an extra parameter to specify which project the jobs are in.

I'm also a little disappointed that DataStage isn't aware of this limit, and didn't put up a nice "Your project directory is full, please delete any un-necessary jobs or create a new project" message.

ray.wurlod · Post by **ray.wurlod** » Wed Jul 29, 2009 4:53 pm

PhilHibbs wrote:DataStage appears to create 6 directories for each job in a project.

For server jobs they are (where nnn is the job number):
DS_TEMPnnn
RT_BPnnn
RT_BPnnn.O
RT_CONFIGnnn
RT_LOGnnn
RT_STATUSnnn

Parallel jobs also create RT_SCnnn

ArndW · Post by **ArndW** » Thu Jul 30, 2009 1:17 am

You can safely do a RESIZE of the DS_TEMPnnn, RT_CONFIGnnn and RT_STATUSnnn to static hashed files with a modulo of 1, this saves 3 subdirectories and 9 inodes per job.

PhilHibbs · Post by **PhilHibbs** » Thu Jul 30, 2009 7:18 am

Interesting. I mentioned this to my boss but he ruled it out as an option, we're going to create a new project. Oh well.