Heavy I/O - need help with job design

[smeliot]

We have a master sequence that calls a number of sequences, all of which have five jobs. The first job creates a dataset, then the next four jobs read the dataset, and create 16 files (complex flat files) each. The last four jobs of each sub-sequence are identical, except that the constraints are hard coded (where a key = 1, 2, 3, 4, etc. all the way up to 64). The file name has the key number hard coded on the end as well (i.e., #target_dir##file_name#_1).

Obviously this is heavy on the I/O. We are having issues with resource constraints (I don't have the exact error messages because the job logs have been deleted, and the last couple of attempts to run this have failed due to source data not being found - I'm assured that this does fail with resource constraints, though), and I'm looking at the design. In my head, it would make more sense to have one write job for each sub-sequence, passing in a parameter to use in the constraint and the target file name, and then in the sequence, loop from 1 to 64 calling the job and passing the current loop number as the parameter value.

While this makes sense to me from a design perspective (less hard coding, only have to change in one place), I'm not sure if it would help or hinder in terms of I/O usage. Does anyone have any thoughts that they could share?

(and yes, I have done searches here, but haven't found anything yet that helps. I cannot get to the operating system myself on this development server (you need to see the rolling of the eyes that went with that statement), and was hoping to gain some ground before attempting to get with someone who can look at some statistics while this is running)

Thanks.

[ArndW]

I would start at the "other" end - instead of trying to run all these jobs as quickly as possible with as much parallelism as possible, I would ask "how much time will I have to do this processing?". The answer to that question might mean that you can design your top-level master sequence to each sub-sequence sequentially instead of concurrently; and those might also run the 4 job sequentially as well. The total I/O will not decrease, but be spread over more wall-clock time and thus perhaps not overload your machine.

If, on the other hand, the goal is truly to run in a minimal amount of time then you would need to look at your file systems (datasets and flat files don't necessarily need recoverable journalling file systems). Perhaps your system has local and SAN drives where the local drives might be faster. If you have multiple controllers then you can spread you datasets and flat files across them (since often controllers cache and process data in parallel) and, if all else fails, you can look at hardware.

[smeliot]

Thanks, but the issue here is not speed (although that may become an issue later), but with not aborting due to Resource Constraints. The master sequence does call the sub-sequences sequentially, and each sub-sequence calls its five jobs sequentially as well.

I will definitely ask about local vs. SAN disks, though - thanks!

[ArndW]

Normally jobs will abort because their startup time is too long and this is due to the system becoming overloaded by too many concurrent process. If all of your jobs are sequential then you have some other issue. You should post the exact error message once you reproduce the problem.

[smeliot]

It took forever to get them to let me run this <sigh>. Anyway, the jobs within the sequence that failed are some of the ones that read a dataset, and then write to 16 complex flat file stages. There are sorts on the links going into the CFFs.

all jobs are running sequentially
job that reads complex flat files and creates datasets - fine
job that creates a dataset - fine
three jobs that read the dataset, then writes to 16 cff (and sorts going in to stage) - fine
then fourth job that reads the dataset, then writes to 16 cff (with sorts), 5 of the writes have fatal errors:

cff.linkname_Sort,0: Failure during execution of operator logic.
cff.linkname_Sort,0: Fatal Error: Fork failed: Resource temporarily unavailable
and Players and SectionLeaders terminate unexpectedly
and cf: Fatal Error: waitForWriteSignal(): Premature EOF on node p19715dtw094 Socket operation on non-socket
then more Players and SectionLeaders terminate unexpectedly
and my favorite:
node_node2: Internal Error: (0): processmgr/msgport:C: 1256: FAILURE: Impossible code statement reached

Thanks in advance ...

[ArndW]

...
cff.linkname_Sort,0: Fatal Error: Fork failed: Resource temporarily
...

is the important message. This means that UNIX could not "fork" a process. Since this is integral to the operating system, it is not surprising that the jobs failed.

Either you have a low system limit on nproc or have completely overloaded physical and virtual memory. The former is much more likely. Your setting should be in the thousands and is probably 16 or 32.

[smeliot]

thank you - I knew the problem was resource constraints, just didn't know why. So, is nproc a Unix setting or a DS setting?

[ray.wurlod]

NPROC is a UNIX setting. Your UNIX system administrator will be able to help. You *may* be able to get away with increasing ulimit -n but the better long-term solution is to increase NPROC as well. Note that some UNIX implementations use a slightly different name for this parameter - it's the number of processes than any one process may start.