How does DS use /tmp?

[rleishman]

We are approaching implementation of our DW in a few months, and have just been advised by the hardware mob that their standard Linux build includes a 500Mb /tmp.
Our Unix SysAdmin tells us that is pretty small for a DW, and I'm inclined to agree. Normally I would just ask for 10Gb or so (disk is cheap, right), but this site is an enormous beurocracy and pushing back will create possibly needless headaches.

Thinking about it, I'm not sure that we'll need much /tmp.

- We do all sorts on the database server (different box)
- We don't try to suck entire files into memory in unix/perl scripts
- We don't use Aggregator/Sorter stages. We only use, OCI, transformer, hash, seq file, link coll.
- All hash files and temp storage files are put onto dedicated filesystem
- We do cache hash file, but they are all <5Mb; usually a few Kb.

The incoming data files will be medium-sized (<50Mb) and no more than 3 transforms using this quantity of data would ever be running concurrently.

My question:
- What else does DS use /tmp for? Caching/paging?
- The $TMPDIR environment variable says it is for Parallel jobs. Is that completely true?
- Given the above outline, would I be mad to let them give me a /tmp of this size.

[kcbland]

In the uvconfig file there is a temp variable that defaults to /tmp. You could consider mapping this to a different directory if you wish. There's also the variables in the dsenv.

Server's pretty negligible on /tmp usage. You'll probably be fine.

[ray.wurlod]

DataStage uses the directory mention in the UVTEMP parameter in the uvconfig file (rather than the directory referred to in $TMPDIR) for its temporary storage for server jobs. Its default value is /tmp. UniVerse best practice is to move this to somewhere else with lots more space, particularly on Solaris (which comes with a small /tmp and also mounts swap there).

[rleishman]

Perfect. Thanks Ken, thanks Ray.

[ray.wurlod]

The UVTEMP directory gets used by the Engine for:

- temporary files when compiling sica constraints ('compconXXXX') as part of DataStage SQL security
- temporary files used for sorting (partitions files that are then merged) 'sort'XXX')
- temporary files for EXECUTE...CAPTURING ('captureXXX') which includes calls to ExecSH, ExecDOS, ExecUV and DSExecute()
- temporary files for DataStage SQL SELECT statements ('selectXXXX')
- temporary files for BASIC compiler ('bascXXX')
- temporary files used by hash.help ('hhelpXXX')
- temporary files used by hash.test ('htestXXX')
- temporary files used by init.catalog ('uvXXXX') for example when clearing the BASIC Catalog
- temporary files used for Btree navigation ('btreeXXX') to resolve a location in a secondary index
- temporary files used by sorter to communicate to it ('selectXXXnotify')
- temporary files used by SDML (SQL) queries ('sdmlXXX')
- temporary files when using DATA statements ('DATAXXX' ) in BASIC programs and/or in paragraphs
- temporary files used by 'verify' function ("verifyXXX") in particular the VERIFY.SQL command
- temporary files used by MASTER command ('masterXX')

Adapted from a post by Dave Meeks on the Oliver UniVerse forum.

[chulett]

I've also seen files in there that seem to have been used by IPC stages. Plus I always redirect the temporary files used by the Sort stage into /tmp rather than let them default into and build up in the Project directory.

[dls]

[quote="chulett"]I've also seen files in there that seem to have been used by IPC stages.[/quote]

Craig, your post coincides with a problem we are experiencing.

Usage related to IPC stage seems to be conditional. I'll continue to investigate, unless someone knows under what condition(s) IPC uses this directory.

[Bala R]

Hi dls,
As I have always seen IPC using /tmp can you please tell how you could reason out that its conditional?

Thanks.

[cnguyen]

Does anyone know which DataStage process create /tmp/man???? files? We have changed the UVSPOOL, UVTEMP to point to a different directory. We have also changed the TMPDIR parameter for all projects. However, there we still see some man??? files in the /tmp directory. We appreciate any help.

Thanks

[cnguyen]

Please ignore this post. the /tmp/mam???? files were actually Unix temp files which were generated by the man command in Unix. They are not DataStage temp files.

Thanks

Does anyone know which DataStage process create /tmp/man???? files? We have changed the UVSPOOL, UVTEMP to point to a different directory. We have also changed the TMPDIR parameter for all projects. However, there we still see some man??? files in the /tmp directory. We appreciate any help.

Thanks

[chulett]

Good. I looked when you first posted this but couldn't find any on my server, so waited for others to chime in.