How does DS use /tmp?
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 252
- Joined: Mon Sep 19, 2005 10:28 pm
- Location: Melbourne, Australia
- Contact:
How does DS use /tmp?
We are approaching implementation of our DW in a few months, and have just been advised by the hardware mob that their standard Linux build includes a 500Mb /tmp.
Our Unix SysAdmin tells us that is pretty small for a DW, and I'm inclined to agree. Normally I would just ask for 10Gb or so (disk is cheap, right), but this site is an enormous beurocracy and pushing back will create possibly needless headaches.
Thinking about it, I'm not sure that we'll need much /tmp.
- We do all sorts on the database server (different box)
- We don't try to suck entire files into memory in unix/perl scripts
- We don't use Aggregator/Sorter stages. We only use, OCI, transformer, hash, seq file, link coll.
- All hash files and temp storage files are put onto dedicated filesystem
- We do cache hash file, but they are all <5Mb; usually a few Kb.
The incoming data files will be medium-sized (<50Mb) and no more than 3 transforms using this quantity of data would ever be running concurrently.
My question:
- What else does DS use /tmp for? Caching/paging?
- The $TMPDIR environment variable says it is for Parallel jobs. Is that completely true?
- Given the above outline, would I be mad to let them give me a /tmp of this size.
Our Unix SysAdmin tells us that is pretty small for a DW, and I'm inclined to agree. Normally I would just ask for 10Gb or so (disk is cheap, right), but this site is an enormous beurocracy and pushing back will create possibly needless headaches.
Thinking about it, I'm not sure that we'll need much /tmp.
- We do all sorts on the database server (different box)
- We don't try to suck entire files into memory in unix/perl scripts
- We don't use Aggregator/Sorter stages. We only use, OCI, transformer, hash, seq file, link coll.
- All hash files and temp storage files are put onto dedicated filesystem
- We do cache hash file, but they are all <5Mb; usually a few Kb.
The incoming data files will be medium-sized (<50Mb) and no more than 3 transforms using this quantity of data would ever be running concurrently.
My question:
- What else does DS use /tmp for? Caching/paging?
- The $TMPDIR environment variable says it is for Parallel jobs. Is that completely true?
- Given the above outline, would I be mad to let them give me a /tmp of this size.
Ross Leishman
In the uvconfig file there is a temp variable that defaults to /tmp. You could consider mapping this to a different directory if you wish. There's also the variables in the dsenv.
Server's pretty negligible on /tmp usage. You'll probably be fine.
Server's pretty negligible on /tmp usage. You'll probably be fine.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
DataStage uses the directory mention in the UVTEMP parameter in the uvconfig file (rather than the directory referred to in $TMPDIR) for its temporary storage for server jobs. Its default value is /tmp. UniVerse best practice is to move this to somewhere else with lots more space, particularly on Solaris (which comes with a small /tmp and also mounts swap there).
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The UVTEMP directory gets used by the Engine for:
- temporary files when compiling sica constraints ('compconXXXX') as part of DataStage SQL security
- temporary files used for sorting (partitions files that are then merged) 'sort'XXX')
- temporary files for EXECUTE...CAPTURING ('captureXXX') which includes calls to ExecSH, ExecDOS, ExecUV and DSExecute()
- temporary files for DataStage SQL SELECT statements ('selectXXXX')
- temporary files for BASIC compiler ('bascXXX')
- temporary files used by hash.help ('hhelpXXX')
- temporary files used by hash.test ('htestXXX')
- temporary files used by init.catalog ('uvXXXX') for example when clearing the BASIC Catalog
- temporary files used for Btree navigation ('btreeXXX') to resolve a location in a secondary index
- temporary files used by sorter to communicate to it ('selectXXXnotify')
- temporary files used by SDML (SQL) queries ('sdmlXXX')
- temporary files when using DATA statements ('DATAXXX' ) in BASIC programs and/or in paragraphs
- temporary files used by 'verify' function ("verifyXXX") in particular the VERIFY.SQL command
- temporary files used by MASTER command ('masterXX')
Adapted from a post by Dave Meeks on the Oliver UniVerse forum.
- temporary files when compiling sica constraints ('compconXXXX') as part of DataStage SQL security
- temporary files used for sorting (partitions files that are then merged) 'sort'XXX')
- temporary files for EXECUTE...CAPTURING ('captureXXX') which includes calls to ExecSH, ExecDOS, ExecUV and DSExecute()
- temporary files for DataStage SQL SELECT statements ('selectXXXX')
- temporary files for BASIC compiler ('bascXXX')
- temporary files used by hash.help ('hhelpXXX')
- temporary files used by hash.test ('htestXXX')
- temporary files used by init.catalog ('uvXXXX') for example when clearing the BASIC Catalog
- temporary files used for Btree navigation ('btreeXXX') to resolve a location in a secondary index
- temporary files used by sorter to communicate to it ('selectXXXnotify')
- temporary files used by SDML (SQL) queries ('sdmlXXX')
- temporary files when using DATA statements ('DATAXXX' ) in BASIC programs and/or in paragraphs
- temporary files used by 'verify' function ("verifyXXX") in particular the VERIFY.SQL command
- temporary files used by MASTER command ('masterXX')
Adapted from a post by Dave Meeks on the Oliver UniVerse forum.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
[quote="chulett"]I've also seen files in there that seem to have been used by IPC stages.[/quote]
Craig, your post coincides with a problem we are experiencing.
Usage related to IPC stage seems to be conditional. I'll continue to investigate, unless someone knows under what condition(s) IPC uses this directory.
Craig, your post coincides with a problem we are experiencing.
Usage related to IPC stage seems to be conditional. I'll continue to investigate, unless someone knows under what condition(s) IPC uses this directory.
/tmp/man files
Please ignore this post. the /tmp/mam???? files were actually Unix temp files which were generated by the man command in Unix. They are not DataStage temp files.
Thanks
Thanks
cnguyen wrote:Does anyone know which DataStage process create /tmp/man???? files? We have changed the UVSPOOL, UVTEMP to point to a different directory. We have also changed the TMPDIR parameter for all projects. However, there we still see some man??? files in the /tmp directory. We appreciate any help.
Thanks