Multiple writes to a Hash file
Posted: Wed Aug 11, 2004 7:58 pm
I have a DS job that has three transform stages, all writing to the same hash file. The job exceute SQL to access the MAX(BATCH_ID) from a number of Oracle tables. The first stage accesses a single table and completes in a second or so, and writes this row to the hash file after a delete/create of the hash file.
The create is defaulting to WRITE DEFERRED caching (because I did not think to question the wisdom or otherwise of this). There is a message in the log saying that WRITE DEFERRED caching is dissabled. Why?
The the other two stages eventually write about 100 rows each to the hash file, after maybe 15-20 minutes elapsed.
My question is, what will happen if these two stages were to start writing to the hash file simultaneously. Is there any locking mechanism used with hash files that will prevent or allow concurrent writes to the same hash file? Should the caching be defined as NONE, or WRITE IMMEDIATE, in the definition options?
(I have seen two individual jobs running where the first was reading a hash file, then the second started later, deleted/created the hash file and wrote to it, hence destroying the data that the first job thought was there! These jobs now use different names. It was not easy to find that bug!).
TIA,
Neil Courtney
The create is defaulting to WRITE DEFERRED caching (because I did not think to question the wisdom or otherwise of this). There is a message in the log saying that WRITE DEFERRED caching is dissabled. Why?
The the other two stages eventually write about 100 rows each to the hash file, after maybe 15-20 minutes elapsed.
My question is, what will happen if these two stages were to start writing to the hash file simultaneously. Is there any locking mechanism used with hash files that will prevent or allow concurrent writes to the same hash file? Should the caching be defined as NONE, or WRITE IMMEDIATE, in the definition options?
(I have seen two individual jobs running where the first was reading a hash file, then the second started later, deleted/created the hash file and wrote to it, hence destroying the data that the first job thought was there! These jobs now use different names. It was not easy to find that bug!).
TIA,
Neil Courtney