DSXchange

I have a scenrio where the ETL process uses remove command to delete .done file and triggers the next job in the sequence creating anew .done before loading into the table? why and when do we use .done command.This loading depends on the job that processes files from other ETL project. Is it uesd to check whether the previous ETL process completed or not? I want to know more on this..pls help me

I don't think .done is a command but it could be a naming convention someone developed. It sounds like you may have to trace through the jobs and scripts and events that create the trigger files to understand it better.

Sounds like a pretty standard semaphore technique but you're the only one with access to 'the code' and thus the ability to track it down. They're typically used to signal that a file (or set of files) have transferred completely. Rather than polling for the actual file and risk starting to process it before it has (or they have) finished transferring, an empty semaphore file is sent last and is what is actually polled for; when it arrives it gets deleted and the 'real' files are processed.

Many times the semaphore or 'trigger' files have the same name as their matching transfer file, just with a fixed extension (like .done) appended to it. You can poll for "*.done" files and when one is found strip off the extension to know which file has arrived.

With parallel compatabilities of todays hardware/software the data or a file is allowed to read/write at the same time and this can mislead to a dirty read.

To avoid situations, .done files are used.
In some platforms a Header, Trailer records are used.
In most ER Application, details and header tables and Joinable records are used.

And this Architectual solutions are engineered as per the usage and Atomicity at row, table, file, App level...