Touch File mechanism

samarvind · Post by **samarvind** » Mon Oct 22, 2007 7:18 am

Hi All,

We need to design a system such as a job sequence which should be immediately triggered once a touch file (used to indicate the initial process is finished) is placed on the FTP server or some other common server.

Basically we have a two separate process, once the first process is finished it will simply put a text file in a common server indicating their process is over , upon seeing this file , the second process which is a job sequence should be triggered automatically.

1) I have tried for wait for file activity which keeps on running till the file is available but if the file is in FTP server how can we use this stage.
2) Can we achieve this using the crontab ?

Thanking you all in advance for your valuable suggestions.

chulett · Post by **chulett** » Mon Oct 22, 2007 7:54 am

The 'semaphore' file would need to be touched into existance on the DataStage server for it to be of any easy use. Otherwise, you'd need to use another mechanism to poll for its arrival.

jdmiceli · Post by **jdmiceli** » Wed Oct 24, 2007 2:00 pm

You could use a Sequence job to and schedule that. Set up the wait for file stage to look for the touch file.

samarvind · Post by **samarvind** » Thu Oct 25, 2007 3:36 am

jdmiceli wrote:You could use a Sequence job to and schedule that. Set up the wait for file stage to look for the touch file.

Yes, we can use the DS wait for file stage but anyhow we need to run the sequence at one point of time to wait for the file and it will wait continuously till the file is available and I doubt triggering this job and waiting for the file will impact the DS server performance.

chulett · Post by **chulett** » Thu Oct 25, 2007 7:12 am

So... where are we at with this? I don't see a question in your latest post, so not sure if you have a direction, the answers you need...

samarvind · Post by **samarvind** » Thu Oct 25, 2007 8:10 am

Yes Chulett, I am still thinking of any other ways of doing it and meanwhile expecting various suggestions from you people.

Basically I want to achieve the trigger based mechanism in DS with OS being UNIX. I got one solution of running the script in infinite loop as and when the file is processed , the other script will be called which triggers the DS sequence.

Is there any other way of doing it. Also will be please explain me on how the semaphore stuffs work.

samarvind · Post by **samarvind** » Fri Oct 26, 2007 7:05 am

Will there be any disadvantages if there is an Unix script running in infinite loop and once the file is found it will trigger the main process and after the successful completion of the main process, the file will be deleted and process will continue running in loop for 24/7.

Do let me know any other suggestions or on how to improvise this design

Thanks for your suggestions in advance.

samarvind · Post by **samarvind** » Thu Aug 29, 2013 8:16 am

Hi,

We have arrived at a solution in UNIX to implement the touch file mechanism with the following steps using a UNIX script.

1) Script touches a file in a directory where all the source file arrives
2) Based on the modified date of the touch file, look for any files in the directory <= Touch file modified date
3) Store all the file in a list and only move those to the respective directory
4) schedule the UNIX script in crontab every 15 minutes or so based on the agreed frequency.

This removes any dependency with the various source system data arriving at different times, only the source file which completed its transfer before the touch file was created would be processed.

Rgds
AS

chulett · Post by **chulett** » Thu Aug 29, 2013 9:20 am

Wow... better late than never I suppose.