Hi Gurus,
I have a case, our team has designed jobs to fetch data from a folder and put it in a staging table and from there they load it to DW. All these are available in sequence.
The initial part will be a file (eg: xxxxx.D.MI5.010101120345) will be dropped by another team into our folder. we pick the file and do the rest of the processing. This seq fails when more than 1 file is available in the folder. one of the solution proposed was to write a shell script to check for file availablity in the folder. once the file is available we should trigger the sequence to execute.
note - i know the design is not good. but we need to handle this case.
pls help me out in solving this case. i dont have exposure in shell scripting. if any script is availble pls share.
Advance Thanks,
bgs_vb
checking for file availability in a folder
Moderators: chulett, rschirm, roy
Re: checking for file availability in a folder
Did you try this-
In the properties of Seq file stage.
Click on Read method under Source and select "File pattern" from the pop down menu.
I strongly feel that this should work in your case.
Good luck!
In the properties of Seq file stage.
Click on Read method under Source and select "File pattern" from the pop down menu.
I strongly feel that this should work in your case.
Good luck!
Those who throw objects at crocodiles should be asked to retreive 'em.
Sequence has both parallel and server jobs
in my case i have almost 8 jobs both parallel & sequence combined and put in the SEQUENCER. 1st job is a sequence job which fetches file from folder stage and processes it.
as i said earlier, a file is dropped in the folder by another program anytime of the day and i should 1st check if any file has come into the respective folder, if so available then execute the whole sequence.
one more option i tried was with START_LOOP, End_LOOP stage. i had issue there as i could not fetch the file name, again, as it is dropped in a folder.
start_loop 1---> start_loop 2---> wait_for_file_stage ---> sequencer ---> end_loop 1 |
|
end_loop 2 (goes to start_loop2)
in the wait_for_file stage my file is returned as 'sssss.p.qs9.123456789' and not as ssssss.p.qs9.123456789
as i said earlier, a file is dropped in the folder by another program anytime of the day and i should 1st check if any file has come into the respective folder, if so available then execute the whole sequence.
one more option i tried was with START_LOOP, End_LOOP stage. i had issue there as i could not fetch the file name, again, as it is dropped in a folder.
start_loop 1---> start_loop 2---> wait_for_file_stage ---> sequencer ---> end_loop 1 |
|
end_loop 2 (goes to start_loop2)
in the wait_for_file stage my file is returned as 'sssss.p.qs9.123456789' and not as ssssss.p.qs9.123456789
-
- Participant
- Posts: 612
- Joined: Thu May 03, 2007 4:59 am
- Location: Melbourne
When the 'wait_for_file_stage' detects a file in the waiting folder, move the file into a temporary folder if you want. Then call the sequence which is having the job with folder stage where you can 'fetch the file name'. Put the above sequence in a loop to go back and wait for another file.
Joshy George
<a href="http://www.linkedin.com/in/joshygeorge1" ><img src="http://www.linkedin.com/img/webpromo/bt ... _80x15.gif" width="80" height="15" border="0"></a>
<a href="http://www.linkedin.com/in/joshygeorge1" ><img src="http://www.linkedin.com/img/webpromo/bt ... _80x15.gif" width="80" height="15" border="0"></a>
bgs_vb,
Write a small shell script continously sweeping the folder for the file with the given specific pattern, if the file is present you can trigger the sequence. In korn shell use if [[ -f *pattern ]]. You dont need Start Loop and End Loop stages, just a while loop that runs for a specific time with a 30 sec sleep time would be fine.
dsrules
Write a small shell script continously sweeping the folder for the file with the given specific pattern, if the file is present you can trigger the sequence. In korn shell use if [[ -f *pattern ]]. You dont need Start Loop and End Loop stages, just a while loop that runs for a specific time with a 30 sec sleep time would be fine.
dsrules
______________________________________
"Everytime I close the door on reality, it comes in through the windows." - Jennifer Yane
"Everytime I close the door on reality, it comes in through the windows." - Jennifer Yane
-
- Participant
- Posts: 73
- Joined: Tue May 10, 2005 6:21 pm
- Location: Sydney
Re: Sequence has both parallel and server jobs
I would run a shell script as a backgroup process to check the existing of the file and then trigger the DataStage to run if detected.
bgs_vb wrote: as i said earlier, a file is dropped in the folder by another program anytime of the day and i should 1st check if any file has come into the respective folder, if so available then execute the whole sequence.
Edward Yuan
as i said earlier i have very little exposure to unix script
if anyone has that script could share.
Thanks
bgs_vb
Thanks
bgs_vb