Wait for File Wild Card Seq File

[nvalia]

Hi All,

We have a requirement to poll for a Dated file in a specified directory. I was thinking of using a Wait For File Stage in the Sequence job to do this (we do not have a Scheduler) but I found in another post that Wait for File does not support Wild Card Search (File Pattern)

I tested this and it is True..It needs a specific file name like xyz.txt and not something like xyz_20120327.txt

Is there any other way we can implement this in Datastage?

Thanks,
NV

[Poovalingam]

We can do in below steps,
1. Execute command activity to get the exact name of the file.
2. User Variable Activity to create a variable with value of exact file name got through exec command activity. Field(ExecCommandStageName.$CommandOutput,@FM,1)
3. Pass the uservariable to the wait for file activity file name.

Cheers,
Poova.

[BI-RMA]

If your filename is actually xyz_#DATE_YYYYMMDD#.txt you can provide parameter #DATE_YYYYMMDD# from the sequence, of course. WaitForFile activity does support the use of parameters.

[nvalia]

Poovalingam..

I do not know when the file comes in and hence need the Wait for File Stage to poll and trigger the process on arrival..

So when will I run the Execute Command stage as you mentioned below?
Can you please explain your solution in that context as I do not understand

Thanks

[Poovalingam]

I'm sorry. That will not work as our requirement is to wait for a unknown name file. It's better to ask for a dummy file and we can wait for the dummy file in data stage job. That dummy file should be placed immediately followed by the actual file from the upstream system.
Else I think below approach will work.
1. Start loop (1 to 10 - based on our requirement)
2. Exec command activity checking for a file pattern using a if condition. If file found print 'Found' else sleep for some seconds.
2. Two output link from Exec.command activity. First output link using a custom trigger condition checking for the output 'Found' and second using otherwise condition to end loop activity.

Regards,
Poova.

[nvalia]

Thanks Poovalingam..
I was able to get this to work based on your 3 step solution.

So now the job starts at a predefined time..the Loop starts and looks for the file (wild card search) through the Script, if file exists then it exits and the downstream process starts if does not find any files, it sleeps for 2 mins in the script and retutns back to simply start the next iteration..

So this way based on the Iterations we set and the sleep time defined in the script we can poll the file for the required amount of time and at preset intervals

[nvalia]

To add further..

I am using this Trigger Condition for Success in the Execute Commannd stage (Script will output 0 and exit with 0 for success)

ECA_BatFile.$ReturnValue=0 AND
Convert(@FM,"", ECA_BatFile.$CommandOutput)=0


For other loop link..simply use Otherwise as the trigger condition to continue the loop