How to wait for and process multiple and changing name file

[angelovaira]

Hi,

I have 2 questions and hope that some of you has already met this kind of problem

1. How can a sequencer job of datastage wait for a file that always change his name.
For example I receive sequential files from a system that give me files named with the creation date and an id,
like this: filename_yyyymmdd_hhmmss_nnnnn, where nnnnn is a progressive id.
The filename does not change but yyyymmdd_hhmmss_nnnnn change everytime.

2.In case the system send me 2 or more files at the same time, i have to process the one with the smaller id first.

Datastage Version 8.1

Thanks a lot to everybody,
Angelo.

[chulett]

You'll need to create a script for that as the WaitForFile stage does not support wildcards. The 'smaller id first' part is just a matter of ordering the list you process properly.

[Pagadrai]

Hi,

I have 2 questions and hope that some of you has already met this kind of problem

1. How can a sequencer job of datastage wait for a file that always change his name.
For example I receive sequential files from a system that give me files named with the creation date and an id,
like this: filename_yyyymmdd_hhmmss_nnnnn, where nnnnn is a progressive id.
The filename does not change but yyyymmdd_hhmmss_nnnnn change everytime.

2.In case the system send me 2 or more files at the same time, i have to process the one with the smaller id first.

Datastage Version 8.1

Thanks a lot to everybody,
Angelo.

Hi Angelo,
1) I think we can use job parameters for the 'Wait for file' activity stage.
( i dont have access to datastage right now to double check, but will check and confirm)

2) If you have multiple files to process and you want to process them as per the order of ID, you can follow this:
-> write a unix script that will return the list of
files in that source dir in the order you want and get the list as comma separated values.
( i assume that only unprocessed files are in source dir)
->You can call this script in the sequence using execute command stage.
->Use a loop activity in the sequence to call DS job in loop, with the counter value given by the comma seperated list of file names i.e output of the unix script.

let me know if it helps or you need more info.

[chulett]

1) Parameters, yes - wildcards, no.

[ray.wurlod]

You can use an Execute Command activity to execute an ls -1 command and pipe its results through sort. Then use a User Variables activity to change the field marks in the command output to commas, and the result becomes your "list of things" list (comma delimited) in a Start Loop activity. Use a Nested Condition activity to answer the question are there any files there at all?

[angelovaira]

Thanks to everybody for suggestions, i will try and let you know...
Angelo.

[angelovaira]

Thanks to everybody for suggestions, i will try and let you know...
Angelo.

[rameshrr3]

In execute command activity in a sequencer
Try sh-c with argument

Code: Select all

 'ls -trx #FILEDIR##FileNamePattern# | tr "\n" ","' 

to get a comma seperated list, I use it my process, The first value in the output list is the oldest created file.

You may want to sort further on the sequence number- maybe using a routine or a sort command.

Regards
Ramesh