How can I process squential files in date order?

[devesh_ssingh]

Hello Mentors,

I have asked to design a job where 3 files are kept on server , now i have to form a loop through which oldest file should come first, then newest and then the most recent.
how could one do it in datastage??
i am afraid of any looping logic other than job sequencer?

Regards,
Devesh

[*Note - Title changed to be more descriptive - Andy*]

[gssr]

i have to form a loop through which oldest file should come first, then newest and recent.


Regards,
Devesh

What exactly you are doing with the files, just moving or extracting ???

[devesh_ssingh]

extracting in loop.....

[gssr]

Then.......
Can you give detail about the job structure, source ,transformation (if any), Target database..

Do you want to load all the three files into one table, If yes u can use file pattern!!

[ArndW]

How do you determine the relative ages of the files? Is it part of the file name, or an OS attribute (date-time-created or date-time-modified or the like)?

[ray.wurlod]

Why do you need a loop? Sort the files into date order (for example using an ls -lrt command) and use that result as the command line for a cat command. For example

Code: Select all

cat `ls -lrt /path/filepattern`

Use that as the Filter command in a Sequential File stage.
If the file names don't all fit on one line, adjust the Filter command appropriately - you may need to use a for loop therein.

[devesh_ssingh]

@ray
i have to achive this in datastage.
purpose is to get the data extracted to 3 datasets...for every new data
again new files.
the source file will be replaced in next run and have to extract to new dataset like in codes...

for( i=1, i<=3)
{
srci--->copy--->ds1
}
endloop;

src will be picked based on descending timestamp...
oldest to newset...

i donno how to loop somthing like this in datastage...can we acheived it by sequencer any mean...

i hope this give more clear picture

[chulett]

Yes, use the Loop stages in a Sequence job to run your file loading job. Use the UserVariables stage to retrieve the filenames in the proper order in a delimited format.

[ray.wurlod]

My answer, had you been able to read it all, IS in DataStage.

Premium membership is not expensive, at less than 30c (Rs12) per day, and well worth the investment.

[chulett]

My answer assumes they actually do need to 'loop' because the files need to be processed individually to completion in the proper order.

[ray.wurlod]

I think my solution would achieve that. Particularly if the final line in each file were "special" in some way.