File Pattern and File Order

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
wjfitzgerald
Participant
Posts: 72
Joined: Tue Feb 05, 2008 4:38 am

File Pattern and File Order

Post by wjfitzgerald »

Hi,

I am using the File Pattern attribute in a sequential file stage to read a number of files. the files all have the same metadata. the file titles are all of the format FileName.Date.time, where the FileName is common. The files are being read, but not in any particular order.

Obviously, I want to read all the files in date time order, but cannot find how to enforce this. the stage runs in parallel mode and there is no drop down available to change it to sequential.

The crux of the issue is that if I have 3 files a, b, and c, I would like to read the records from a first then b and finally c, however I am ending reading c then a then b, or something similar.

I have read a mountain of post on the file pattern, but have not found one concerning the order of processing yet.

If you have any thoughts or tips, I would be very grateful.

Regards

John Fitz
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You have not control over the file pattern, unless you can dream up a regular expression that assembles the file names in date order. Prefer a File property with a Filter command that issues a cat command to assemble the data in date order.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply