File Pattern -- Reading sequentially
Moderators: chulett, rschirm, roy
File Pattern -- Reading sequentially
Hi,
I have 3 files which match a pattern.
JobName_Date1.txt
JobName_Date2.txt
JobName_Date3.txt
I want the file with JobName_Date1.txt to be read first and then JobName_Date2.txt and then JobName_Date3.txt
Can we specify this order anywhere that it has to read in a sorted order?
Thanks,
Anitha
I have 3 files which match a pattern.
JobName_Date1.txt
JobName_Date2.txt
JobName_Date3.txt
I want the file with JobName_Date1.txt to be read first and then JobName_Date2.txt and then JobName_Date3.txt
Can we specify this order anywhere that it has to read in a sorted order?
Thanks,
Anitha
-
- Premium Member
- Posts: 892
- Joined: Thu Oct 16, 2003 5:18 am
-
- Premium Member
- Posts: 258
- Joined: Tue Jul 04, 2006 10:35 pm
- Location: Toronto
DataStage installs the MKS Toolkit on Windows so you should in fact have UNIX-like capabilities and thus commands like "ls" can be used. Or just stick with "dir /b", either will display files alphabetically by default.
That "/b" option gives you a "bare" listing, one with just the filenames which is what would be appropriate in the Sequential File stage's File Pattern option.
That "/b" option gives you a "bare" listing, one with just the filenames which is what would be appropriate in the Sequential File stage's File Pattern option.
Last edited by chulett on Thu Dec 30, 2010 9:41 am, edited 1 time in total.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 81
- Joined: Tue Aug 15, 2006 8:31 am
- Location: Zürich
- Contact:
Setup your Seq stage to read the 3 files using a wildcard. Create a new column as part of the take-on that is the name of the source file (use a parameter maybe?)
Use a hash partition method on the source file key you created above and the data will be grouped in the same way as the take-on.
Cheers,
Ray D
Use a hash partition method on the source file key you created above and the data will be grouped in the same way as the take-on.
Cheers,
Ray D
-
- Premium Member
- Posts: 258
- Joined: Tue Jul 04, 2006 10:35 pm
- Location: Toronto
You've got things in the right place as it is trying to do what you asked. And I wasn't specific enough in my previous reply - that "dir /b" output is an example of what the stage needs, not the actual command itself to be used there. Sorry.
Try simply putting "D:/Employee*.txt" there and letting us know how it goes.
Try simply putting "D:/Employee*.txt" there and letting us know how it goes.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
If I simply put "D:/Employee*.txt", its getting read by alphebetical order. Now, I want it to be read according to the system time and date.
Is there anything called as indirect file reading in DataStage? Meaning, all the files to be read will be put in one file and DataStage should read this file to get the file names and in turn read the files.
Is there anything called as indirect file reading in DataStage? Meaning, all the files to be read will be put in one file and DataStage should read this file to get the file names and in turn read the files.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Capture the output of ls -1rt Employee*.txt command, convert the line terminators to, say, commas, and use that string as the "list of things" processed by a Start Loop activity.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
You could use a FileSet stage. Create a fileset file (name.fs) containing rows which look like this:Shruthi wrote: Is there anything called as indirect file reading in DataStage? Meaning, all the files to be read will be put in one file and DataStage should read this file to get the file names and in turn read the files.
nodename:full_filename
and pass the fileset name to the FileSet stage. You'll probably need to set the "Use Schema defined in File Set" to False.
An example would be:
compute1:/home/dsadm/my_input_file_1.txt
compute1:/home/dsadm/my_input_file_2.txt
However, building on Craig and Ray's suggestions you should be able to do the following with the SeqFile stage
ls -1t D:/Employee*.txt
Either method will work and I expect the second one will be easier for you.
Regards,
[/b]
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.