Sequential File path query
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
Using an external source stage is one option, otherwise you can use a bash command in the file pattern expression of the sequential file stage, like
Eric
Code: Select all
find #PATH1#/`echo "#THEGIVENDATE#" | tr -d '-'`/#PATH2# -name "*#FILEPATTERN#*"
or
ls #PATH1#/`echo "#THEGIVENDATE#" | tr -d '-'`/#PATH2#/#FILEPATTERN#
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
@Eric
Thanks for your response.
yes I am using file patterns, but can you tell me where I have to put this code ? Is it in the File Path in the Sequential File or in the Filter property.
Because if its in the filter property then it wont work.
And what #PATH1 and #PATH2 stands for according my example ?
I'll try it tomorrow as I left my office(replying from home )
Thanks for your response.
yes I am using file patterns, but can you tell me where I have to put this code ? Is it in the File Path in the Sequential File or in the Filter property.
Because if its in the filter property then it wont work.
And what #PATH1 and #PATH2 stands for according my example ?
I'll try it tomorrow as I left my office(replying from home )
Thanx and Regards,
ETL User
ETL User
You should use it in the file path (it will retrieve file names and pass it as input files list). Filter property is used to pre-parse your data using an external program (somewhat like external source stage could be used).
PATH1 is the first part of your file path, and PATH2 the second part of it (presuming your timestamped folder is an intermediate folder in your file path).
Here, PATH1 is /data/XXX/ and PATH2 is empty (so you can remove it).
Eric
PATH1 is the first part of your file path, and PATH2 the second part of it (presuming your timestamped folder is an intermediate folder in your file path).
Here, PATH1 is /data/XXX/ and PATH2 is empty (so you can remove it).
Eric
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
Hi,
I tried your both logics. While viewing the data your code works like charm.
But when I run the job, my job aborts while fetching the last row with the below error
My file has following properties :-
Final Delimeter: End
Deliemter: Comma
Quote: none
Row Count: 100
I think while fetching the last row, it is finding a problem.
I dont know what it is.
I tried your both logics. While viewing the data your code works like charm.
But when I run the job, my job aborts while fetching the last row with the below error
Code: Select all
Source subproc: cat: 0652-050 Cannot open ls.
Filter status 512;
filter process failed: 2;
Import error at record 100.
Final Delimeter: End
Deliemter: Comma
Quote: none
Row Count: 100
I think while fetching the last row, it is finding a problem.
I dont know what it is.
Thanx and Regards,
ETL User
ETL User
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
Where did you put the command line?
It seems that you are using a filter command instead of a file pattern command.
You should have
Source=>Read Method=>File Pattern
Source=>File Pattern=>the command line I gave you adapted to your needs
No filter can be used (at least in v8.1) when you select File Pattern as Read Method.
I've just tested on my env and it's working well. Ensure that there's no space in your command and that the back apostrophe " ` " are correctly sets
Eric
It seems that you are using a filter command instead of a file pattern command.
You should have
Source=>Read Method=>File Pattern
Source=>File Pattern=>the command line I gave you adapted to your needs
No filter can be used (at least in v8.1) when you select File Pattern as Read Method.
I've just tested on my env and it's working well. Ensure that there's no space in your command and that the back apostrophe " ` " are correctly sets
Eric
-
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
@Craig/Eric,
Yes I have put the command lines where you have mentioned
The code what you have given is working fine while viewing the data from the Stage.
But when I run the job, it aborts while reading the last line(telling you this for the third time )
Even the performance statistics shows fetching 100 rows but after a couple of seconds the job fails.
@Eric,
Have you tested this code at your end ?
Didnt you get the same error when you ran the job ?
Yes I have put the command lines where you have mentioned
I havent used the filter property, as I have told you earlier.You should have
Source=>Read Method=>File Pattern
Source=>File Pattern=>the command line I gave you adapted to your needs.
It seems that you are using a filter command instead of a file pattern command
The code what you have given is working fine while viewing the data from the Stage.
But when I run the job, it aborts while reading the last line(telling you this for the third time )
Even the performance statistics shows fetching 100 rows but after a couple of seconds the job fails.
@Eric,
Have you tested this code at your end ?
Didnt you get the same error when you ran the job ?
Thanx and Regards,
ETL User
ETL User
We know what you've told us. Eric has already told you he tested it and it worked fine. You need to answer the questions asked. Don't just say "I put the command lines where you mentioned" as Eric told you to "adapt them to your needs".
Show us your exact commands (asking this from you for the third time) and where you put them. Then we can stop guessing what might be going wrong.
Show us your exact commands (asking this from you for the third time) and where you put them. Then we can stop guessing what might be going wrong.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers