Weird problem with Sequential File Stage
Moderators: chulett, rschirm, roy
Weird problem with Sequential File Stage
My job contains two parameters for the
File directory - #FileDirectory#
FileName- #Filename#
The Sequential File stage contains a filter with some awk code in it to filter out some records with the two parameters at the end of the awk code line... Filter looks like this:
awk -F'\t' '{if(<condition>) {print $0}}' #FileDirectory#/#Filename#
Now when i run the job setting the #Filename# to blanks. The job runs successfully with the director log showing
'imported 0 records', etc ...
Shouldn't the job abort when it cant locate a file? When i remove the filter and set the #Filename# to blanks, the job does abort but when the filter is present, the job runs successfully with no records imported.
This is really weird, is there an explanation for this? Or can u guys suggest how to make the abort in such a situation.. ?
File directory - #FileDirectory#
FileName- #Filename#
The Sequential File stage contains a filter with some awk code in it to filter out some records with the two parameters at the end of the awk code line... Filter looks like this:
awk -F'\t' '{if(<condition>) {print $0}}' #FileDirectory#/#Filename#
Now when i run the job setting the #Filename# to blanks. The job runs successfully with the director log showing
'imported 0 records', etc ...
Shouldn't the job abort when it cant locate a file? When i remove the filter and set the #Filename# to blanks, the job does abort but when the filter is present, the job runs successfully with no records imported.
This is really weird, is there an explanation for this? Or can u guys suggest how to make the abort in such a situation.. ?
Why do you need to abort it anyway. If there is no records, dont execute it, and notify or do whatever.
You can do this in a Execute command activity to populate a file. And using "test -s newfilename", you can check the file size and based on the result, you can execute the job or you can trigger the other stream.
You can do this in a Execute command activity to populate a file. And using "test -s newfilename", you can check the file size and based on the result, you can execute the job or you can trigger the other stream.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Re: Weird problem with Sequential File Stage
TryHavoc wrote: awk -F'\t' '{if(<condition>) {print $0}}' #FileDirectory#/#Filename#
awk -F'\t' '{if(<condition> && FILENAME != #FileDirectory#) {print $0} else {exit 1}}'
Re: Weird problem with Sequential File Stage
thanks for taking the time off to reply robinson... I tried the above. But the following two issues arise:trobinson wrote:TryHavoc wrote: awk -F'\t' '{if(<condition>) {print $0}}' #FileDirectory#/#Filename#
awk -F'\t' '{if(<condition> && FILENAME != #FileDirectory#) {print $0} else {exit 1}}'
1) The condition (which is basically a filter condition on columns) does not work i.e. the contents of the file do not display
2) Again when a blank filename is provided it does not abort and again reads from standard input.
I'm assuming u meant the command to be:
awk -F'\t' '{if(<condition> && FILENAME != #FileDirectory#) {print $0} else {exit 1}}' #FileDirectory#/#Filename#
You are missing the forward slash after the FileDirectory. Try this
Code: Select all
awk -F'\t' '{if(<condition> && FILENAME != #FileDirectory#/) {print $0} else {exit 1}}' #FileDirectory#/#Filename#
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
It worked for me both within DS and from the command line. Perhaps FILENAME = #FileDirectory#/ will never equal #FileDirectory# (without the slash).
I would suspect a snytax problem if the awk with the condition worked before. Perhaps you need to enclose the && (condition) in parens to ensure it is completely separate from the logic of the first <condition>.
Barring that, how about pasting the entire awk statement?
I would suspect a snytax problem if the awk with the condition worked before. Perhaps you need to enclose the && (condition) in parens to ensure it is completely separate from the logic of the first <condition>.
Barring that, how about pasting the entire awk statement?
Tried it out now, still not workingtrobinson wrote:It worked for me both within DS and from the command line. Perhaps FILENAME = #FileDirectory#/ will never equal #FileDirectory# (without the slash).
I would suspect a snytax problem if the awk with the condition worked before. Perhaps you need to enclose the && (condition) in parens to ensure it is completely separate from the logic of the first <condition>.
Barring that, how about pasting the entire awk statement?
![Sad :(](./images/smilies/icon_sad.gif)
Okay the entire awk statement is as follows:
awk -F'\t' '{if(($2 ~ /^[sS]/ && $3 == "10") && FILENAME != home/Havoc/Test) {print $0} else {exit 1}}' Test.txt
This results in displaying no lines.
When providing this statement with the '/' at the end it results in :
Error being displayed:
$ awk -F'\t' '{if(($2 ~ /^[sS]/ && $3 == "10") && FILENAME != home/Havoc/Test/) {print $0} else {exit 1}}' Test.txt
syntax error The source line is 1.
The error context is
{if(($2 ~ /^[sS]/ && $3 == "10") && FILENAME != >>> home/Havoc/Test/) <<<
awk: The statement cannot be correctly parsed.
The source line is 1.
awk: Quitting
The source line is 1.
My bad. My awk WAS slightly different then yours.
I changed mine and added double quotes.
Here's the error when it works correctly;
Sequential_File_0,0: Filter status 256;
filter process failed: 1;
import error at record 0.
Here is my awk;
awk -F'\t' '{if( FILENAME != "#PATH#/") {print $0} else {exit 1}}' #PATH#/#FILE#
Notice the double quotes.
I changed mine and added double quotes.
Here's the error when it works correctly;
Sequential_File_0,0: Filter status 256;
filter process failed: 1;
import error at record 0.
Here is my awk;
awk -F'\t' '{if( FILENAME != "#PATH#/") {print $0} else {exit 1}}' #PATH#/#FILE#
Notice the double quotes.
Oh okay .... I tried it again with a blank filename and the job is still successfully running with 0 records imported. I dont know if i'm blind or i'm not doing something right... The parameter values being passed aretrobinson wrote:My bad. My awk WAS slightly different then yours.
I changed mine and added double quotes.
Here's the error when it works correctly;
Sequential_File_0,0: Filter status 256;
filter process failed: 1;
import error at record 0.
Here is my awk;
awk -F'\t' '{if( FILENAME != "#PATH#/") {print $0} else {exit 1}}' #PATH#/#FILE#
Notice the double quotes.
#PATH#= /home/Havoc/Test
#FILE#= Test.txt
When i run the job for #FILE# set to all blanks (basically no filename), the job still runs with 0 records imported
SeqTest,0: Import complete; 0 records imported successfully, 0 rejected.
/ at the path isn't necessarily giving a problem.
Since a backslash is present in File path, it need to be quoted.
awk -F"," '{if( FILENAME != /dir1/dir/) ..... giving out the error that you have reported, where as awk -F"," '{if( FILENAME != "/dir1/dir") ..... is not.
Since a backslash is present in File path, it need to be quoted.
awk -F"," '{if( FILENAME != /dir1/dir/) ..... giving out the error that you have reported, where as awk -F"," '{if( FILENAME != "/dir1/dir") ..... is not.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
It works when i give the awk statement as (but again when it comes to giving a blank filename it reads from unix std input) - with quotes :kumar_s wrote:/ at the path isn't necessarily giving a problem.
Since a backslash is present in File path, it need to be quoted.
awk -F"," '{if( FILENAME != /dir1/dir/) ..... giving out the error that you have reported, where as awk -F"," '{if( FILENAME != "/dir1/dir") ..... is not.
awk -F'\t' '{if( FILENAME != "/dir1/dir/") {print $0} else {exit 1}}' dir1/dir/Test.txt
awk -F'\t' '{if( FILENAME != "/dir1/dir") {print $0} else {exit 1}}' dir1/dir/Test.txt
When i give it as (without quotes) :
awk -F'\t' '{if( FILENAME != /dir1/dir/) {print $0} else {exit 1}}' dir1/dir/Test.txt
The statement doesnt correctly parse in UNIX.
Can someone tell me if i'm doing something wrong? ..
kumar_s wrote:What is the value that you have given in "Filename" option?
The given option works. If I dont pass any value to parameter it gives out
"DSD.SEQOpen GCI $DS.SEQOPEN error 11.|" from datastage.
You mean the #Filename# parameter? ... Well for successfull run i just type in the Filename... - Test.txt
When i say blank filename, its all spaces with the cursor placed right at the start(leftmost) of the Value textbox.
Datatype for #Filename# parameter is String.