Weird problem with Sequential File Stage
Moderators: chulett, rschirm, roy
Hi kumar,kumar_s wrote:Under the General Tab, you might use Filter command to this, are you using /dev/null in "File Name" option? ...
yes i am placing the following as the filter in the stage:
awk -F'\t' '{if( FILENAME != " #FileDirectory#/ " ) {print $0} else {exit 1}}' #FileDirectory#/#Filename#
where #FileDirectory# = /dir1/dir
and #FileName# = Test.txt
As i mentioned earlier if i remove " " from the above awk, the file isnt read itself.
I didnt quite get what you meant by:
are you using /dev/null in "File Name" option? ...
Sorry for bumping this old topic back up .. But, after a week, I still have not found a solution to the problem ... 
Kumar, did you mean we have to check for FILENAME = "/dev/null" in the initial If condition? I've still not found a solution to this problem..
Is there anyway we can leverage the ARGC,ARGV variables to exit/abort the job?
Kumar, did you mean we have to check for FILENAME = "/dev/null" in the initial If condition? I've still not found a solution to this problem..
Is there anyway we can leverage the ARGC,ARGV variables to exit/abort the job?
What is the exact issue now?
As trobinson and I mentioned, it gave out error when there is no file.
You might have entered the awk command in option provided for "Filter". What did you mentioned under the option provided for "File name"?
As trobinson and I mentioned, it gave out error when there is no file.
You might have entered the awk command in option provided for "Filter". What did you mentioned under the option provided for "File name"?
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
What is the exact issue now?
As trobinson and I mentioned, it gave out error when there is no file.
You might have entered the awk command in option provided for "Filter". What did you mentioned under the option provided for "File name"?
As trobinson and I mentioned, it gave out error when there is no file.
You might have entered the awk command in option provided for "Filter". What did you mentioned under the option provided for "File name"?
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Thanks for the quick reply kumar...kumar_s wrote:What is the exact issue now?
As trobinson and I mentioned, it gave out error when there is no file.
You might have entered the awk command in option provided for "Filter". What did you mentioned under the option provided for "File name"?
Yeah I have placed the awk in the filter option of the Seq File stage.
But I still do not understand what you mean by option provided for "File name"...
The awk script is exactly similar to the one you mentioned here:
/ at the path isn't necessarily giving a problem.
Since a backslash is present in File path, it need to be quoted.
awk -F"," '{if( FILENAME != /dir1/dir/) ..... giving out the error that you have reported, where as awk -F"," '{if( FILENAME != "/dir1/dir") ..... is not.
And i also tried with robinson's and still the job doesn't abort but runs with 0 records imported.
I think it must be the FILENAME != "#FileDirectory#/".
Ever.
How come?
Did you try to hard code the condition (ex. && 1 != 2 ) to exercise the else {exit 1}
in order to see if the overall flow will work? When I tested it I deleted the value of the file parameter. I didn't blank it out. It became empty. Is this what you are doing? Could there be errant spaces after the "/" in the condition?
Ever.
How come?
Did you try to hard code the condition (ex. && 1 != 2 ) to exercise the else {exit 1}
in order to see if the overall flow will work? When I tested it I deleted the value of the file parameter. I didn't blank it out. It became empty. Is this what you are doing? Could there be errant spaces after the "/" in the condition?
I didnt quite understand the 'Ever' in your second line ..trobinson wrote:I think it must be the FILENAME != "#FileDirectory#/".
Ever.
How come?
Did you try to hard code the condition (ex. && 1 != 2 ) to exercise the else {exit 1}
in order to see if the overall flow will work? When I tested it I deleted the value of the file parameter. I didn't blank it out. It became empty. Is this what you are doing? Could there be errant spaces after the "/" in the condition?
Yes, I deleted the value for the parameter... as in ... it was left blank ... with the cursor on left most side of the textbox .. But it doesnt work...
I dont think its a problem with the errant spaces ... Do u know what kumar is talking abt using the awk as a filter?
Are you using the awk as a filter for the Sequential File stage? ... That's how i'm doing it ... Is that wrong ?
My Bad!!! Its "File=?" and not File Name.
I gave File = /dev/null
and Filter = awk -F'\t' '{if( FILENAME != " #FileDirectory#/ " ) {print $0} else {exit 1}}' #FileDirectory#/#Filename#
And when I give a file name which doesn't available, it gave out the error
Sequential_File_0,0: Source subproc: awk: Cannot find or open file /dir1/filename.txt.
The source line number is 1.
.
Sequential_File_0,0: Filter status 512;
filter process failed: 2;
import error at record 0.
Were you trying some other test cases??
I gave File = /dev/null
and Filter = awk -F'\t' '{if( FILENAME != " #FileDirectory#/ " ) {print $0} else {exit 1}}' #FileDirectory#/#Filename#
And when I give a file name which doesn't available, it gave out the error
Sequential_File_0,0: Source subproc: awk: Cannot find or open file /dir1/filename.txt.
The source line number is 1.
.
Sequential_File_0,0: Filter status 512;
filter process failed: 2;
import error at record 0.
Were you trying some other test cases??
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Thanks for replying kumar,kumar_s wrote:My Bad!!! Its "File=?" and not File Name.
I gave File = /dev/null
and Filter = awk -F'\t' '{if( FILENAME != " #FileDirectory#/ " ) {print $0} else {exit 1}}' #FileDirectory#/#Filename#
And when I give a file name which doesn't available, it gave out the error
Sequential_File_0,0: Source subproc: awk: Cannot find or open file /dir1/filename.txt.
The source line number is 1.
.
Sequential_File_0,0: Filter status 512;
filter process failed: 2;
import error at record 0.
Were you trying some other test cases??
Just tried this out now .. Still doesnt seem to work . It works perfectly fine when it comes to a successful read and when an incorrect filename's mentioned. But when providing no value to the #FileName# parameter, the job still continues to run.
So yes, the test case over here would be when provided with a blank filename .. Unix still reads from standard input thus 0 imported records and job runs ..
When I give a blank filename or blank folder name, it got aborted. But interestingly with different error message
Sequential_File_0,0: Source subproc: awk: Input line cannot be longer than 3,000 bytes.
The input line number is 4. The file is /.
The source line number is 1.
What is your server flavour? Is it not AIX?
Sequential_File_0,0: Source subproc: awk: Input line cannot be longer than 3,000 bytes.
The input line number is 4. The file is /.
The source line number is 1.
What is your server flavour? Is it not AIX?
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Working on HP-Unix ...kumar_s wrote:When I give a blank filename or blank folder name, it got aborted. But interestingly with different error message
Sequential_File_0,0: Source subproc: awk: Input line cannot be longer than 3,000 bytes.
The input line number is 4. The file is /.
The source line number is 1.
What is your server flavour? Is it not AIX?
