Weird problem with Sequential File Stage

[Havoc]

My job contains two parameters for the

File directory - #FileDirectory#
FileName- #Filename#

The Sequential File stage contains a filter with some awk code in it to filter out some records with the two parameters at the end of the awk code line... Filter looks like this:


awk -F'\t' '{if(<condition>) {print $0}}' #FileDirectory#/#Filename#

Now when i run the job setting the #Filename# to blanks. The job runs successfully with the director log showing

'imported 0 records', etc ...

Shouldn't the job abort when it cant locate a file? When i remove the filter and set the #Filename# to blanks, the job does abort but when the filter is present, the job runs successfully with no records imported.

This is really weird, is there an explanation for this? Or can u guys suggest how to make the abort in such a situation.. ?

[kumar_s]

An empty string will be passed as parameter, and hence the result of the your code will return an empty values to standard output, and that was read as input by the sequential file stage.

[Havoc]

Thanks for replying kumar,

Okay that makes sense... But , is there a way to make the job abort in this case? .. I'm trying my best to avoid using the Filter stage..

Anyway to make the job abort even with the awk script in place..

[kumar_s]

Why do you need to abort it anyway. If there is no records, dont execute it, and notify or do whatever.
You can do this in a Execute command activity to populate a file. And using "test -s newfilename", you can check the file size and based on the result, you can execute the job or you can trigger the other stream.

[trobinson]

awk -F'\t' '{if(<condition>) {print $0}}' #FileDirectory#/#Filename#

Try
awk -F'\t' '{if(<condition> && FILENAME != #FileDirectory#) {print $0} else {exit 1}}'

[Havoc]

awk -F'\t' '{if(<condition>) {print $0}}' #FileDirectory#/#Filename#

Try
awk -F'\t' '{if(<condition> && FILENAME != #FileDirectory#) {print $0} else {exit 1}}'

thanks for taking the time off to reply robinson... I tried the above. But the following two issues arise:

1) The condition (which is basically a filter condition on columns) does not work i.e. the contents of the file do not display

2) Again when a blank filename is provided it does not abort and again reads from standard input.

I'm assuming u meant the command to be:

awk -F'\t' '{if(<condition> && FILENAME != #FileDirectory#) {print $0} else {exit 1}}' #FileDirectory#/#Filename#

[DSguru2B]

You are missing the forward slash after the FileDirectory. Try this

Code: Select all

awk -F'\t' '{if(<condition> && FILENAME != #FileDirectory#/) {print $0} else {exit 1}}' #FileDirectory#/#Filename# 

[trobinson]

It worked for me both within DS and from the command line. Perhaps FILENAME = #FileDirectory#/ will never equal #FileDirectory# (without the slash).

I would suspect a snytax problem if the awk with the condition worked before. Perhaps you need to enclose the && (condition) in parens to ensure it is completely separate from the logic of the first <condition>.

Barring that, how about pasting the entire awk statement?

[Havoc]

It worked for me both within DS and from the command line. Perhaps FILENAME = #FileDirectory#/ will never equal #FileDirectory# (without the slash).

I would suspect a snytax problem if the awk with the condition worked before. Perhaps you need to enclose the && (condition) in parens to ensure it is completely separate from the logic of the first <condition>.

Barring that, how about pasting the entire awk statement?

Tried it out now, still not working

Okay the entire awk statement is as follows:

awk -F'\t' '{if(($2 ~ /^[sS]/ && $3 == "10") && FILENAME != home/Havoc/Test) {print $0} else {exit 1}}' Test.txt

This results in displaying no lines.

When providing this statement with the '/' at the end it results in :

Error being displayed:

$ awk -F'\t' '{if(($2 ~ /^[sS]/ && $3 == "10") && FILENAME != home/Havoc/Test/) {print $0} else {exit 1}}' Test.txt
syntax error The source line is 1.
The error context is
{if(($2 ~ /^[sS]/ && $3 == "10") && FILENAME != >>> home/Havoc/Test/) <<<
awk: The statement cannot be correctly parsed.
The source line is 1.
awk: Quitting
The source line is 1.

[trobinson]

My bad. My awk WAS slightly different then yours.
I changed mine and added double quotes.

Here's the error when it works correctly;

Sequential_File_0,0: Filter status 256;
filter process failed: 1;
import error at record 0.

Here is my awk;
awk -F'\t' '{if( FILENAME != "#PATH#/") {print $0} else {exit 1}}' #PATH#/#FILE#

Notice the double quotes.

[Havoc]

My bad. My awk WAS slightly different then yours.
I changed mine and added double quotes.

Here's the error when it works correctly;

Sequential_File_0,0: Filter status 256;
filter process failed: 1;
import error at record 0.

Here is my awk;
awk -F'\t' '{if( FILENAME != "#PATH#/") {print $0} else {exit 1}}' #PATH#/#FILE#

Notice the double quotes.

Oh okay .... I tried it again with a blank filename and the job is still successfully running with 0 records imported. I dont know if i'm blind or i'm not doing something right... The parameter values being passed are

#PATH#= /home/Havoc/Test
#FILE#= Test.txt

When i run the job for #FILE# set to all blanks (basically no filename), the job still runs with 0 records imported

SeqTest,0: Import complete; 0 records imported successfully, 0 rejected.

[kumar_s]

/ at the path isn't necessarily giving a problem.
Since a backslash is present in File path, it need to be quoted.
awk -F"," '{if( FILENAME != /dir1/dir/) ..... giving out the error that you have reported, where as awk -F"," '{if( FILENAME != "/dir1/dir") ..... is not.

[Havoc]

/ at the path isn't necessarily giving a problem.
Since a backslash is present in File path, it need to be quoted.
awk -F"," '{if( FILENAME != /dir1/dir/) ..... giving out the error that you have reported, where as awk -F"," '{if( FILENAME != "/dir1/dir") ..... is not.

It works when i give the awk statement as (but again when it comes to giving a blank filename it reads from unix std input) - with quotes :

awk -F'\t' '{if( FILENAME != "/dir1/dir/") {print $0} else {exit 1}}' dir1/dir/Test.txt

awk -F'\t' '{if( FILENAME != "/dir1/dir") {print $0} else {exit 1}}' dir1/dir/Test.txt

When i give it as (without quotes) :

awk -F'\t' '{if( FILENAME != /dir1/dir/) {print $0} else {exit 1}}' dir1/dir/Test.txt

The statement doesnt correctly parse in UNIX.

Can someone tell me if i'm doing something wrong? ..

[kumar_s]

What is the value that you have given in "Filename" option?
The given option works. If I dont pass any value to parameter it gives out
"DSD.SEQOpen GCI $DS.SEQOPEN error 11.|" from datastage.

[Havoc]

What is the value that you have given in "Filename" option?
The given option works. If I dont pass any value to parameter it gives out
"DSD.SEQOpen GCI $DS.SEQOPEN error 11.|" from datastage.

You mean the #Filename# parameter? ... Well for successfull run i just type in the Filename... - Test.txt

When i say blank filename, its all spaces with the cursor placed right at the start(leftmost) of the Value textbox.

Datatype for #Filename# parameter is String.