Comparing the CommandOutput of 2 Execute Command Activities

[msukkar]

I am new to DataStage (my 4th month now) and this is my first post. I hope you believe me that I DID search before post but without finding what I am looking for. If this is not the appropriate forum, please let me know. DS version is 8.1

My input file has a trailer record that contains like this
End of File | xxxxxxx

where xxxxxxx is the record count of the whole file (including the trailer record itself).

I am trying to do file validation inside a DataStage sequence job. My approach is as follows:

Code: Select all

ExecuteActivity (CheckFileCount) --> ExecuteActivity (CheckTrailerCount) ----(countok)-------> JobActivity(ExtractSource)
                                                                             | (countfailed)
                                                                           Abort

CheckFileCount:
Command: wc
Parameter: -l /path/to/file  | awk '{printf $1}'
Triggers: Unconditional

CheckTrailerCount:
command: grep
Parameter: "End of File" /path/to/file | awk '{printf $5}'
Triggers: (here is my trouble)
countok : Custom(Conditional) : Field(Trim(CheckTrailerCount.$CommandOutput), ' ', 1) = Field(Trim(CheckFileCount.$CommandOutput), ' ', 1)
countfailed: Custom(Conditional): Field(Trim(CheckTrailerCount.$CommandOutput), ' ', 1) <> Field(Trim(CheckFileCount.$CommandOutput), ' ', 1)

When I run the commands in UNIX, they are executing fine and giving me the correct output I need.

however, in DS the sequence is always branching to abort (because of the trigger). Director log is showing the output of these 2 activities as follows:

Code: Select all

JobName..JobControl (@CheckFileCount): Executed: wc -l /path/to/file  | awk '{printf $1}'
Reply=0
Output from command ====>
1153047

Jobname..JobControl (@CheckTrailerCount): Executed: grep "End of File" /path/to/file | awk '{printf $5}'
Reply=0
Output from command ====>
1153047

Jobname..JobControl (@Coordinator): Summary of sequence run
17:07:47: Sequence started
17:08:04: CheckFileCount (COMMAND wc) started
17:08:08: CheckFileCount finished, reply=0
17:08:08: CheckTrailerCount (COMMAND grep) started
17:08:08: CheckTrailerCount finished, reply=0
17:08:08: Stop Executed
17:08:08: Sequence terminated

I tried different variations of the trigger syntax as follows:

Code: Select all

Field(Trim(Convert(@FM," ",CheckTrailerCount.$CommandOutput))," ",1,1) = Field(Trim(Convert(@FM," ",CheckFileCount.$CommandOutput))," ",1,1)

Trim(CheckFileCount.$CommandOutput, @FM) = Trim(CheckTrailerCount.$CommandOutput, @FM)

After many trials, I tired to isolate the problem, and tried to hardcode the number and compare it to each commandoutput individually. For CheckFileCount the comparison succeeds and the job does not abort. However, for CheckTrailerCount the comparison is failing and the job is aborting. I am wondering why, since both commands are the output of awk.

I highly appreciate if someone can shed a light on this problem or suggest a better approach to solve the same task.

Thanks a million

[ray.wurlod]

Make sure you eliminate trailing newline characters. They will be returned as field marks (@FM). Add some diagnostics to verify that the commands' outputs are being returned correctly to DataStage.

[msukkar]

Thanks Ray for the hint. I echoed CommandOutput to another file and I saw that the CheckTrailerCount is actually storing CRLF at the end of it.

I found a similar EReplace expression from another DSX thread here viewtopic.php?t=140129 and it solved my problem.

Code: Select all

EReplace(EReplace(Trim(CheckTrailerCount.$CommandOutput, @FM), char(10), ''), char(13), '') = EReplace(EReplace(Trim(CheckFileCount.$CommandOutput, @FM), char(10), ''), char(13), '')

Thanks again

[ray.wurlod]

Overkill. All you need to trim is the @FM character.

[msukkar]

Overkill. All you need to trim is the @FM character. ...

I tried that in my first post but it did not work for some reason.