ExternalTargetStage: data feed [send/receive] mechanism

[ray.wurlod]

It is correct (as documented) behaviour. The External Source program is invoked once, and generates as many lines as it will onto stdout. These are delivered into the job design via the output link from the External Source stage. In your shell script $1 can only ever mean the first token from the script's command line.

Your presumption, that it is passed a parameter for each row, is simply not the correct operational model.

An External Target stage is also invoked only once. Rows delivered via the input link to the External Target stage are consumed by stdin of the External Target program.

[Ultramundane]

Ray explained how it functions very well.

I typed a tidbit of code you could model your script after to read a record coming in into arguments. It is just a model to help you out. At a minimum, you will need to modify it with correct number arguments, the correct internal field separator, and the correct F_DO_STUFF function for your purpose.

Code: Select all

#!/bin/ksh

################################################
##
## Author: Ryan Putnam
##
## Intent: A demonstration
##
################################################

###############
## FUNCTIONS ##
###############
## A SIMPLE FUNCTION TO DO STUFF
F_DO_STUFF()
{
   print "$1,$2,$3,$4"
}

##############
## MAINLINE ##
##############
## GET stdin INTO ARGUMENTS
cat - | {
VIFS="${IFS}"
IFS=";"

while read ARG1 ARG2 ARG3 ARG4 filler
do
   F_DO_STUFF "${ARG1}" "${ARG2}" "${ARG3}" "${ARG4}"
done

IFS="${VIFS}"
}

exit 0