Filter stage and dynamic file creation

[ray.wurlod]

I don't believe it can be done using dynamically generated file names without creating some code. I would be happy to be corrected.

What's the basis of the Filter stage? You can't have a dynamic number of output links! What's your strategy for separating the countries? About the only strategy with a Filter stage would be to have an output link for every possible country. Some output links will receive zero rows. These can be detected subsequently.

[Ultramundane]

Could you use an external target stage for your job? You could create a shell script similar to the one below. It will create a file for each distinct value of $1 (the first field), named $1.file.

Code: Select all

#!/bin/ksh

AFS="${1}"
DIR="${2}"

awk -vAFS="${AFS}" -vDIR="${DIR}" 'BEGIN {FS=AFS;OFS=AFS;} 
{
  ## CTRY FILE NAME
  FNAME=$1".file";
  print $0 >DIR""FNAME;
}'

exit 0

[ray.wurlod]

Nice technique. Maybe >> would be better in the print command. This would not overwrite the file for each new record from that country.

[Ultramundane]

awk will keep track of what files it has used/opened. Once awk is stated, any new file used/opened will be overwritten the first time. Any file re-used after that will be appended.

Basically, > is the same thing as
>
followed by
>>

in a normal shell script.

Thanks,
Ryan

[ray.wurlod]

My bad. Failed to take into account that the print was an awk print - that is, in an awk block. Come's from replying after my bed time.