Sequencial file name capture in a job

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

jhmckeever
Premium Member
Premium Member
Posts: 301
Joined: Thu Jul 14, 2005 10:27 am
Location: Melbourne, Australia
Contact:

Post by jhmckeever »

For older versions of DS (where this functionality wasn't available) I use the following script. It takes a file pattern and echos all lines from all matching files with each line prefixed with the name of the source file:

Code: Select all

#! /usr/bin/ksh
#------------------------------------------------
# Usage: getFiles.ksh {FilePattern}
#------------------------------------------------

FilePattern="$1"
if [ "$FilePattern" = "" ]
   then echo "Must supply a file pattern"
   exit 1
fi

# Iterate over files matching the supplied pattern
while [ "$1" != "" ]
do
   # Get the filename
   SetName=`echo $1`

   # Iterate over each line of the file
   cat $1 |
   while read Job
   do
      echo "$SetName    $Job"
   done # read line (from file)

   shift

done # read Filenames
(It might look a little weird in places as I've stripped out some funtionality specific to our installation, but it should still work fine.)

HTH, :)
J.
<b>John McKeever</b>
Data Migrators
<b><a href="https://www.mettleci.com">MettleCI</a> - DevOps for DataStage</b>
<a href="http://www.datamigrators.com/"><img src="https://www.datamigrators.com/assets/im ... l.png"></a>
kksr
Participant
Posts: 37
Joined: Fri Dec 30, 2005 5:57 am

Patch details for reading multiple files using file pattern

Post by kksr »

Hi ,

We are also facing similar issue while reading multiple files using file pattern option , but we are unable to find the mentioned patch (96576 for DS EE 7.5.1A ) in IBM web site .Please share the Patch/ fix details for this issue .

DS Versionn : 7.5.1A
OS : AIX


Thanks
Kiran
KKSR
Sreenivasulu
Premium Member
Premium Member
Posts: 892
Joined: Thu Oct 16, 2003 5:18 am

Post by Sreenivasulu »

Different approach ->
You can use the loop stage to the get the file name ?
1.Use Execute Command Activity to fire a shell command 'ls *.csv' (*.csv is the pattern)
2.Use loop stage with list
3.Use the UserVariable activity to get the file name

Regards
Sreeni
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Re: Patch details for reading multiple files using file patt

Post by chulett »

kksr wrote:we are unable to find the mentioned patch (96576 for DS EE 7.5.1A ) in IBM web site. Please share the Patch/ fix details for this issue.
That would need to come from your official support provider.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply