Is there a way to Capture what Sequential files were Process

[joefonseca79]

I have a csv files that need to be processed 3 times a day at 9:30, 1:30 and 2:30
They are *.DAT files.

The Souce / Read Method = File Pattern
The job is processing on 1 Node to not mix up the rows.

Example Files

Filename Date/Time
File1.DAT 05/31/11 08:02 AM
File2.DAT 05/31/11 08:12 AM
File3.DAT 05/31/11 08:23 AM
File4.DAT 05/31/11 08:27 AM

They all are processed and then appended to each other and output a file OMA.DAT

Is there a way to track or write to a file the name of the file that DataStage is processing for an audit trail?

[arunkumarmm]

I'm not sure if there is a way to list the file names that were processed but you can use a 'ls' command with the same file pattern and print it to the log.

[joefonseca79]

Is LS command a unix / linux command? Or can you do that in seqence job?

-Joe

[arunkumarmm]

Its a UNIX command to list the files in a specify path or current directory. And YES, you have the command activity in the job sequence to execute any command/script.

[joefonseca79]

Here's what I put together but it's not writing anything to the file.

ls -a /data/DataStage/OPTICOM/SOURCE_FILES test.txt

and it's not writting to the file. Does that look wrong for a script?

-Joe

[arunkumarmm]

Well, you are not trying to write it to a output file in your command. Try this:

ls -a /data/DataStage/OPTICOM/SRC_FilePattern* > /data/DataStage/OPTICOM/FileNames.txt

[joefonseca79]

Great thanks for the example. I got this going. One last ? do you know how to make the file that I'm writing to dynamic to add date time in file name?

[arunkumarmm]

Add a parameter to you job, and pass the value (job start date or current timestamp) from the sequence and append the parameter to file name as required.

[ray.wurlod]

Or use the date command surrounded by backquotes as part of the file name.

Code: Select all

ls -a /data/DataStage/OPTICOM/SRC_FilePattern* > /data/DataStage/OPTICOM/FileNames_`date +"%Y%m%d"`.txt

[joefonseca79]

Here's the code that I did. it reads the contents of the folder and then writes that to LOGMMDDYYYYHHMM.

Then it takes all the files in the folder with *.DAT and MOVES them to the BACKUP folder.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Code: Select all

clear
echo "OPTICOM BACKUP START"
echo " "
echo " "
echo " " 
ls -all /data/DataStage/OPTICOM/SOURCE_FILES/*.* > /data/DataStage/OPTICOM/SOURCE_FILES/BACKUP/LOG.`date '+%m%d%Y-%H%M'`.txt
echo " "
echo " "
echo  "BATCH MOVE OF FILES TO BACKUP FOLDER"
echo " "
echo " "
mv /data/DataStage/OPTICOM/SOURCE_FILES/*.DAT /data/DataStage/OPTICOM/SOURCE_FILES/BACKUP
echo " "
echo " "
echo " "
echo " "
echo " "
echo " "
echo "OPTICOM BACKUP COMPLETE"
echo " "
echo " "

[chulett]

Hopefully, nothing new has shown up between the time you processed them and you ran this.

[joefonseca79]

I have Datastage that picks up all the files and processes them. Then it sends the file via email. Then the unix script goes and makes the file and moves them to the backup drive.

Orders are sent to us every 15 minutes. So I think I'll be good.