Handling multipls files.

[naren6876]

I have multiple files in my production folder .
how do i handle all the files one after other.
After processing the file i need to remove it from the production folder.

I have designed a sequence job with the nested condition ,based on the filename it will call the JOBS.At present iam using the job parameter as filename for the nested condition.Here manually iam giving the filename.It works fine.
I want to automate this.

plz let me know

[DSguru2B]

You will need to code a job sequence utility to automate the process.
regards

[leomauer]

You can do it with Unix script.
Something like that:

Code: Select all

cd [folder name] 
for fn in $(ls)
do
  dsjob .... [parmeter name]=$fn  ... [DS job name] 
  rm $fn
done

[ray.wurlod]

If you have version 7.5 you can use a StartLoop activity based on a list of file names.

[naren6876]

Hi Ray,
Could you please explain me bit more about Start Loop activity.
Which type of Loop do i need to use?. either Numeric or other one. If the second one what should i pass in the delimited values. and How ?.

TIA

If you have version 7.5 you can use a StartLoop activity based on a list of file names.

[dnsjain]

Or you can write a bacth job and so some combination of DS Basic and Unix command coding.

Here is the code for Batcj job.

*Get list of files from input directory.
vDirPathName = "/input/"
vCmd = "ls -rt " : vDirPathName
Call DSExecute("UNIX", vCmd, ScreenOutput, SystemReturnCode)
If SystemReturnCode <> 0 Then
Message = "Error when executing command: "
Call DSLogWarn(Message, "ls- lrt")
End
FileList= Convert(char(10),@FM,ScreenOutput)
FileCnt = DCOUNT(FileList, @FM) -1

*For every file present in the input diectory.
If FileCnt > 0 Then
FOR idx = 1 TO FileCnt
* Call your job in loop

Next idx
End If

* Execute mv command to move the file to different directory

Dinesh

[ray.wurlod]

Use the "in list" loop rather than the counted loop.

The current index value is available to a downstream Job Activity, so you can make the file name a job parameter in the job invoked by it.

I was assuming an invariant comma-delimited list of file names when I made the initial suggestion; would have to investigate using a job parameter for this list. Why don't you have a play, and let us know how you fared?

[naren6876]

Hi Dinesh,
I have used your code and am able to run the job.Here iam calling the sequence job in the loop which accepts a filename as Job Parameter.
But, in this code it is not passing the filename to the sequence job.

How do i pass the filename as parameter to sequence job?

TIA

Or you can write a bacth job and so some combination of DS Basic and Unix command coding.

Here is the code for Batcj job.

*Get list of files from input directory.
vDirPathName = "/input/"
vCmd = "ls -rt " : vDirPathName
Call DSExecute("UNIX", vCmd, ScreenOutput, SystemReturnCode)
If SystemReturnCode <> 0 Then
Message = "Error when executing command: "
Call DSLogWarn(Message, "ls- lrt")
End
FileList= Convert(char(10),@FM,ScreenOutput)
FileCnt = DCOUNT(FileList, @FM) -1

*For every file present in the input diectory.
If FileCnt > 0 Then
FOR idx = 1 TO FileCnt
* Call your job in loop

Next idx
End If

* Execute mv command to move the file to different directory

Dinesh

[naren6876]

Hi Ray,
As you suggest I have used the Start Loop Activity and End Lop Activity.
When i supply the comma delimited file names it works fine.
When i use the job parameter for the List of filenames, thats also works fine.
My question is How to pass those list of filenames.By manually typing?.
But, i need to automate this job?.

Any suggessions pls

TIA

Use the "in list" loop rather than the counted loop.

The current index value is available to a downstream Job Activity, so you can make the file name a job parameter in the job invoked by it.

I was assuming an invariant comma-delimited list of file names when I made the initial suggestion; would have to investigate using a job parameter for this list. Why don't you have a play, and let us know how you fared?

[ray.wurlod]

Create an earlier Routine Activity to return the comma-separated list of filenames. When supplying the job parameter value, click on Use Job Parameter and choose the return value from the Routine Activity.

You could also use an Execute Command Activity, but you'd have to create a script to turn the output of the ls command into a comma-separated list.

[naren6876]

Hi Ray,

I want to use this code in Routine Activity.

FilePath = "/input/"
Call DSExecute("UNIX", "ls -rt FilePath", Output, SystemReturnCode)
If SystemReturnCode <> 0 Then
Message = "Error when executing command: "
Call DSLogWarn(Message, "ls -rt")
End

How to convert this Output into comma-separated list of filenames?.

TIA
Naren

Create an earlier Routine Activity to return the comma-separated list of filenames. When supplying the job parameter value, click on Use Job Parameter and choose the return value from the Routine Activity.

You could also use an Execute Command Activity, but you'd have to create a script to turn the output of the ls command into a comma-separated list.

[ray.wurlod]

Code: Select all

FUNCTION GetFileNames(FilePath)

Command = "ls -1rt " : FilePath
Ans = ""

* Capture a list of file names (note option "1" in ls command, so that
* output contains ONLY a list of file names, one per line).
Call DSExecute("UNIX", Command, Output, SystemReturnCode) 

* Ensure UNIX command executed successfully.
If SystemReturnCode <> 0 
Then 
   Message = "Error when executing command: " : Command
   Call DSLogWarn(Message, "Obtaining File Names")  
   Ans = ""
End 
Else

   * Result of command is a field mark (@FM) delimited string, one field
   * for each line of output.  We process one file name at a time using an
   * uncounted loop (since we have no a priori knowledge of the number
   * of file names to be processed..
   Loop

      Remove FileName From FileNameList Setting MoreFiles

      * Not interested in blank lines.
      If Trim(FileName) > " "
      Then
         Ans<-1> = FileName  ; * add element to dynamic array
      End

   While MoreFiles
   Repeat

   * Finally, convert dynamic array delimiter characters (@FM) into 
   * commas, so that result is a comma-delimited list.
   Ans = Convert(@FM, ",", Ans)

End

RETURN(Ans)

[palmeal]

I'm looking to do something similar to the code that Ray has posted above but rather than listing multiple filenames I want to read in one file and return a list of all entries in that file in the final return clause. I then want to use this list in a StartLoop (#Routine_Activity_37.$ReturnValue#) to process through each entry one at a time.
I previously had this scenario working with a bunch of Server Jobs and Basic code but I've been advised by an Ascential consultant to change all this to Job Sequences.

In Rays code I'm failing to see how the filenames are pulled out from the return of the DSExecute and into the FileName variable - I'm sure the code is fine and it is just my understanding that is lacking. I have plugged this code into a test routine (keeping the ls -lrt to get it working) and run through the code (I made sure that I had four files to pick up).

When I run the code the final return brings back an empty string and if I add a message to be returned each time the process goes through the loop I only get one message returned.

I also tried writing the value of the output string to the log and it showed job_seq1..JobControl (alan.dat): delme.dat. I expected 4 filenames to returned here.


I can find documentation regarding DSExecute but nothing at all for the line
Remove FileName From FileNameList Setting MoreFiles. Remove..From..Setting looks like a construct that I can't find anywhere and I don't understand how

Can anyone point me in the right direction where I'm going wrong.

Any help is appreciated.

[palmeal]

Seems that I have achieved by what I want to do by doing the following :-

Command = "ls -1rt *.dat"

* Capture a list of file names (note option "1" in ls command, so that
* output contains ONLY a list of file names, one per line).
Call DSExecute("UNIX", Command, Output, SystemReturnCode)

* Work out how many rows have been returned
num_rows = count(Output, @FM)
Call DSLogInfo("num_rows", num_rows)

Ans = ""
Counter = 1
Loop While Counter <= num_rows
If Counter = 1 then
Ans = Output<Counter>
End
Else
Ans = Ans : "," : Output<Counter>
End

Counter += 1
Repeat

Call DSLogInfo("Ans", Ans)

Return(Ans)

Obviously I need to add a check for SystemReturnCode.

[naren6876]

instead of
Remove FileName From FileNameList Setting MoreFiles.
use Remove FileName From Output Setting MoreFiles you will be all set.

This will work like, First it reads from the Output and pass this value to FileName and then delete that value from the Output.So that the pointer always ponted to firstvalue in the Output.Then we are return that in a array.


Loop

Remove FileName From Output Setting MoreFiles

* Not interested in blank lines.
If Trim(FileName) > " "
Then
Ans<-1> = FileName ; * add element to dynamic array
End

While MoreFiles
Repeat


[quote][/quote]