DSXchange

Posted: **Wed May 27, 2015 8:42 am**

Hi,
I have the following requirement and I need input for designing of my job

copy multiple files from source to target  directory with timestamp added to the Filename.

Current design to copy single file in a sequence

Code: Select all

UserVariable_Activity (Generate timestamp)----> Execute Command Activity (cp source dir file  target dir file_timestamp )

How can I modify the above design to copy multiple files in a single sequence ?

Thanks for the input

Posted: **Wed May 27, 2015 8:52 am**

Define 'multiple files' - multiples based on what, a wildcard pattern?

Me, I'd write a script.

Posted: **Wed May 27, 2015 9:03 am**

There is no wildcard pattern ..Each file is generated by individual job.

1.Alternative I can think is having a seperate link for each file

Code: Select all

UV -->Execute activity (file1) 
UV -->Execute activity (file2)
UV -->Execute activity  (file3)

2.Looping ?Can I implement this way?

Code: Select all

Start loop -->Read file1 -->generate timestamp -->cp file1 -->end loop 
Start loop -->Read file2 -->generate timestamp -->cp file2 -->end loop

Thanks

Posted: **Wed May 27, 2015 9:24 am**

As a general statement, I would think #2 would be more better.

Posted: **Wed May 27, 2015 1:56 pm**

@Chulett,

Draft Design :

Code: Select all

Start Loop  <-----------------------        End Loop
|                                                         |
|                                                         |
uv_activity ---------------------------Exec_Activity

In uv_activities define variables like below

Code: Select all

uv_timestamp  time_stamp logic
uv_file1  filename1
uv_file2  filename2
..so on

Exec_Activity

Code: Select all

cp sourcedir uv_file1.txt target dir uv_file1_uv_timestamp.txt

Loop type specifies is Numeric loop From 1 Step1 To 6 (6 times)

How do we implement the logic for the first iteration of loop read file1 ,second iteration of loop read file2 so on ?Is it possible ?

I am working on this above approach ..any input is appreciated

Thanks

Posted: **Wed May 27, 2015 7:43 pm**

You may want to reconsider using a numeric loop. Rather, can you build a delimited list of the filenames to process first? You can then pass that list to the Start Loop stage and it can loop through the list until it runs out of elements in the list. It can then leverage the built-in loop counter to pass filename #1 to the job / activity on the first iteration, filename #2 on the second iteration, etc etc.

Make any sort of sense? Is this the first time you've built any kind of a looping Sequence job? (just trying to level set here)

Posted: **Fri May 29, 2015 9:17 am**

We have a generic FTP job that loops for the number of files being transfered. Each invocation reads a text file that lists the filenames. I like simple and easy, so it goes like this:

Code: Select all

1) Find number of files in current list. I use UV stages as needed.
2) Set end of loop counter with the file count.
3) Read text file for the filename for each iteration of the loop.
4) I can add or delete files from each list as needed, no changes to DS code.

My file list looks like this:

Code: Select all

1:filename1
2:filename2
3:filename3
etc.

Something very important in my design is error handling and restartability. The main sequence stage -- FTP for me, cp command for you -- is checkpointed. I can rerun a job and have it restart at the failed iteration without further intervention.

DSXchange

Execute multiple cp commands in a Sequence job

Execute multiple cp commands in a Sequence job