Consecutive runs of same jobs with Toggled filenames

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
gateleys
Premium Member
Premium Member
Posts: 992
Joined: Mon Aug 08, 2005 5:08 pm
Location: USA

Consecutive runs of same jobs with Toggled filenames

Post by gateleys »

I have a job where both input and output are sequential files with the same column definitions-

Code: Select all

file1 ---->Tfr ----> file2
How can I run this file multiple times (say 10 times), one after the other such that in every other run, the 2 files switch their names? That is, if the above shown design is my first run, then in 2nd run, file2 would be input filename and file1 will be output filename-

Code: Select all

2nd run (filenames toggle):
file2 ---->Tfr ----> file1

3rd run (filenames toggle):
file1 ---->Tfr ----> file2

4th run (filenames toggle):
file2 ---->Tfr ----> file1

5th run (filenames toggle):
file1 ---->Tfr ----> file2

and so on..
Thanks
gateleys
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You need some kind of persistent "memory" of the file most recently processed. Because the file names will be parameters you can get this from the most recent "job started" log entry (in a Routine activity). Or you can design to store it in a third file somewhere. Pick up the name in a Routine or Execute Command activity, and update it once the job completes successfully (perhaps as an after-job subroutine invoking ExecSH to use echo to send the parameter value into the file).
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
parag.s.27
Participant
Posts: 221
Joined: Fri Feb 17, 2006 3:38 am
Location: India
Contact:

Re: Consecutive runs of same jobs with Toggled filenames

Post by parag.s.27 »

gateleys wrote:I have a job where both input and output are sequential files with the same column definitions-

Code: Select all

file1 ---->Tfr ----> file2
How can I run this file multiple times (say 10 times), one after the other such that in every other run, the 2 files switch their names? That is, if the above shown design is my first run, then in 2nd run, file2 would be input filename and file1 will be output filename-

Code: Select all

2nd run (filenames toggle):
file2 ---->Tfr ----> file1

3rd run (filenames toggle):
file1 ---->Tfr ----> file2

4th run (filenames toggle):
file2 ---->Tfr ----> file1

5th run (filenames toggle):
file1 ---->Tfr ----> file2

and so on..
Thanks
gateleys

Hi,

I had same task. What i did is little different. Actually, i used loop activity jobs in sequence,

between these loop activities i used a routine job that reads a specific column from a file, the data in the column becomes the part of source file name as well as target file name for the next loop iteration.

for example.

Code: Select all

Start Loop-------->UserVariableStage----->READDATA_routine
                  |                                            (it reads data from file
                  |                                             to use it as next file name)
                  |                                                       |
                  |                                                       |
                  V                                                      V
         EndLoop<--------------------------MAIN JOB

here my initial src file name during 1st loop iteration is JAN_2006.rpt
then during next iteration its FEB_2006. thus JAN, FEB... comes from data.


Hope it helps a bit
Thanks & Regards
Parag Saundattikar
Certified for Infosphere DataStage v8.0
kumar_s
Charter Member
Charter Member
Posts: 5245
Joined: Thu Jun 16, 2005 11:00 pm

Post by kumar_s »

If number of iteration is know, the job can be made as multiple instance and individual parameters with different values can be passed.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
parag.s.27
Participant
Posts: 221
Joined: Fri Feb 17, 2006 3:38 am
Location: India
Contact:

Post by parag.s.27 »

kumar_s wrote:If number of iteration is know, the job can be made as multiple instance and individual parameters with different values can be passed.
Actually in my case, the no. of iterations are not known.

there is a column in which date is comming as and i check for month in that date. if the month is new and year is same or month is same but year is new and so on.

thus my iterations are not fixed.

so after reading data when i get null for new value or EOF is reached then iterations syops
Thanks & Regards
Parag Saundattikar
Certified for Infosphere DataStage v8.0
gateleys
Premium Member
Premium Member
Posts: 992
Joined: Mon Aug 08, 2005 5:08 pm
Location: USA

Post by gateleys »

I wish I had the fortune to have DS7.5.x available. My jobs are based on 7.0 version, hence there are no start loop and end loop stages. Besides, the number of iterations are not known, and so multiple instances may not be a good idea. In another post, Arnd suggested that I do it using creating my own loop using the job control code generated via an empty canvas. Will have to try that. Anyways, thanx guys. :)
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

gateleys wrote:In another post, Arnd suggested that I do it using creating my own loop using the job control code generated via an empty canvas. Will have to try that.
'Fraid that will be your only option in 7.0.
-craig

"You can never have too many knives" -- Logan Nine Fingers
Post Reply