Unidata stage - advice on best approach

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
nmacolin
Participant
Posts: 19
Joined: Mon Jun 16, 2008 6:01 pm

Unidata stage - advice on best approach

Post by nmacolin »

Hi all,

Need some advice on the best approach here.
( Business Problem explained below )
Technical Problem.
I have a Unidata Job which accepts a parameter, based on the paramter extracts the correct data and Generates a file.
I would like to repeat the job for each list item in a file. There are 100's of items. I tried using a sequential file linked to the Unidata Stage but it assumes you want to write to unidata, which I'm not. So this fails.
Is there a better approach to this ? How can I call this Job for all the items in my file list. Given that the list is of unknown size and could be 100s to 1000s.

Business Problem
The situation. Unidata 7.1 where the application creates a new directory structure for each new project the company undertakes.
So we have a head directory and one level of subdirecories where each directory is either a Company Entity or a Project belonging to one of the companies.
I have figured out how to neavigate the file structures to produce a listing of all the Companies. Each Company has a corresponding directory which will have within it a listing of all the projects. The project name corresponds to a directory which then has all the data files ... Invoices, Purchases, Variations, Costs etc.
So the same file names are repeated under each directory structure to hold the projects data for each project.

You help is appreciated Nick
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Use a job sequence that has a Start Loop activity with the list of file names.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
nmacolin
Participant
Posts: 19
Joined: Mon Jun 16, 2008 6:01 pm

Post by nmacolin »

Hi Ray,

Can you elaborate.

I'm not sure how this is going to solve the problem.

I have a sequential file with a Company name per row.
I need to pass the Company Name one by one from the file into the Job as a parameter and repeat this Job for the total number of Companies listed in the file.

BTW, the loop stage Start Loop Tab : Loop Definition does not allow me to define a file as input so can you please elaborate.

Is there a better more logical way ?

Thanks
Nick
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You could use an upstream Execute Command activity to read the file, and convert the line terminators in the command output to yield - for example - a comma-separated list that could be used in the StartLoop activity.

Now that I have this extra information, I could suggest the alternative of a server job that processes the file of file names and uses the SDK routine UtilityRunJob to execute the job that processes each file.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

An 'upstream' UserVariables Activity stage can also be used to build a filename list and feed that to the Start Loop as a delimited list.
-craig

"You can never have too many knives" -- Logan Nine Fingers
nmacolin
Participant
Posts: 19
Joined: Mon Jun 16, 2008 6:01 pm

Unidata best approach - advice

Post by nmacolin »

Thanks guys.

Yes used an "Execute Command" stage to call a Windows Script file which converted the file into a comma delimited string and passed that to a "User Variables" stage and into the loop.

All I have to worry about now is the limitation on string length which can be processed by the Loop stage. Does anyone know what the limits are for the size of a string in the "Loop" stage ?

Thanks
Nick
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

No practical limit.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply