Parallel Equivalent to UtilityRunJob

Developer9 · Post by **Developer9** » Mon Apr 07, 2014 3:33 pm

Is there a Parallel routine or function equivalent to UtilityRunJob server routine? I need to run a datastage parallel job from another job.

chulett · Post by **chulett** » Mon Apr 07, 2014 3:37 pm

Not unless you write one, as far as I know. If you really need to do this, I'd probably look into using it via a BASIC Transformer in a Server Shared Container. Or if there's no burning need for the driving job to be a PX job, create it as a Server job.

ray.wurlod · Post by **ray.wurlod** » Mon Apr 07, 2014 4:55 pm

What Craig said, but stronger.

If you were to write your own you would need to encapsulate the DataStage C API in order to have the ability to set parameters, request runs, discover status, and so on.

A review of the source code for UtilityRunJob will give you the logic.

You would also need to take steps to ensure that the routine is not called from every node, since only once specific instance of a DataStage job can be run at any one time. (Specific instance includes multi-instance with a specific invocation ID.)

Developer9 · Post by **Developer9** » Mon Apr 07, 2014 8:10 pm

Chulett thanks for the reply, my input files has datastage job names and i have a server job which reads the file and runs the job in the file. We are now getting a request for using multiple files with multiple job names, i didnt see any option in server jobs to handle file patterns, if my server job supports reading multiple files then this will be my first preference. any thoughts??

chulett · Post by **chulett** » Mon Apr 07, 2014 9:24 pm

From what I recall, there isn't a direct option in the Server Sequential File stage like there is on the Parallel side to read files via a File Pattern. However, it can be handled via the Filter option. Any command line option that can deliver rows to standard out will feed the stage as if it was reading a single file, so (for example) something as simple as:

Code: Select all

cat MyFile*.txt

Could be made to work with your pattern of choice. Now, realize that if you have a header record in each file then it will need to be dealt with in some fashion. We've used Perl or awk in the past to read files via a pattern and strip the header from each on the way in. It also becomes a little trickier if you need to know what filename each record came from but again something like Perl or awk would allow you to add that as part of each record, a new column with the filename. Heck, we even had one developer do all that in a Korn shell script... it all depends on where your skill sets are and what you feel comfortable maintaining.

ray.wurlod · Post by **ray.wurlod** » Mon Apr 07, 2014 9:33 pm

As Craig notes, use a server job. In the Sequential File stage specify that it uses a Filter command and setup the filter command as cat file1 file2 file3 ...

The stage will read stdout of the filter command rather than the file name you (have to) supply.

Add downstream logic to discard any header rows. Use extended column properties in the Sequential File stage to handle any "missing" columns from those header rows, or perhaps read the whole line into a single VarChar and parse in a downstream Transformer stage.

chulett · Post by **chulett** » Tue Apr 08, 2014 11:41 am

It is worth noting (again) that the filename property is required. So rather than putting something meaningless in there, understand that the stage will automatically combine it with the filter command to produce the stream of data that the stage will read.

Meaning, you should be able to put your pathed filename with whatever pattern you need as the filename and "cat" by itself as the Filter and it will automatically combine the two to "cat /your/files/pattern*.*" when it executes.