Orchestrate script calling issue

dsusr · Post by **dsusr** » Thu Sep 15, 2005 4:12 am

Hi All,

Although osh is aframework on which DataStage PX runs but is there any way by which we can call the osh script from the DataStage job.

My requirement is first to do the processing in osh script and give the output of that orchestrate script to a DS job but there is a constraint that we cannot write the data to the disk.

So will there be any way to call it or should we continue the processing logic of Job also in osh script.

Regards
dsusr

ray.wurlod · Post by **ray.wurlod** » Thu Sep 15, 2005 4:17 am

Of course it's possible. But there's no point.

dsusr · Post by **dsusr** » Thu Sep 15, 2005 4:40 am

ray.wurlod wrote:Of course it's possible. But there's no point.

Hi Ray,

Yes there is no point in using DS job and we could have done the processing in osh script only but we are just trying to incorporate that functionality in osh that we are not able to implement in DS job.

So the use of DataStage job along the osh script will help in maintaining the system since we can modify the functionality in GUI easily if required.

Can you please tell the way by which we can call the osh from the job.

Thanks
dsusr

Eric · Post by **Eric** » Thu Sep 15, 2005 4:53 am

What osh functionality are you trying to use that you cannot acheive using the DataStage GUI to create a job?

dsusr · Post by **dsusr** » Thu Sep 15, 2005 5:15 am

Eric wrote:What osh functionality are you trying to use that you cannot acheive using the DataStage GUI to create a job?

Hi Eric,

Our requirement is to dynamically create a transformation at runtime. For example there can be n different formats of file which I have to map to fixed output format.

We will get the Schema for each file and also the mapping of input field to output field at runtime only. So what I am doing is generating the expression file at runtime and using that expression file in transform operator in osh.

I think this change in transformation rule is not possible at runtime in a job but If you think it is possible to do through GUI, then suggestions are welcome

Thanks & Regards
dsusr

nnes · Post by **nnes** » Thu Sep 15, 2005 7:03 am

dsusr wrote: I think this change in transformation rule is not possible at runtime in a job but If you think it is possible to do through GUI, then suggestions are welcome

Thanks & Regards
dsusr

Which got me thinking, and unfortunately I think it is not possible. You could generate the xml for the job programatically and then import, compile and run it from the command line automagically.

Nestr

dsusr · Post by **dsusr** » Fri Sep 16, 2005 6:51 am

Hi All,

Is there any way to give the output of Osh script to DataStage job without creating a persistent dataset i.e. without writing the data to the disk.

Or Is there any way to call the Osh script from the DataStage job.

Nestr,

It is true that we can implement the same ogic through xml only but the problem is that I dont have xml knowledge and also I have the osh script in place.

Thanks & Regards
dsusr

ray.wurlod · Post by **ray.wurlod** » Fri Sep 16, 2005 7:42 am

You could possibly invoke it using a Wrapper stage. The important thing is to handle stdin and stdout properly.

dsusr · Post by **dsusr** » Fri Sep 16, 2005 8:13 am

ray.wurlod wrote:You could possibly invoke it using a Wrapper stage. The important thing is to handle stdin and stdout properly.

Hi Ray,

I have tried using the wrapper stage but the problem with wrapper stage is that it can take only unix commands and also it will run with the default sheel of server and not with osh.

Since orchestrate script can run in only Orchestrate shell (Osh) this possiblity is not working.

Can someone give any suggestions of implementing the above logic through DS Jobs.

Thanks
dsusr

ray.wurlod · Post by **ray.wurlod** » Fri Sep 16, 2005 3:56 pm

osh is the name of a shell, just like sh, csh, ksh. You can invoke it from UNIX, therefore you can invoke it from a Wrapper stage.

dsusr · Post by **dsusr** » Sun Sep 18, 2005 10:40 pm

ray.wurlod wrote:osh is the name of a shell, just like sh, csh, ksh. You can invoke it from UNIX, therefore you can invoke it from a Wrapper stage.

Hi Ray,

I have tried invoking the shell by writing the following command in one file:-
osh "transform -flag run -dir /home/dsadm/testDS/OshTestSal -name trafile.so"

and calling this file from Wrapper stage. When I am calling this script from wrapper stage, job is getting aborted by giving the following error at runtime:-

main_program: APT_PipeConnection: new connection 1 requests file descriptor -1, which was also requested by connection 0
main_program: Fatal Error: Incompatible connection - operator type = APT_OSL_SubprocOperator

Is there any way to resolve this ?

Thanks
dsusr

ray.wurlod · Post by **ray.wurlod** » Mon Sep 19, 2005 1:02 am

Doubt it. Looks like recursive calls to osh (one being your job, the other being the invocation from a Wrapper stage) aren't permitted. But knowing that will probably be valuable one day.

What happens - even though you'd need intermediate persistent data set - if you fork a separate, background osh process, perhaps using nohup?

This is another guess - I don't have access to EE at the moment.

cmmurari · Post by **cmmurari** » Mon Sep 19, 2005 4:15 am

Hi All,

Its simple in EE 7.5 . We have Datastage Plug-in Stage Command Stage The Command Stage is an active stage that can execute various external
commands, including DataStage engine commands osh, programs, and jobs from anywhere in the Ascential DataStage data flow. You can execute any command, including its arguments, that you can type to the shell of the operating system,such as Windows or UNIX. Examples include Perl scripts, DOS batch files, UNIX scripts, and other command-line executable programs that you can call if they are not interactive.
A custom client GUI is available for the Command Stage Plug-in.

Cheers,
krish