Convert opd to Buildop Stage

[sadnemous]

Though we have datastage 8.5 (GUI) we use the power of parallel processing directly using osh script. e.g. simple script we write:

Code: Select all

osh -echo "
fc_ModifyLocation
< [ds] input.ds
>| [ds] output.ds
" 

After spending time, now I am really curious to switch GUI from terminal. I have been successful to create simple job.

I am stuck with opd. We have loads of existing opd. I wanted to import an exisiting one. e.g.

Code: Select all

$operator fc_ModifyLocation
$class  fc_ModifyLocation

$input record( 
  loc: string[10];
  inrec: *;
)

$output record( 
  location: string[10];
  outrec: *;
)

$pre

$action
  out.location = in.loc
$post

$end

I tried a lot to convert this opd:
1. I created simple buildop stage in datastage
2. I had to refer a table
3. successfully build and tested by using in a simple job (input dataset->buildop->output dataset)
4. export the whole build stage (buildop with table defination) into local directory as xml
5. learnt from the xml, and then replicated similar xml for said opd (fc_ModifyLocation).
6. then tried to import this xml, probably successfully imported but the problem with inrec: * and outrec: *.
7. did some googling, few said to use outrec[]: string; which is not helping.


Could you please help whether :
1. I am right direction
2. I am doing anything wrong
3. Is there better way to import opd (may be accessing metadata, and inserting information or anything else)

Thanks for reading so far and giving me valuable time. Please help.

Soumen

[ArndW]

That looks like a heck of a lot of work and complexity for a DataStage designer "modify" stage with one mapping that would take about 30 seconds to put into a job.

Are you sure you wish to solve this using buildops?

[sadnemous]

That step is only an example ( OR rather simplest example)

Code: Select all

osh -echo "
  fc_ModifyLocation
  <  [ds] input.ds
  >| [ds] output.ds
"

My problem is with inrec: *; and outrec: *;

[ArndW]

You've stated that you have problems with the "*". What does that mean? Error messages?

[chulett]

sadnemous: not sure why the fascination with the color blue, best to stick with

Code: Select all

[/b] tags for things you should preserve the formatting of, like your osh code and not worry about what color to make things. :wink: 

And there's no need to quote the post you are replying to, it's all right there already, so please look for the [b][color=orange]Reply to topic[/color][/b] link rather than [i]Reply with quote.[/i]

Thanks!

ps. Oh, and if you accidentally double-post (which happens here quite a bit) you can delete your own posts [i]as long as they haven't been replied to,[/i] so just delete the last one rather than blank out the first one. FYI.

[chulett]

That step is only an example ( OR rather simplest example)

Usually the simplest example gets you the simplest response and that typically doesn't work out well for either side. Best to put all of your cards on the table right up front so we know what we're dealing with. IMHO.

As Arnd noted, can you supply some specifics around the issues you are having? You've noted what you are having problems with but not the actual problem. Or problems.

[jwiles]

Within the context of a buildop (opd file) or a custom operator, inrec: * and outrec: * are called "schema variables" and represent all columns which have not been explicitly named within the schema (similar to Runtime Column Propagation in the GUI). For more detail, look in the IIS 8.5 Custom Operator Reference document, available from here.

Do not include the schema variables in your hand-built job export XML. They are only valid within the context of the opd file. If you need your new BuildOps to transfer all unnamed columns (i.e. RCP), set RCP to True within the Interfaces tab in the BuildOp GUI. Refer to the Parallel Job Advanced Developer's Guide, available at the same link given above.

Regards,

[sadnemous]

Thank you jwiles for useful information. You have directed me right way. Now one of my team-mate developing a Perl script to convert our existing opd into proper formatted xml file so that we can import in DataStage Client. So, if anyone like me is trying to convert opd to buildop stage here are the steps:

1. parse the $input and $output section and create the table definition
2. parse the rest of opd and create xml, for input and output table definition refer to the just created table definition (point 1)
3. go to DataStage Client -> import -> DataStage Components (XML) -> select your xml file. It will import buildop stage along with table definitions.

Thanks,
Soumen