Specifying column information at runtime.

gsherry1 · Post by **gsherry1** » Wed Oct 26, 2005 12:25 pm

I understand the features of RCP in only naming columns necessary for transformation. I also understand that I can specific column information at runtime using a schema file.

Is it possible to name a table definition from repository at runtime rather than a schema file? Anybody know if such a feature might be allowed in Hawk release?

Thanks in advance.

ray.wurlod · Post by **ray.wurlod** » Wed Oct 26, 2005 3:59 pm

It's not currently possible, and highly unlikely to be possible in the next release. DataStage is not intended to work with dynamic metadata.

A search of the forum will uncover at least one technique for tricking DataStage into doing so, but it's not possible "out of the box".

RCP is not possible in server jobs, but you marked your post as server even though you posted in the parallel forum.

kduke · Post by **kduke** » Wed Oct 26, 2005 5:56 pm

Ray, did they move this post? What you talkin about?

kwwilliams · Post by **kwwilliams** » Wed Oct 26, 2005 6:17 pm

gsherry1 has a heading listing his job type as server. He referred to RCP which is only available in Parallel and posted in parallel. My guess is that gsherry1 saved those preferences and didn't change them for this post. Just a guess though.

ray.wurlod · Post by **ray.wurlod** » Wed Oct 26, 2005 9:01 pm

Well spotted, Keith, and the reason that I made the comment in dark red rather than in black.

track_star · Post by **track_star** » Thu Oct 27, 2005 12:56 pm

Just a note on this....

gsherry--The exact feature you're looking for is not now, nor probably ever will be a feature of PX. The problem is that you're asking for an interface to the Universe repository from an osh script. The script gets created prior to runtime, and unless you're using a Custom operator, it can't make that connection to select a specific table definition and use the column definitions in the currently running job. With that said......

Dynamic metadata IS possible in PX, you just have to create a Custom stage (operator) that can read it. The coding is a combination of C++ and Orchestrate functions/classes. It still won't do what you're thinking of, but dynamic metadata is possible (and not really a trick).

I'm not sure what "trick" Ray was referring to (sorry I didn't search the forum to find it), but one that I am aware of involves a schema file being called and then using RCP. While this can work in very simple jobs (seq file to database), it won't work for everything. There are some instances where the GUI requires more info in the job design so that it can create the osh script correctly. If the columns aren't explicitly defined in the job, it won't compile.

ray.wurlod · Post by **ray.wurlod** » Thu Oct 27, 2005 4:06 pm

The "trick" - which will work in server or in parallel jobs - is to create "generic" column definitions then to use user-defined SQL the whole of which is supplied through a job parameter (such as #SQLStatement#). This may, for example, need to select constants to fill the unneeded columns or columns that don't actually exist in the currently-processed table.