How to dynamically change metadata without changing job
Moderators: chulett, rschirm, roy
How to dynamically change metadata without changing job
Hi,
We have a requirement where we need to identify those jobs that are impacted because of metadata change.
I know this can be achieved through Usage Analysis.
But we also want that if metadata changes for that job, the job should dynamically include the changed metadata in place of the existing meta data definition and execute without any code change to the job.
Is it possible with Runtime column propagation? Please suggest.
Can somebody please suggest some ideas on this or if you have encountered any situation like this one?
Thanks.
We have a requirement where we need to identify those jobs that are impacted because of metadata change.
I know this can be achieved through Usage Analysis.
But we also want that if metadata changes for that job, the job should dynamically include the changed metadata in place of the existing meta data definition and execute without any code change to the job.
Is it possible with Runtime column propagation? Please suggest.
Can somebody please suggest some ideas on this or if you have encountered any situation like this one?
Thanks.
Nash
Also, can you outline what the "dynamic metadata" is? Is it columns added or removed, changes in datatype, changes in arrangement/order of columns?...etc.
Some changes are handled automatically, some could be managed by things like RCP, some might require additional stages or transformations......... lots of possibilities here.
Ernie
Some changes are handled automatically, some could be managed by things like RCP, some might require additional stages or transformations......... lots of possibilities here.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
How to dynamically change metadata without changing job
The input file will have few mandatory columns and few optional columns. These optional columns may differ in each input file. The order of columns can also changeMike wrote:RCP could be used to some degree... but why?
You want to intentionally add a destabilizing feature to your application design?
Mike
The idea is to just map the columns from source and create an output in various formats lik csv, dat, etc
We donot want to edit the job once built. There will not be any complex transformations other than few simple business rules.
Please advise if we can achieve it thru RCP
Thanks
Nash
-
- Premium Member
- Posts: 151
- Joined: Fri Feb 13, 2009 4:19 pm
Hinash wrote:Thankschulett wrote:Somewhat. Any columns handled by RCP will be strictly "pass-through", i.e. with zero in the way of transformations. ...
i had worked in the same kind of situation. but mine was little different. we had a ods where all the transformations will be done and with this as a source we had another data warehouse where we need to populate the tables. we had nearly 200 target tables, but we had only 5 jobs that will handle the entire situation. We had the following conditions:
Meta data : run time propagation
SQL Query, other dates, key columns : everything will be passed as a parameter from a parameter file. if this is what you are looking for let me know, i think i can help you.
Thanks
Karthick
Karthick
-
- Premium Member
- Posts: 151
- Joined: Fri Feb 13, 2009 4:19 pm
Hinash wrote:Thankschulett wrote:Somewhat. Any columns handled by RCP will be strictly "pass-through", i.e. with zero in the way of transformations. ...
i had worked in the same kind of situation. but mine was little different. we had a ods where all the transformations will be done and with this as a source we had another data warehouse where we need to populate the tables. we had nearly 200 target tables, but we had only 5 jobs that will handle the entire situation. We had the following conditions:
Meta data : run time propagation
SQL Query, other dates, key columns : everything will be passed as a parameter from a parameter file. if this is what you are looking for let me know, i think i can help you.
Thanks
Karthick
Karthick
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: