DSXchange

Hi,

I am new to datastage, earlier worked on some other ETL tools, I have develoed a job in which I am populating to a target file with 1000 columns. I have few set of rules which will populate say 200 columns of this file with after applying some business rules and rest 800 columns I will set to NULL. Question here is target file metadata keeps on changing from time to time since projects is in early stages, problem I am facing is whenever target metadata gets changed, I will load it in output of transformer, it will erase all rules I have written in transfomer. Is there any effective solution to this? Thanks

One solution that applies, if you do not require transformation on any newly added columns, is to investigate the use of "runtime column propagation" in the documentation.

At initial stages, you will not have all mapping defined. So it will not be possible to hold the columns.

Mapping them to any value may not be valid.

So it will be useful to include a modify stage and keep only the required columns for now. Later extend it.

I'm curious what kind of changes you are seeing in your target metadata and what exactly you mean by "load it in output of transformer"? If the changes are not substantial, make them manually. If you are only adding new columns then don't allow it to overwrite the existing ones.

Still, a better explanation of what you are doing would help get you better answers. Also realize that in most databases that I am aware of, you can simply not mention a column in the output link to have it set to null as that is the typical default value, so leaving them out of the job unless they actually need a derivation would probably help as well.

After the transformer, keep a copy stage before you write to a flat file. In this way you can shield the transformations for existing columns and map the new columns and write the derivations if needed.