xml metadata importer tool

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
parvathi
Participant
Posts: 103
Joined: Wed Jul 05, 2006 4:48 am
Contact:
Contact parvathi

xml metadata importer tool

Post by parvathi »

Hi
I have exported a job design in xml format...
the format of xml output is as follows
/*********************************************
<?xml version="1.0" encoding="UTF-8" ?>
- <DSExport>
<Header CharacterSet="CP1252" ExportingTool="Ascential DataStage Export" ToolVersion="4" ServerName="10.237.237.36" ToolInstanceID="DWEucanDev" Date="2006-08-28" Time="14.47.09" ServerVersion="7.5.1" />
- <Job Identifier="xmltest" DateModified="2006-08-28" TimeModified="14.46.06">
- <Record Identifier="ROOT" DateModified="1899-12-30" TimeModified="00.00.01" Type="JobDefn" Readonly="0">
<Property Name="Name">xmltest</Property>
<Property Name="Description" />
<Property Name="NextID">1</Property>
<Property Name="Container">V0</Property>
<Property Name="FullDescription" />
<Property Name="JobVersion">50.0.0</Property>
<Property Name="BeforeSubr" />
<Property Name="AfterSubr" />
<Property Name="JobControlCode" />
<Property Name="ControlAfterSubr">0</Property>
- <Collection Name="MetaBag" Type="MetaProperty">
+ <SubRecord>
<Property Name="Owner">APT</Property>
<Property Name="Name">AdvancedRuntimeOptions</Property>
<Property Name="Value">#DSProjectARTOptions#</Property>
<Property Name="Conditions" />
</SubRecord>
</Collection>
<Property Name="NULLIndicatorPosition" />
<Property Name="OrchestrateCode" />
<Property Name="NULLIndicatorValue" />
<Property Name="IsTemplate">0</Property>
<Property Name="FromTemplate" />
<Property Name="DBMSSystemName" />
<Property Name="NLSMapName" />
<Property Name="NLSLocale" />
<Property Name="JobType">0</Property>
<Property Name="PlatformType" />
<Property Name="Category">Learning</Property>
<Property Name="CenturyBreakYear">30</Property>
<Property Name="CobolProgramName" />
<Property Name="CompileJCLName" />
<Property Name="RunJCLName" />
<Property Name="DBMSUserName" />
<Property Name="DBMSPassword" />
<Property Name="NextAliasID">2</Property>
<Property Name="ParameterFileName" />
<Property Name="ParameterFileDDName">DD00001</Property>
<Property Name="ReservedWordCheck">1</Property>
<Property Name="TransactionSize" />
<Property Name="CodeGenLocation" />
<Property Name="ValidationStatus">0</Property>
<Property Name="Uploadable">0</Property>
<Property Name="PgmCustomizationFlag">0</Property>
<Property Name="JobReportFlag">0</Property>
<Property Name="AllowMultipleInvocations">0</Property>
<Property Name="DateFormat" />
<Property Name="JobSeqCodeGenOpts" />
<Property Name="Act2ActOverideDefaults">0</Property>
<Property Name="Act2ActEnableRowBuffer">0</Property>
<Property Name="Act2ActUseIPC">0</Property>
<Property Name="Act2ActBufferSize" />
<Property Name="Act2ActIPCTimeout" />
<Property Name="ExpressionSemanticCheckFlag" />
<Property Name="TraceOption" />
<Property Name="EnableCacheSharing">0</Property>
<Property Name="OSHCollatingSequence" />
<Property Name="TeradataTDPid" />
<Property Name="TeradataAccountId" />
<Property Name="UserName" />
<Property Name="Password" />
<Property Name="RuntimeColumnPropagation">0</Property>
<Property Name="RelStagesInJobStatus">-1</Property>
<Property Name="WebServiceEnabled">0</Property>
<Property Name="MFProcessMetaData" />
<Property Name="MFProcessMetaDataMachineProfile" />
<Property Name="MFProcessMetaDataIP" />
<Property Name="MFProcessMetaDataUserName" />
<Property Name="MFProcessMetaDataPassword" />
<Property Name="MFProcessMetaDataXMLLocation" />
<Property Name="MFProcessMetaDataXMLFilename" />
<Property Name="MFProcessMetaDataXMLFileExchangeMethod" />
<Property Name="IMSProgType" />
<Property Name="CopyLibPrefix">ARDT</Property>
</Record>
- <Record Identifier="V0" DateModified="1899-12-30" TimeModified="00.00.01" Type="ContainerView" Readonly="0">
<Property Name="Name">Job</Property>
<Property Name="Description" />
<Property Name="NextID">1</Property>
<Property Name="InputPins" />
<Property Name="OutputPins" />
<Property Name="IsTopLevel">0</Property>
<Property Name="StageList">V0S0|V0S1|V0S2</Property>
<Property Name="StageXPos">144|288|504</Property>
<Property Name="StageYPos">240|240|240</Property>
<Property Name="StageTypes">CSeqFileStage|CTransformerStage|CSeqFileStage</Property>
<Property Name="NextStageID">5</Property>
<Property Name="SnapToGrid">1</Property>
<Property Name="GridLines">0</Property>
<Property Name="ZoomValue">100</Property>
<Property Name="StageXSize">48|48|48</Property>
<Property Name="StageYSize">48|48|48</Property>
<Property Name="ContainerViewSizing">-032 0000 0559 0576 0000 0000 0000 0000</Property>
</Record>
- <Record Identifier="V0S0" DateModified="1899-12-30" TimeModified="00.00.01" Type="SeqFileStage" Readonly="0">
<Property Name="Name">Sequential_File_0</Property>
<Property Name="Description" />
<Property Name="NextID">2</Property>
<Property Name="InputPins" />
<Property Name="OutputPins">V0S0P1</Property>
<Property Name="Directory" />
<Property Name="UnixFormat">2</Property>
<Property Name="NLSMapName" />
<Property Name="PipeStage">0</Property>
<Property Name="UnicodeBOM">1</Property>
<Property Name="UnicodeSwapped">1</Property>
<Property Name="AllowColumnMapping">0</Property>
<Property Name="WithFilter">0</Property>
<Property Name="StageType">CSeqFileStage</Property>
</Record>
- <Record Identifier="V0S0P1" DateModified="1899-12-30" TimeModified="00.00.01" Type="SeqOutput" Readonly="0">
<Property Name="Name">DSLink3</Property>
<Property Name="Description" />
<Property Name="Partner">V0S1</Property>
<Property Name="FileName">D:\Angel_DS\Angel\cnt_in.txt</Property>
<Property Name="ColDelim">,</Property>
<Property Name="QuoteChar">"</Property>
<Property Name="ColHeaders">0</Property>
<Property Name="NullString" />
<Property Name="FixedWidth">0</Property>
<Property Name="ColSpace">0</Property>
<Property Name="EnforceMetaData">0</Property>
<Property Name="Readtimeout">0</Property>
- <Collection Name="Columns" Type="OutputColumn">
- <SubRecord>
<Property Name="Name">sNo</Property>
<Property Name="Description" />
<Property Name="DataType" />
<Property Name="SqlType">4</Property>
<Property Name="Precision">10</Property>
<Property Name="Scale">0</Property>
<Property Name="Nullable">0</Property>
<Property Name="KeyPosition">1</Property>
<Property Name="DisplaySize">1</Property>
<Property Name="Derivation" />
<Property Name="Group" />
<Property Name="ParsedDerivation" />
<Property Name="SourceColumn" />
<Property Name="Transform" />
<Property Name="SqlName" />
<Property Name="KeyExpression" />
<Property Name="SortKey" />
<Property Name="SortType" />
<Property Name="SPValue" />
<Property Name="NLSMap" />
<Property Name="TableDef">Sequential\Angel\cnt_in.txt</Property>
<Property Name="Association" />
<Property Name="FieldPos" />
<Property Name="FieldType" />
<Property Name="AllowCRLF" />
<Property Name="ColNull" />
<Property Name="DateMask" />
<Property Name="DependField" />
<Property Name="LevelNo">0</Property>
<Property Name="MFUpdateValue" />
<Property Name="NativeType" />
<Property Name="Occurs">0</Property>
<Property Name="PadNulls" />
<Property Name="FillerParents" />
<Property Name="RedefinedField" />
<Property Name="SignIndicator" />
<Property Name="SignOption">0</Property>
<Property Name="SortingOrder" />
<Property Name="StageVars" />
<Property Name="ArrayHandling" />
<Property Name="SyncIndicator">0</Property>
<Property Name="Usage" />
<Property Name="PadChar" />
<Property Name="ColumnReference">sNo</Property>
<Property Name="FlattenDependingField" />
<Property Name="APTFieldProp">quote=none</Property>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
<Property Name="SPParameterType" />
<Property Name="PKeyParsedDerivation" />
<Property Name="PKeySourceColumn" />
<Property Name="PKeyTransform" />
<Property Name="PKeyIsCaseless" />
</SubRecord>
- <SubRecord>
<Property Name="Name">sName</Property>
<Property Name="Description" />
<Property Name="DataType" />
<Property Name="SqlType">12</Property>
<Property Name="Precision">60</Property>
<Property Name="Scale">0</Property>
<Property Name="Nullable">0</Property>
<Property Name="KeyPosition">0</Property>
<Property Name="DisplaySize">1</Property>
<Property Name="Derivation" />
<Property Name="Group">0</Property>
<Property Name="ParsedDerivation" />
<Property Name="SourceColumn" />
<Property Name="Transform" />
<Property Name="SqlName" />
<Property Name="KeyExpression" />
<Property Name="SortKey">0</Property>
<Property Name="SortType">0</Property>
<Property Name="SPValue" />
<Property Name="NLSMap" />
<Property Name="TableDef" />
<Property Name="Association" />
<Property Name="FieldPos" />
<Property Name="FieldType" />
<Property Name="AllowCRLF">0</Property>
<Property Name="ColNull" />
<Property Name="DateMask" />
<Property Name="DependField" />
<Property Name="LevelNo">0</Property>
<Property Name="MFUpdateValue" />
<Property Name="NativeType" />
<Property Name="Occurs">0</Property>
<Property Name="PadNulls">0</Property>
<Property Name="FillerParents" />
<Property Name="RedefinedField" />
<Property Name="SignIndicator" />
<Property Name="SignOption">0</Property>
<Property Name="SortingOrder">0</Property>
<Property Name="StageVars" />
<Property Name="ArrayHandling" />
<Property Name="SyncIndicator">0</Property>
<Property Name="Usage" />
<Property Name="PadChar" />
<Property Name="ColumnReference" />
<Property Name="FlattenDependingField" />
<Property Name="APTFieldProp" />
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
<Property Name="SPParameterType" />
<Property Name="PKeyParsedDerivation" />
<Property Name="PKeySourceColumn" />
<Property Name="PKeyTransform" />
<Property Name="PKeyIsCaseless" />
</SubRecord>
</Collection>
<Property Name="Waitforwriter">0</Property>
<Property Name="PadChar">#</Property>
<Property Name="LeftTextPos">220</Property>
<Property Name="TopTextPos">264</Property>
<Property Name="SuppressTruncWarn">0</Property>
</Record>
- <Record Identifier="V0S1" DateModified="1899-12-30" TimeModified="00.00.01" Type="TransformerStage" Readonly="0">
<Property Name="Name">Transformer_1</Property>
<Property Name="Description" />
<Property Name="NextID">3</Property>
<Property Name="InputPins">V0S1P1</Property>
<Property Name="OutputPins">V0S1P2</Property>
<Property Name="PrimaryId" />
<Property Name="BeforeSubr" />
<Property Name="AfterSubr" />
<Property Name="ValidationStatus">0</Property>
<Property Name="StageType">CTransformerStage</Property>
<Property Name="NLSLocale" />
<Property Name="CompilerFlags" />
<Property Name="LinkerFlags" />
</Record>
- <Record Identifier="V0S1P1" DateModified="1899-12-30" TimeModified="00.00.01" Type="TrxInput" Readonly="0">
<Property Name="Name">DSLink3</Property>
<Property Name="Description" />
<Property Name="Partner">V0S0</Property>
<Property Name="KeyExpression" />
<Property Name="LinkType">1</Property>
<Property Name="MultiRow" />
</Record>
- <Record Identifier="V0S1P2" DateModified="1899-12-30" TimeModified="00.00.01" Type="TrxOutput" Readonly="0">
<Property Name="Name">DSLink4</Property>
<Property Name="Description" />
<Property Name="Partner">V0S2</Property>
<Property Name="Constraint" />
<Property Name="Reject">0</Property>
<Property Name="ErrorPin" />
<Property Name="RowLimit">0</Property>
<Property Name="ParsedConstraint" />
<Property Name="SourceColumns" />
<Property Name="RoutineTransforms">\(1B)</Property>
- <Collection Name="Columns" Type="OutputColumn">
- <SubRecord>
<Property Name="Name">sNo</Property>
<Property Name="Description" />
<Property Name="DataType" />
<Property Name="SqlType">4</Property>
<Property Name="Precision">10</Property>
<Property Name="Scale">0</Property>
<Property Name="Nullable">0</Property>
<Property Name="KeyPosition">1</Property>
<Property Name="DisplaySize">1</Property>
<Property Name="Derivation">DSLink3.sNo</Property>
<Property Name="Group" />
<Property Name="ParsedDerivation">DSLink3.sNo</Property>
<Property Name="SourceColumn">DSLink3.sNo</Property>
<Property Name="Transform" />
<Property Name="SqlName" />
<Property Name="KeyExpression" />
<Property Name="SortKey" />
<Property Name="SortType" />
<Property Name="SPValue" />
<Property Name="NLSMap" />
<Property Name="TableDef">Sequential\Angel\cnt_in.txt</Property>
<Property Name="Association" />
<Property Name="FieldPos" />
<Property Name="FieldType" />
<Property Name="AllowCRLF" />
<Property Name="ColNull" />
<Property Name="DateMask" />
<Property Name="DependField" />
<Property Name="LevelNo">0</Property>
<Property Name="MFUpdateValue" />
<Property Name="NativeType" />
<Property Name="Occurs">0</Property>
<Property Name="PadNulls" />
<Property Name="FillerParents" />
<Property Name="RedefinedField" />
<Property Name="SignIndicator" />
<Property Name="SignOption">0</Property>
<Property Name="SortingOrder" />
<Property Name="StageVars" />
<Property Name="ArrayHandling" />
<Property Name="SyncIndicator">0</Property>
<Property Name="Usage" />
<Property Name="PadChar" />
<Property Name="ColumnReference">sNo</Property>
<Property Name="FlattenDependingField" />
<Property Name="APTFieldProp">quote=none</Property>
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
<Property Name="SPParameterType" />
<Property Name="PKeyParsedDerivation" />
<Property Name="PKeySourceColumn" />
<Property Name="PKeyTransform" />
<Property Name="PKeyIsCaseless" />
</SubRecord>
- <SubRecord>
<Property Name="Name">sName</Property>
<Property Name="Description" />
<Property Name="DataType" />
<Property Name="SqlType">12</Property>
<Property Name="Precision">60</Property>
<Property Name="Scale">0</Property>
<Property Name="Nullable">0</Property>
<Property Name="KeyPosition">0</Property>
<Property Name="DisplaySize">1</Property>
<Property Name="Derivation">DSLink3.sName</Property>
<Property Name="Group">0</Property>
<Property Name="ParsedDerivation">DSLink3.sName</Property>
<Property Name="SourceColumn">DSLink3.sName</Property>
<Property Name="Transform" />
<Property Name="SqlName" />
<Property Name="KeyExpression" />
<Property Name="SortKey">0</Property>
<Property Name="SortType">0</Property>
<Property Name="SPValue" />
<Property Name="NLSMap" />
<Property Name="TableDef" />
<Property Name="Association" />
<Property Name="FieldPos" />
<Property Name="FieldType" />
<Property Name="AllowCRLF">0</Property>
<Property Name="ColNull" />
<Property Name="DateMask" />
<Property Name="DependField" />
<Property Name="LevelNo">0</Property>
<Property Name="MFUpdateValue" />
<Property Name="NativeType" />
<Property Name="Occurs">0</Property>
<Property Name="PadNulls">0</Property>
<Property Name="FillerParents" />
<Property Name="RedefinedField" />
<Property Name="SignIndicator" />
<Property Name="SignOption">0</Property>
<Property Name="SortingOrder">0</Property>
<Property Name="StageVars" />
<Property Name="ArrayHandling" />
<Property Name="SyncIndicator">0</Property>
<Property Name="Usage" />
<Property Name="PadChar" />
<Property Name="ColumnReference" />
<Property Name="FlattenDependingField" />
<Property Name="APTFieldProp" />
<Property Name="ExtendedPrecision">0</Property>
<Property Name="TaggedSubrec">0</Property>
<Property Name="OccursVarying">0</Property>
<Property Name="SPParameterType" />
<Property Name="PKeyParsedDerivation" />
<Property Name="PKeySourceColumn" />
<Property Name="PKeyTransform" />
<Property Name="PKeyIsCaseless" />
</SubRecord>
</Collection>
<Property Name="StageVars" />
<Property Name="LeftTextPos">420</Property>
<Property Name="TopTextPos">264</Property>
</Record>
- <Record Identifier="V0S2" DateModified="1899-12-30" TimeModified="00.00.01" Type="SeqFileStage" Readonly="0">
<Property Name="Name">Sequential_File_2</Property>
<Property Name="Description" />
<Property Name="NextID">2</Property>
<Property Name="InputPins">V0S2P1</Property>
<Property Name="OutputPins" />
<Property Name="Directory" />
<Property Name="UnixFormat">2</Property>
<Property Name="NLSMapName" />
<Property Name="PipeStage">0</Property>
<Property Name="UnicodeBOM">1</Property>
<Property Name="UnicodeSwapped">1</Property>
<Property Name="AllowColumnMapping">0</Property>
<Property Name="WithFilter">0</Property>
<Property Name="StageType">CSeqFileStage</Property>
</Record>
- <Record Identifier="V0S2P1" DateModified="1899-12-30" TimeModified="00.00.01" Type="SeqInput" Readonly="0">
<Property Name="Name">DSLink4</Property>
<Property Name="Description" />
<Property Name="Partner">V0S1</Property>
<Property Name="FileName">D:\Angel_DS\Angel\paruxml1.txt</Property>
<Property Name="Delimiter">,</Property>
<Property Name="QuoteChar">000</Property>
<Property Name="ColHeaders">0</Property>
<Property Name="OmitNewLine">0</Property>
<Property Name="Append">0</Property>
<Property Name="Backup">0</Property>
<Property Name="NullString" />
<Property Name="FixedWidth">0</Property>
<Property Name="ColSpace">0</Property>
<Property Name="EnforceMetaData">0</Property>
<Property Name="Writetime">0</Property>
<Property Name="WaitforReader">0</Property>
<Property Name="Flushrow">0</Property>
<Property Name="PadChar">#</Property>
</Record>
</Job>
- <!-- No records found in this section - Shared containers
-->
</DSExport>
while loading the columns in the xml input stage using xml metada importer

the following columns only could be captured
CharacterSet
ExportingTool
ToolVersion
ServerName
ToolInstanceID
Date
Time
ServerVersion
Identifier
DateModified
TimeModified
RecordIdentifier
RecordDateModified
RecordTimeModified
Type
Readonly
Name
Property
CollectionType
CollectionName
PropertyName
SubRecordProperty
RecordPropertyName
RecordProperty

But i want to even to get the columnas like this

<Property Name="Precision">60</Property>
and change the precision to 70

since i am not able to capture it in the xml metadata importer is there any method of doing so...
is it possble to the minute details or not..???

can anybody help me out
Top
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:
Contact kduke

Post by kduke »

If you want to go against repository tables then do a search on DS_JOBOBJECTS. This is where the column names are stored for each link. The record structure of this hashed file or table varies by the OLETYPE field. This field represents the stage type and or link type as to direction either input or output. Their input is what we think of as output. A typical OLETYPE is CODBCOutput which is a link which is a source. It has an associated CODBCStage with DSN, user and password. You need to join using OBJIDNO = JOBNO to get NAME (job name) out of DS_JOBS.

Figure out which link types or OLETYPEs you need and we can help you extract these column names.
Mamu Kim
Top
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Why don't you take a step back and tell us what the ultimate goal of all these xml questions you've been asking lately... what in the heck are you trying to accomplish?

If the whole point is to drive cetain kinds of changes, like the precision change noted above, why not just do what most people would do and use a text editor on the exported file? :?
-craig

"You can never have too many knives" -- Logan Nine Fingers
Top
parvathi
Participant
Posts: 103
Joined: Wed Jul 05, 2006 4:48 am
Contact:
Contact parvathi

Post by parvathi »

the ultimate goal is ..

first think that i have developed a job with certain stages ,compiled and run ...
Now i export this in an xml format....

With this xml file i should be able to capture all the properties of job including stage column names ,its length, precision etc... with the xml meta data importer tool in xml input stage and make the necessary changes...(But when I have done it I was able to capture only few but not all)

and then generate an output xml with the changes made...

This xmloutput file generated should be imported in manager and the changes made before should be reflected in the job design after importing...



Is this possible with an XML.....??

Can you tell me exactly what the xml transformer does...??

Or there any methods or tool to interpret the xml file like
this job, stages... its properties..., its colmns instead of going and examing the xml file



can suggest some idea or a different method to approach
Top
kduke
Charter Member
Charter Member
Posts: 5227
Joined: Thu May 29, 2003 9:47 am
Location: Dallas, TX
Contact:
Contact kduke

Post by kduke »

I doubt if it is possible. The XML output of a job is very complex with lots of repeating groups. It does sound reasonable though. I would not trust it knowing how complex this data structure is. I would think it would be easier to edit the repository directly.

There maybe cases where your idea may work like in the generation of new jobs from existing jobs.
Mamu Kim
Top
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Can you explain the nature of the changes you are looking at making through this methodology? List of examples, perhaps?
-craig

"You can never have too many knives" -- Logan Nine Fingers
Top
parvathi
Participant
Posts: 103
Joined: Wed Jul 05, 2006 4:48 am
Contact:
Contact parvathi

Post by parvathi »

chulett,

the nature of changes that i want to make is that..
suppose i have seqential stage in the job that has been exported as an xml file...

I may require to change the column names or length of the columns, its precision, or I may change the delimiter to a different value... all such kinds of these changes...
not only in the sequential file ,whatever stage i use i should be able to edit...
But it is very difficult to capture these tags or attributes of these tags from the xml exported file through the xml metadata importer tool

please suggest me if you have any idea
Top
Post Reply

Return to “IBM<sup>®</sup> Infosphere DataStage Server Edition”