XML Stage vs XML Pack
Moderators: chulett, rschirm, roy
XML Stage vs XML Pack
Hi All,
I have worked in Informatica ETL tool before for 7 years and recently I moved to Datastage. I am working for a project where the requirement is to read and write XML targets. I tried using XML Pack(XML Input,XML Output) for reading XML's,validate XML's against XSD's and writing to XML target and I was successful in doing that for some sample XML's!! I now need to know the following:
1.What is the difference between XML Pack vs XML Stage?
2.How I can get an XML Stage in our current Datstage 8.5FP2 Server edition? Is it available in 8.7 Server edition if we upgrade?
3.I tried processing XML files with almost 30K records and it just runs in 1 minute. By using XML stage will the performance increase?
4.Validation capability of XML Pack- to what detail level the validation goes to?
This is my first post on the forum
Many Thanks!!
I have worked in Informatica ETL tool before for 7 years and recently I moved to Datastage. I am working for a project where the requirement is to read and write XML targets. I tried using XML Pack(XML Input,XML Output) for reading XML's,validate XML's against XSD's and writing to XML target and I was successful in doing that for some sample XML's!! I now need to know the following:
1.What is the difference between XML Pack vs XML Stage?
2.How I can get an XML Stage in our current Datstage 8.5FP2 Server edition? Is it available in 8.7 Server edition if we upgrade?
3.I tried processing XML files with almost 30K records and it just runs in 1 minute. By using XML stage will the performance increase?
4.Validation capability of XML Pack- to what detail level the validation goes to?
This is my first post on the forum
Many Thanks!!
Welcome!
The new XML stage was introduced in the 8.5 release so should be something you have but perhaps there is a specific patch you need to add it. ALso, here are articles by a couple of our rockstars that might help.
From Vincent here and here. And from Ernie here.
The new XML stage was introduced in the 8.5 release so should be something you have but perhaps there is a specific patch you need to add it. ALso, here are articles by a couple of our rockstars that might help.
From Vincent here and here. And from Ernie here.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
The main drawback of the old XML pack is that it is not as fast, the entire XML schema needs to be read and validated before data is processed resulting in a slow start up speed and failures on very large files. The new XML assembly stage does not have a memory limitation. The new stage tends to have better management of XSD files, version 9.1 has layering and filtering of XSD files, and more transformation and composition functions. It is much better at creating complex XML files.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
When it comes to XML Ernie is the devil, I'm the devil's sidekick.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Dear All,
Thanks for your reply !! I have already read all the blogs and they are great !!
My questions are:-
1.We are already planning to move to 8.7 now, so if we move do we still need to install any fix to get the new XML stage?Or will it come by default?
2.I am comfortable using XML Pack. Now for using XML Stage what materials I should refer which gives me step by step details of using XML Stage.
I have to start all over again it seems. The project requirement is to use XML files for data transfer within applications which means that XML would be used across all applications.
Thanks again.
Thanks for your reply !! I have already read all the blogs and they are great !!
My questions are:-
1.We are already planning to move to 8.7 now, so if we move do we still need to install any fix to get the new XML stage?Or will it come by default?
2.I am comfortable using XML Pack. Now for using XML Stage what materials I should refer which gives me step by step details of using XML Stage.
I have to start all over again it seems. The project requirement is to use XML files for data transfer within applications which means that XML would be used across all applications.
Thanks again.
1. It should come with 'base' release of 8.7 but should also be simple to add to your existing release.
2. Documented in each release. Start here and choose the release you are interested in reading about. This is a direct link to the 8.7 pdf for the XML Transformation Guide, the documentation for the new stage.
2. Documented in each release. Start here and choose the release you are interested in reading about. This is a direct link to the 8.7 pdf for the XML Transformation Guide, the documentation for the new stage.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Bear in mind XML data itself may be sourced and written to many differnt types of data sources.
I've worked with XML files, XML data columns in database tables( DB2, Oracle) and XML data that is read or written to a Message ( MQ) . There can even be custom programs that spew XML data to your datastage job/process or 'swallow' it from datastage.
I've worked with XML files, XML data columns in database tables( DB2, Oracle) and XML data that is read or written to a Message ( MQ) . There can even be custom programs that spew XML data to your datastage job/process or 'swallow' it from datastage.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Welcome aboard!
Everything said above is spot-on.....and the xml Stage will just "be there" with your 8.7 install, as will the others. There are times when the old stages are appropriate, particularly when the content is tiny (really small transactional types of xml documents) and when there is no xsd (the xml Stage requires one).
Ernie
Everything said above is spot-on.....and the xml Stage will just "be there" with your 8.7 install, as will the others. There are times when the old stages are appropriate, particularly when the content is tiny (really small transactional types of xml documents) and when there is no xsd (the xml Stage requires one).
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
You shouldn't have any issues with a 30MB file... I only started having problems with the old stages around the 200-300MB size range. The new stage being (from what I understand) a streaming stage should not have any such issues. And by "issues" I mean of the "fall over dead" variety.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
yeah...20 to 30mb is still "small"...but certainly, as that number grows, or if you have 5,000 20 to 30mb documents, then the story changes. ...but if it's just 20mb, and its once per day?
Again, your mileage may vary. At some point, regardless of which way you go, start the learning curve for the new xml stage....when you get to something that (a) is truly huge in size (b) requires an xsd...and the xsd is huge and complex (c) is about "writing" complex multi-path xml....then you will want and need the new xml stage. Don't wait till then to learn it.
download the new redbook on it.
Ernie
Again, your mileage may vary. At some point, regardless of which way you go, start the learning curve for the new xml stage....when you get to something that (a) is truly huge in size (b) requires an xsd...and the xsd is huge and complex (c) is about "writing" complex multi-path xml....then you will want and need the new xml stage. Don't wait till then to learn it.
download the new redbook on it.
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>