XML Stage vs XML Pack

dspr · Post by **dspr** » Thu Jan 31, 2013 5:01 am

Hi All,

I have worked in Informatica ETL tool before for 7 years and recently I moved to Datastage. I am working for a project where the requirement is to read and write XML targets. I tried using XML Pack(XML Input,XML Output) for reading XML's,validate XML's against XSD's and writing to XML target and I was successful in doing that for some sample XML's!! I now need to know the following:
1.What is the difference between XML Pack vs XML Stage?
2.How I can get an XML Stage in our current Datstage 8.5FP2 Server edition? Is it available in 8.7 Server edition if we upgrade?
3.I tried processing XML files with almost 30K records and it just runs in 1 minute. By using XML stage will the performance increase?
4.Validation capability of XML Pack- to what detail level the validation goes to?

This is my first post on the forum

Many Thanks!!

chulett · Post by **chulett** » Thu Jan 31, 2013 8:40 am

Welcome!

The new XML stage was introduced in the 8.5 release so should be something you have but perhaps there is a specific patch you need to add it. ALso, here are articles by a couple of our rockstars that might help.

From Vincent here and here. And from Ernie here.

vmcburney · Post by **vmcburney** » Thu Jan 31, 2013 6:50 pm

The main drawback of the old XML pack is that it is not as fast, the entire XML schema needs to be read and validated before data is processed resulting in a slow start up speed and failures on very large files. The new XML assembly stage does not have a memory limitation. The new stage tends to have better management of XSD files, version 9.1 has layering and filtering of XSD files, and more transformation and composition functions. It is much better at creating complex XML files.

chulett · Post by **chulett** » Thu Jan 31, 2013 8:57 pm

Speak of the devil.

vmcburney · Post by **vmcburney** » Fri Feb 01, 2013 2:43 am

When it comes to XML Ernie is the devil, I'm the devil's sidekick.

ray.wurlod · Post by **ray.wurlod** » Fri Feb 01, 2013 4:24 am

I always understood that the devil was in the details...

dspr · Post by **dspr** » Fri Feb 01, 2013 5:10 am

Dear All,
Thanks for your reply !! I have already read all the blogs and they are great !!

My questions are:-

1.We are already planning to move to 8.7 now, so if we move do we still need to install any fix to get the new XML stage?Or will it come by default?
2.I am comfortable using XML Pack. Now for using XML Stage what materials I should refer which gives me step by step details of using XML Stage.

I have to start all over again it seems. The project requirement is to use XML files for data transfer within applications which means that XML would be used across all applications.

Thanks again.

chulett · Post by **chulett** » Fri Feb 01, 2013 7:45 am

1. It should come with 'base' release of 8.7 but should also be simple to add to your existing release.

2. Documented in each release. Start here and choose the release you are interested in reading about. This is a direct link to the 8.7 pdf for the XML Transformation Guide, the documentation for the new stage.

rameshrr3 · Post by **rameshrr3** » Fri Feb 01, 2013 2:20 pm

Bear in mind XML data itself may be sourced and written to many differnt types of data sources.
I've worked with XML files, XML data columns in database tables( DB2, Oracle) and XML data that is read or written to a Message ( MQ) . There can even be custom programs that spew XML data to your datastage job/process or 'swallow' it from datastage.

ray.wurlod · Post by **ray.wurlod** » Fri Feb 01, 2013 3:12 pm

You can download and install version 8.7.0.1 (that is, 8.7 with Fix Pack 1 pre-installed). This will save you some time.

dspr · Post by **dspr** » Mon Feb 04, 2013 7:43 am

Thanks all for the replies !! I am really glad that I am now a premium member of such a good forum. Good source of information for a beginner like me.

eostic · Post by **eostic** » Tue Feb 05, 2013 8:53 am

Welcome aboard!

Everything said above is spot-on.....and the xml Stage will just "be there" with your 8.7 install, as will the others. There are times when the old stages are appropriate, particularly when the content is tiny (really small transactional types of xml documents) and when there is no xsd (the xml Stage requires one).

Ernie

dspr · Post by **dspr** » Tue Feb 05, 2013 9:31 am

Thanks Ernie!! What about XML file sizes of 30 MB?....Will the performance be while processing 30MB XML files? I tried using XML Input for 20K rows file size of 1.5 MB and the processing was really fast.
Also what about validation? Does XML Stage has better validation compared to XML pack?
Thanks.

chulett · Post by **chulett** » Tue Feb 05, 2013 10:26 am

You shouldn't have any issues with a 30MB file... I only started having problems with the old stages around the 200-300MB size range. The new stage being (from what I understand) a streaming stage should not have any such issues. And by "issues" I mean of the "fall over dead" variety.

eostic · Post by **eostic** » Tue Feb 05, 2013 3:00 pm

yeah...20 to 30mb is still "small"...but certainly, as that number grows, or if you have 5,000 20 to 30mb documents, then the story changes. ...but if it's just 20mb, and its once per day?

Again, your mileage may vary. At some point, regardless of which way you go, start the learning curve for the new xml stage....when you get to something that (a) is truly huge in size (b) requires an xsd...and the xsd is huge and complex (c) is about "writing" complex multi-path xml....then you will want and need the new xml stage. Don't wait till then to learn it.

download the new redbook on it.

Ernie