Can WTX replace DataStage ?

kaps · Post by **kaps** » Wed Mar 17, 2010 11:06 am

We use DataStage to populate the EDW and Marts and also to transfer data between Legacy sources and Distributed source systems.

Now some people are interested in using WTX for the same purpose. I can agree with them on transfering data to OLTP as they may want it in a queue as XML as that can be real time. But can WTX replace in populating EDW and marts ? I have a feeling WTX is not meant for that. But I never worked with WTX. So people who have idea about both please shed some light in this...

Appreciate your time..

Thanks

vmcburney · Post by **vmcburney** » Wed Mar 17, 2010 11:58 pm

By WTX do you mean the WebSphere DataStage TX tool? TX is typically a tool for conversion from one complex message format to another and so it sits in the WebSphere family to use as middleware off an enterprise bus or message queue. It's good at converting messages. It also has a plugin for DataStage for higher volumes and conversion to or from relational data. It's not going to be much good for relational targets like EDW and marts - it's going to be hard to map and harder to scale in batch or bulk load modes and it wont have the same level of support for lookups or the version 8 parallel slowly changing dimension stage and surrogate key stage.

eostic · Post by **eostic** » Thu Mar 18, 2010 7:18 am

Vincent is right.....they complement each other. While both can do basic transformations, they are designed for different purposes. WTX (WebSphere TX as it is called now) excells at moving complex hierarchical structures to "other" complex hierarchical structures. There are other things it does well also, but typically it is not well suited for large scale batch, appeals to a different level of user, and is not deeply connected with the metadata of such things as BI tools.

Ernie

Sreenivasulu · Post by **Sreenivasulu** » Thu Mar 18, 2010 7:42 am

Hi All,
Apart from the load factor the transformation options in WTX is far greater than Datastage.

Regards
Sreeni

eostic · Post by **eostic** » Thu Mar 18, 2010 11:13 am

That's the first time I've heard that comment. Interesting. I have not found that to be true in my usage of it, but perhaps there are things I didn't review during several projects in the past several years using MapStage in combination with WTX or using WTX with RTI, which was supported in release 7.

What I particularly like are some of the more esoteric things that it is designed to connect to....such as SWIFT, EDI and EDIFACT....right out of the box. The metadata (in what are called TypeTrees) is pre-defined.

Either way, it is still a complementary story, because the tools cater not only to individual technologies and applcation styles, but also to different developer communities. Most sites have a need for both kinds of tools.

You might want to ask this question inside the WTX thread of our forum....

Ernie

chulett · Post by **chulett** » Thu Mar 18, 2010 12:04 pm

eostic wrote:That's the first time I've heard that comment. Interesting. I have not found that to be true in my usage of it, but perhaps there are things I didn't review during several projects in the past several years using MapStage in combination with WTX or using WTX with RTI, which was supported in release 7.Ernie

And perhaps Sreeni is smoking crack.

(I kid! I kid!)

And yes, I would second the motion to repost this over in the TX forum.

kaps · Post by **kaps** » Thu Mar 18, 2010 4:45 pm

Thanks for the replies and I will post the samething there...I have the same thoughts like WTX is not meant for batch operation. But In future if we move towards real time DW then would this help or RTI would be the better choice for that ?

eostic · Post by **eostic** » Fri Mar 19, 2010 4:58 am

Nope. Same issue there. Both tools can do "real time". ...and in a data warehousing case, it's usually "real time feeding"....not the purpose of RTI..... simply the need for sourcing data from a real time source such as MQ series, a named pipe, JMS queue, etc.

As noted above, the choice for TX has more to do with the "shape" and format of the sources and targets , the metadata interfaces, the volume considerations and transactional scenarios, and then, if a license is available, quite often the audience and teams that will be using and supporting it.

Ernie

eostic · Post by **eostic** » Fri Mar 19, 2010 5:00 am

Making sure my prior entry was clear......."Nope" refers to the fact that moving to real time does not necessarily dictate a reason to use WTX instead of DataStage. You are on track with DataStage if you need regular ETL functionality (RTI and SOA issues are a whole other topic).

Ernie

Sreenivasulu · Post by **Sreenivasulu** » Fri Mar 19, 2010 9:05 am

Thanks eostic and craig for your views.

Regards
Sreeni

chulett · Post by **chulett** » Fri Mar 19, 2010 9:15 am

More-so Ernie, of course.

IBM Analytics Champion 2009 - 2020 · Post by **asorrell** » Fri Mar 19, 2010 10:23 am

Sorry about being late to the party here... I had a friend that was forced into using WTX for a data warehousing project. Why? Because the company already owned WTX and in their management's appraisal "there's really no big difference between WTX and PX - they both move data, right?".

About three months later when the first jobs were started they found out that they couldn't have been more wrong.

In their case they needed extreme speed because they were planning on moving hundreds of millions of records in a short amount of time - something that PX excels at. Though they were able to get WTX to transfer data, it couldn't approach the load speeds required by the project. In the end, the entire project was killed because of the selection of the wrong ETL product.

There is some overlap between the two products, but assuming they are interchangeable is about as silly as saying a pick-up truck and an 18-wheeler are interchangeable because they "both just move cargo - right?" Right tool for the right job... always better in the long run.

chulett · Post by **chulett** » Fri Mar 19, 2010 10:49 am

I don't personally have any hands on experience with WTX but did evaluate Mercator a number of years ago so I've seen "it" or at least a version of "it". From what I've understood its strengths to be, from hanging out here and from osmotic knowledge trawling the internets, none of this really surprises me. Not even Andy's story, unfortunately.

eostic · Post by **eostic** » Fri Mar 19, 2010 1:05 pm

...this discussion has a history that is as old as ETL and EAI tools, which is upwards of twenty years. "EAI vs ETL" used to be a big debate and discussion, and if you can find old papers, even the analysts always concluded...."the typical customer needs both." The area between them us much greyer than it used to be, but when you push on it, the strengths of their origins still pours out, and whether we're talking about TX vs DS, or Web Methods vs INFA, or SeeBeyond vs either or others, it just about always comes down to the exact same issues as we've discussed here. I don't like to advertise my blog, but I commented on this same subject awhile ago here, if anyone is interested... http://dsrealtime.wordpress.com/2009/05 ... d-subject/

Ernie

chulett · Post by **chulett** » Fri Mar 19, 2010 1:51 pm

I don't like to advertise my blog... except in my signature in every post.

That's OK, Ernie, we like you to advertise your blog, there's good stuffs in there.