ADS as ETL tool for Telecom sector DWH

sh.bangash · Post by **sh.bangash** » Fri Sep 21, 2007 6:39 am

Hi,

I need to know about use of ADS as ETL tool for telecom sector DWH. What forced me to ask this question specific to telecom dwh is the huge data size and very tight data warehouse ETL Time Window.

Please, share your experience...

Thanks and Regards,

chulett · Post by **chulett** » Fri Sep 21, 2007 7:27 am

So... "ADS" is "Ascential DataStage" I assume? There is no longer an "Ascential" here, it's "IBM Websphere DataStage". And were you forced to ask this question strictly regarding the Server product as you've marked the post or is the Enterprise Edition / PX a valid option as well?

I'd wager anyone here doing "huge data size tight ETL window telecom sector DWH" is using PX.

sh.bangash · Post by **sh.bangash** » Fri Sep 21, 2007 9:59 am

Hi Chullett,

you are right 'ADS' is for Ascential Data Stage and also you are right in mentioning that its no more Ascential. My question was specific to Server Edition as it is what currently I have available with us. But, same question stands valid for Enterprise Edition/PX.
What I am interested to know is if this packaged ETL tool can be used for ETL purpose in telecom DWH!

Regards,

ray.wurlod · Post by **ray.wurlod** » Fri Sep 21, 2007 2:51 pm

There is a telco in India using a server-only DataStage implementation, relying upon multi-instance jobs to achieve scalability. They are processing crores of records per day, and the Data Warehouse is many tens of TB in size. Could be approaching 100TB by now. They're happy with it.

chulett · Post by **chulett** » Fri Sep 21, 2007 2:59 pm

Pretty sure Ken has done something similar, maybe not for a telco but from a large volume standpoint.

ArndW · Post by **ArndW** » Fri Sep 21, 2007 6:33 pm

I'm currently at a telco. Using PX and processing hundreds of millions of records per day. I've done similar types of volume using Server and hand-coding the parallelism but a well-designed PX job wins hands-down when it comes to throughput.

vmcburney · Post by **vmcburney** » Sat Sep 22, 2007 7:01 pm

I am currently helping to implement a data warehouse for a very large Telco. The scalability is provided by a grid based approach - able to grow up to hundreds of servers. Parallel version and grid management software makes this possible. There are design techniques for faster throughput such as shadow tables, ETL surrogate keys, changed data capture etc.

sh.bangash · Post by **sh.bangash** » Wed Sep 26, 2007 2:08 am

Fellows,
I still don't have a to the point reply over here. May be more details be inside 'Premium content' but since am not a member and don't see it becoming in next few months, so, from tehre too am not able to see the response...

Regards,

vmcburney · Post by **vmcburney** » Wed Sep 26, 2007 4:27 am

I don't think you missed anything, most of the answers were short enough not to get filtered. It's straight forward - DataStage handles high volumes of data and is scalable using a grid architecture, this makes it ideal for Telco data.

chulett · Post by **chulett** » Wed Sep 26, 2007 7:29 am

Your question:

sh.bangash wrote:What I am interested to know is if this packaged ETL tool can be used for ETL purpose in telecom DWH!

All of the answers you've been given are just longer versions of this: yes!