Has anyone any experience - especially production experience - with using DataStage in the cloud but where the data sources and targets are not in the cloud?
How were the data transferred to and from DataStage, and was the transfer mechanism a bottleneck?
DataStage in the Cloud
Moderators: chulett, rschirm, roy
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
DataStage in the Cloud
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
I've only done it with test data. It really depends on your source/target sites. Normally the connection speeds from most cloud providers (i.e. AWS & SoftLayer) are fast enough but the source or target location is slow. This was especially true for using a remote site as the source since most ISP's degrade the upload performance to improve download speeds.
In my example I had IIS 9.1 running on AWS (N. Virginia) reaching back to an Oracle 11g server in a datacenter hosted at a commercial client's office in Atlanta, GA. In our tests we had frequent disconnections at first but then started to schedule the test jobs during late night periods which helped with the disconnection issues. However, the speed of transfer was still slow (13-15 rows per second using Oracle Connector) and about the same speed with ODBC. After that test we actually ended up taking a backup of the test database and imported it to AWS RDS (hosted database) and then terminated it after the test data was extracted within 2 hours.
Bottom line is unless you are dealing with a remote datacenter with a great connection expect the transfer to be a bottleneck. If possible, I highly recommend keeping all the data contained in the same environment if possible.
Let me know if you have any other questions.
In my example I had IIS 9.1 running on AWS (N. Virginia) reaching back to an Oracle 11g server in a datacenter hosted at a commercial client's office in Atlanta, GA. In our tests we had frequent disconnections at first but then started to schedule the test jobs during late night periods which helped with the disconnection issues. However, the speed of transfer was still slow (13-15 rows per second using Oracle Connector) and about the same speed with ODBC. After that test we actually ended up taking a backup of the test database and imported it to AWS RDS (hosted database) and then terminated it after the test data was extracted within 2 hours.
Bottom line is unless you are dealing with a remote datacenter with a great connection expect the transfer to be a bottleneck. If possible, I highly recommend keeping all the data contained in the same environment if possible.
Let me know if you have any other questions.