Anway feeback on this is really appreicated...
We are planning to replicate similar ENV on a Laptop, so that it will be easy to walk around and work.
(laptop will be used as a server to run the jobs for DS/PX/QS/MS and to test the jobs running against DB2)
We would like to load the following on a laptop:
LATEST VERSIONS OF...
UNIX AIX 5X
IBM DB2
DATASTAGE
QUALITY STAGE
META STAGE
Data is not that big...its just a sample data and we will be only using this ENV to do our job rather than optimizing the jobs...
I have following Qustions:
1. Is there a power full laptop out in the market(from DELL) which can be used for this replication?
2. How many CPU's will be available on the laptop...Is it possible to upgrade the # of CPU's to use them with Parallel Extender jobs on the laptop.
3. How much RAM is advisable for this kind of setup? (Multiple people will be trying to access the laptop to run different job in DS/PX/QS/MS)
4. Basically is it possible to ALMOST replicate the ENV with a laptop, similar to that one on SMP/MPP hardwares?( I guess laptops dont have hardware similar to SMP's or MPP's)
5. I am sure for now that only SERVERS will be installed on the laptop...
LAPTOP Configuration to replicated ENV...AIX/DB2/DS/PX/QS/
Moderators: chulett, rschirm, roy
Can you even run AIX on an Intel-based chipset? From what I recall of my AIX days, I thought it was RISC architecture and needed more like a Motorola... [shrug] And I don't recall ever seeing a "laptop" with multiple CPUs, but ya never know what's out there until you dig around.
Good luck! I'd be interested to know what you find out.
Good luck! I'd be interested to know what you find out.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
vdr123
I think you can do some of this. I doubt if AIX runs on an Intel PC but you could run Linux. I do not know of any laptops with multiple processors but I am sure they exist. All of this does work on Windows. It is easy to export jobs and information from any UNIX DataStage and import into Windows DataStage.
I have thought about this a lot. How can export an environment to build a prototype and take it with you or send it somewhere. Lets say you have consultants all around the world and you want to build a working prototype and send it to them. DwNav was designed to do some of this. I need a web front end on it to complete the picture. Lets say you work for IBM or EDS and you have DataStage experts that need to review all designs and make sure they conform to your best practices. Even if you have a restricted tool set like Ascential. You still need ERwin and few other tools to complete the picture. Most of this can be done through good metadata. You could use MetaStage as your metadata standard but what you need to recreate source and target data sources. You could sample data in an intelligent way.
Here is how this may work. TOAD and SQL Navigator are Oracle tools which can do some of this in nice way. These and other tools can create all the DDL. You need to get rid of all storage requirements because you would need the same amount of disk space. Once you have created source and target databases then you may still need sample data. Lets say you sample the source fact table maybe a 1000 records. Now you need all the related data in source dimension tables. You could do this with foriegn key constraints. It may be difficult to automate this but you could easily do it with DataStage jobs if you can get your laptop on the network. They key would be to automate this process. MetaRecon or ProfileStage can do a lot of this. I think it could be modified to do it all.
Peter Lai a friend of mine and myself did a lot of this in 1996. This allowed us to work from home. We had used the customer application for many years so we knew the source system. I designed the target data warehouse in SQL Server and ERwin. ERwin was supposed to implement table changes for you but it always crashed SQL Server so we used it to generate the DDL. You could either drop the tables or change create tables into alter tables. This is the whole importance of change management. You need this to be automated. ERwin is nice and probably does all this correctly now but still if you have a 1000 tables then you can get lost in ERwin. Views are hard to maintain and ugly. We have all seen sites when ERwin diagrams as wallpaper. You need a way to browse metadata and relate it back to source and target tables and then pull up DataStage. All of this needs to tied to change management. If some number was accurate in the data warehouse last month and is now no longer accurate. What changed? Did the tables definition change or the DataStage jobs change or maybe someone forgot to process one source flat file or maybe a parameter was loaded wrong and we populated from a test database. How do I do a system audit on this? How do I make sure it never happens again? Someone may get fired next time and it might be you.
How can you get started? You need something to export your DDL. You need to decide do you want to store this in tables or text files. I used Access. I wrote basic programs to generate DDL from these metadata tables. Pick a standard. I used Oracle system tables. If you download DwNav which is free for 30 days then all of this is included. I linked to the source Oracle system tables. I copied there structure into Access. At that customer site we also had SQL Server so I used Access to map SQL Server system tables into my metadata tables which look like Oracles. We also had Progress. This was not so easy. We did not have ODBC into Progress but I got a data dictionary dump of all the source tables. It was fixed length. Access has a way to import flat files so I got Progress source tables into the same structure.
Next you need to create the prototype databases. Generate all the DDL for source and target databases. Create all the tables in Oracle and SQL Server or just one or the other. Use TOAD on Oracle to dump dimension tables. TOAD can generate INSERT statements. You could easily write this in DataStage. You do not want lots of rows so avoid large tables or restrict it to the first 1000 rows. The proper way to do this is to sample your fact table and then make sure you get all the related dimension tables.
Once you have a complete sample of your sources then you should be able to build your data warehouse off site. Peter completely automated building our source. We did this on site. It was about 300 mb. It zipped up to 30mb. Here is what he did. It was for an HMO insurance company. He pick 1000 members. He got all of the claims. Claims was the souce for the fact table. Now he needed all the dimention tables like Groups, Physicians, Hospitals, Diagnosis codes and so on. This was done by getting all the Physicians with sample claims or was the members Primary Care Physician. The same for all the other tables. You need to remove all foreign key constraints to do it in this order. If you generate DDL and INSERT statements then you can create this data anywhere.
Now we build the data warehouse and data marts completely off site. We did not need to spend the next 6 months living in a hotel in Tampa. Saved thousands of dollars. We built it at home. We split the work up so we were not working on the same part. We synced up our work every Friday. Peter did all the admin work. I did all the design work. Peter built the jobs. I built the reports. We both got to see our family and friends all the time. We went on site to train and transition our work. The customer received a product and an expertise they could not otherwise afford. I have been trying to convince customers to do this every since. I doubt if anyone will ever let me do it again. I hate being on the road. It is a terrible lifestyle.
My goal is to build a set of tools to allow any to work from anywhere in the world. Maybe the best person for the job in Australia or Florida. Maybe the customer cannot afford to fly him here for 6 months. Maybe you need him to review the design for just a week and suggest performance changes. Maybe you need integrity checks and you are not sure how to do it. I would work for half the money if I get to stay home. I can get a lot more done too. The quality of my life goes way up.
Here is what you need.
1. Standard set of tools.
2. Standard methodolgies.
3. Best practices clearly defined like sequential files for restartablity.
(Ask Kenneth)
4. Web interface to upload designs
5. Method to build prototype off site
6. Upload design changes or suggestions
7. Implementation strategy
I would love to set and build this with Peter and some of my friends. I know it could change the way data warehouse are built and audited. MataStage and ProfileStage and just the begining of what is needed. My tools DwNav and DsWebMon are moving in the right direction. I wish I had Ascential's money or someone like EDS would allow pay me to build a data warehouse practice like it is supposed to be. Make it text book perfect or what a friend would call "Perfect or a little better" so even Hester, Kenneth and Ray would want to use it.
I am too old and too tired to keep fighting a bear with a pocket knife. Can anyone relate? I would go on but I have a headache now.
Kim.
I think you can do some of this. I doubt if AIX runs on an Intel PC but you could run Linux. I do not know of any laptops with multiple processors but I am sure they exist. All of this does work on Windows. It is easy to export jobs and information from any UNIX DataStage and import into Windows DataStage.
I have thought about this a lot. How can export an environment to build a prototype and take it with you or send it somewhere. Lets say you have consultants all around the world and you want to build a working prototype and send it to them. DwNav was designed to do some of this. I need a web front end on it to complete the picture. Lets say you work for IBM or EDS and you have DataStage experts that need to review all designs and make sure they conform to your best practices. Even if you have a restricted tool set like Ascential. You still need ERwin and few other tools to complete the picture. Most of this can be done through good metadata. You could use MetaStage as your metadata standard but what you need to recreate source and target data sources. You could sample data in an intelligent way.
Here is how this may work. TOAD and SQL Navigator are Oracle tools which can do some of this in nice way. These and other tools can create all the DDL. You need to get rid of all storage requirements because you would need the same amount of disk space. Once you have created source and target databases then you may still need sample data. Lets say you sample the source fact table maybe a 1000 records. Now you need all the related data in source dimension tables. You could do this with foriegn key constraints. It may be difficult to automate this but you could easily do it with DataStage jobs if you can get your laptop on the network. They key would be to automate this process. MetaRecon or ProfileStage can do a lot of this. I think it could be modified to do it all.
Peter Lai a friend of mine and myself did a lot of this in 1996. This allowed us to work from home. We had used the customer application for many years so we knew the source system. I designed the target data warehouse in SQL Server and ERwin. ERwin was supposed to implement table changes for you but it always crashed SQL Server so we used it to generate the DDL. You could either drop the tables or change create tables into alter tables. This is the whole importance of change management. You need this to be automated. ERwin is nice and probably does all this correctly now but still if you have a 1000 tables then you can get lost in ERwin. Views are hard to maintain and ugly. We have all seen sites when ERwin diagrams as wallpaper. You need a way to browse metadata and relate it back to source and target tables and then pull up DataStage. All of this needs to tied to change management. If some number was accurate in the data warehouse last month and is now no longer accurate. What changed? Did the tables definition change or the DataStage jobs change or maybe someone forgot to process one source flat file or maybe a parameter was loaded wrong and we populated from a test database. How do I do a system audit on this? How do I make sure it never happens again? Someone may get fired next time and it might be you.
How can you get started? You need something to export your DDL. You need to decide do you want to store this in tables or text files. I used Access. I wrote basic programs to generate DDL from these metadata tables. Pick a standard. I used Oracle system tables. If you download DwNav which is free for 30 days then all of this is included. I linked to the source Oracle system tables. I copied there structure into Access. At that customer site we also had SQL Server so I used Access to map SQL Server system tables into my metadata tables which look like Oracles. We also had Progress. This was not so easy. We did not have ODBC into Progress but I got a data dictionary dump of all the source tables. It was fixed length. Access has a way to import flat files so I got Progress source tables into the same structure.
Next you need to create the prototype databases. Generate all the DDL for source and target databases. Create all the tables in Oracle and SQL Server or just one or the other. Use TOAD on Oracle to dump dimension tables. TOAD can generate INSERT statements. You could easily write this in DataStage. You do not want lots of rows so avoid large tables or restrict it to the first 1000 rows. The proper way to do this is to sample your fact table and then make sure you get all the related dimension tables.
Once you have a complete sample of your sources then you should be able to build your data warehouse off site. Peter completely automated building our source. We did this on site. It was about 300 mb. It zipped up to 30mb. Here is what he did. It was for an HMO insurance company. He pick 1000 members. He got all of the claims. Claims was the souce for the fact table. Now he needed all the dimention tables like Groups, Physicians, Hospitals, Diagnosis codes and so on. This was done by getting all the Physicians with sample claims or was the members Primary Care Physician. The same for all the other tables. You need to remove all foreign key constraints to do it in this order. If you generate DDL and INSERT statements then you can create this data anywhere.
Now we build the data warehouse and data marts completely off site. We did not need to spend the next 6 months living in a hotel in Tampa. Saved thousands of dollars. We built it at home. We split the work up so we were not working on the same part. We synced up our work every Friday. Peter did all the admin work. I did all the design work. Peter built the jobs. I built the reports. We both got to see our family and friends all the time. We went on site to train and transition our work. The customer received a product and an expertise they could not otherwise afford. I have been trying to convince customers to do this every since. I doubt if anyone will ever let me do it again. I hate being on the road. It is a terrible lifestyle.
My goal is to build a set of tools to allow any to work from anywhere in the world. Maybe the best person for the job in Australia or Florida. Maybe the customer cannot afford to fly him here for 6 months. Maybe you need him to review the design for just a week and suggest performance changes. Maybe you need integrity checks and you are not sure how to do it. I would work for half the money if I get to stay home. I can get a lot more done too. The quality of my life goes way up.
Here is what you need.
1. Standard set of tools.
2. Standard methodolgies.
3. Best practices clearly defined like sequential files for restartablity.
(Ask Kenneth)
4. Web interface to upload designs
5. Method to build prototype off site
6. Upload design changes or suggestions
7. Implementation strategy
I would love to set and build this with Peter and some of my friends. I know it could change the way data warehouse are built and audited. MataStage and ProfileStage and just the begining of what is needed. My tools DwNav and DsWebMon are moving in the right direction. I wish I had Ascential's money or someone like EDS would allow pay me to build a data warehouse practice like it is supposed to be. Make it text book perfect or what a friend would call "Perfect or a little better" so even Hester, Kenneth and Ray would want to use it.
I am too old and too tired to keep fighting a bear with a pocket knife. Can anyone relate? I would go on but I have a headache now.
Kim.
Mamu Kim
LAPTOP Configuration to replicated ENV...AIX/DB2/DS/PX/QS/
I thought PX was being ported to LINUX. If that's happened, you might try to put LINUX on a laptop. Or you could try to cluster 2 linux laptops together.
The PX client can sit on any Windows box. It just needs to be able to connect to the server. In that sense, the PX client is mobile.
Why are you doing this?
-BP
The PX client can sit on any Windows box. It just needs to be able to connect to the server. In that sense, the PX client is mobile.
Why are you doing this?
-BP
Thanks for ur input guys...
Kim D,I can understand what ur trying to do...but its like building a set of interated tools working together to generate the DW solutions which is mobile.
I am trying to set up servers on the laptop(mobility is difficult if u cluster more than one laptop)...so that we can just carry it anywhere we want and work from there...well we use clients on windows and only try to run them on UNIX servers(PX).
I am not sure Intel supports this kind of architecture on laptops...I am not aware of any other chipset which can support the above list of things we are plannning to replicate.
Any feedback is helpfull.
-VDR
Kim D,I can understand what ur trying to do...but its like building a set of interated tools working together to generate the DW solutions which is mobile.
I am trying to set up servers on the laptop(mobility is difficult if u cluster more than one laptop)...so that we can just carry it anywhere we want and work from there...well we use clients on windows and only try to run them on UNIX servers(PX).
I am not sure Intel supports this kind of architecture on laptops...I am not aware of any other chipset which can support the above list of things we are plannning to replicate.
Any feedback is helpfull.
-VDR
You can do everything you ask, except run AIX on a laptop. You can, however, execute ksh scripts via tools such as MKS Toolset. Your challenge is to build COMPLETELY portable jobs. This means that all paths have to be parameterized, and your parameter MUST include the trailing slash. You must also pick a standard sequential file format, either NT or UNIX, because the CR/LF issue will apply (I recommend picking Unix). You will also have to make sure that if you issue a DSExecute ANYWHERE that the system type is parameterized or derived prior to calling DSExecute, so that your value for system type is either "NT" or "UNIX".
You can download DB2 EEE for Intel for free from IBM's website. You can install DS Server if your licensing allows. You get into the issue of PX running on Windows for verison 7. I believe the answer is any day now.
As a former road warrior for Ascential, our laptops were loaded with everything under the sun. At onetime, my laptop carried SQL-Server, Informix IDS 7, RedBrick, DB2, Oracle 9, DS, MS, and QM. Standard issue was 1 cpu 512meg 700mhz with 40 gig drive. At one client, we used my laptop for a week for development with 7 guys on it. As long as they didn't run jobs, it handled it. Our jobs were 100% portable, so once the Sun server was back up, we moved back onto that. Remember those days Carter?
You can download DB2 EEE for Intel for free from IBM's website. You can install DS Server if your licensing allows. You get into the issue of PX running on Windows for verison 7. I believe the answer is any day now.
As a former road warrior for Ascential, our laptops were loaded with everything under the sun. At onetime, my laptop carried SQL-Server, Informix IDS 7, RedBrick, DB2, Oracle 9, DS, MS, and QM. Standard issue was 1 cpu 512meg 700mhz with 40 gig drive. At one client, we used my laptop for a week for development with 7 guys on it. As long as they didn't run jobs, it handled it. Our jobs were 100% portable, so once the Sun server was back up, we moved back onto that. Remember those days Carter?
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I wonder who in Ascential gets to figure out the licensing issues!Thank You for your input...
Well...using Linux with Windows is reasonable...
Main idea here is to replicate the same ENV on the laptop...so we have windows for clients and AIX for server.
We do all DEV on WIN and run jobs(PX) on server...
Main problem is with the PX jobs...when we want to run them...I did find on Ascential website which says it supports Linux(redhat) with its OS Release redhat AS 2.1 for DS XE/Qs...but not sure if it has something for Metastage(MS)...
As Kim said, we can use VMware and run WIN and Linux together on a laptop...we can install all clients on WIN and have servers on Linux(with AIX for linux, DB2 for linux)...I read on the websites that there is support for Linux from IBM products.
Kim, can u please through some light on VMware...is it something to do with virtual machines(MEM?) environment...
we can partition the laptop for WIN and linux and then install our products on the partitions...we can run only one OS on the laptop either Win or Linux.....how can VMware be used to run them at the same time...I mean how are they made available at the same time on laptop?
And regarding PORTABILITY from Linux to UNIX...I am guessing it must not be that hard! if we use parameters for all the declarations with the tools...we have to develop the jobs thinking that we have to run them on linux and unix later.
If MS is not available on Linux from DS...then I guess we can deal with metadata on Windows...I am not sure how they(DS/QS with MS) work in tandem when running the server PX jobs?
Again, I am guessing it must be OK with one CPU on a Intel laptop to run few PX jobs at a time...well...laptop will need a air-condition inside it to keep it working normally with all the runs...but as i said tunning of the jobs can be done on the original ENV...we are just trying to get the work done off-site and keep it mobile...
Please comment on this.
Well...using Linux with Windows is reasonable...
Main idea here is to replicate the same ENV on the laptop...so we have windows for clients and AIX for server.
We do all DEV on WIN and run jobs(PX) on server...
Main problem is with the PX jobs...when we want to run them...I did find on Ascential website which says it supports Linux(redhat) with its OS Release redhat AS 2.1 for DS XE/Qs...but not sure if it has something for Metastage(MS)...
As Kim said, we can use VMware and run WIN and Linux together on a laptop...we can install all clients on WIN and have servers on Linux(with AIX for linux, DB2 for linux)...I read on the websites that there is support for Linux from IBM products.
Kim, can u please through some light on VMware...is it something to do with virtual machines(MEM?) environment...
we can partition the laptop for WIN and linux and then install our products on the partitions...we can run only one OS on the laptop either Win or Linux.....how can VMware be used to run them at the same time...I mean how are they made available at the same time on laptop?
And regarding PORTABILITY from Linux to UNIX...I am guessing it must not be that hard! if we use parameters for all the declarations with the tools...we have to develop the jobs thinking that we have to run them on linux and unix later.
If MS is not available on Linux from DS...then I guess we can deal with metadata on Windows...I am not sure how they(DS/QS with MS) work in tandem when running the server PX jobs?
Again, I am guessing it must be OK with one CPU on a Intel laptop to run few PX jobs at a time...well...laptop will need a air-condition inside it to keep it working normally with all the runs...but as i said tunning of the jobs can be done on the original ENV...we are just trying to get the work done off-site and keep it mobile...
Please comment on this.
VDR
VMware is a slick little product. It allows you to create one large blob or file which will contain a whole new OS either Windows or Linux or whatever. You can boot this new OS which runs in a window. You can full screen it so it looks like the original. Both the child OS in VMware and the parent can access network cards. You can even telnet from the parent into the child OS. It would be a slick way to run multiple versions of DataStage as well. A friend of mine Mike Nason (mpe@mindspring.com) uses it all the time and told me about it. It is a very powerful tool.
Kim.
VMware is a slick little product. It allows you to create one large blob or file which will contain a whole new OS either Windows or Linux or whatever. You can boot this new OS which runs in a window. You can full screen it so it looks like the original. Both the child OS in VMware and the parent can access network cards. You can even telnet from the parent into the child OS. It would be a slick way to run multiple versions of DataStage as well. A friend of mine Mike Nason (mpe@mindspring.com) uses it all the time and told me about it. It is a very powerful tool.
Kim.
Mamu Kim