Migration to Version 8
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Migration to Version 8
There is, in my opinion, scope for an FAQ on the do's, don'ts, traps, pitfalls (and, just in case, successes) of migrating to version 8.
I have not done this upgrade yet. I have only installed version 8 "clean".
I have not done this upgrade yet. I have only installed version 8 "clean".
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
This question doesn't seem to get answered cleanly so I'm going to ask it:
Given Server jobs developed:
1. under previous versions let's say 6.? and 7.?
2. according to standard use, no hacker tricks or references to undocumented API's, no Transformer bugs like cascading reference key expressions,etc
3. using no special functions or Universe tricks
(Basically you wrote jobs using the product and didn't do anything "special"
1. Can those jobs simply be imported using a .dsx into an 8.0 project and be compiled and runnable?
2. Are there any job conversion efforts (parameter blocks versus explicit parameters, etc).
The impetus is to weed out the confusion. I've heard so many answers on this topic I don't know which is most accurate. There's always due diligence when doing an onsite upgrade. There's always testing your jobs to make sure everything still works. I have a hard time believing there's actually significant coding and conversion efforts involved to make 7.5 Server jobs work under 8.0.
Given Server jobs developed:
1. under previous versions let's say 6.? and 7.?
2. according to standard use, no hacker tricks or references to undocumented API's, no Transformer bugs like cascading reference key expressions,etc
3. using no special functions or Universe tricks
(Basically you wrote jobs using the product and didn't do anything "special"
1. Can those jobs simply be imported using a .dsx into an 8.0 project and be compiled and runnable?
2. Are there any job conversion efforts (parameter blocks versus explicit parameters, etc).
The impetus is to weed out the confusion. I've heard so many answers on this topic I don't know which is most accurate. There's always due diligence when doing an onsite upgrade. There's always testing your jobs to make sure everything still works. I have a hard time believing there's actually significant coding and conversion efforts involved to make 7.5 Server jobs work under 8.0.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Ray,
kumar_s wrote:
kumar_s wrote:
ray.wurlod wrote:Is it mandate to move the Repository from Universe to DB2 / Oracle / SQLserver?
Can you expand your answer by explaining the key neccesities of keeping a change in the the best-adaptable multivalue RDBMS Universe to an usual RDBMS such as DB2 and Oracle for the storage of Repository items in DataStage? And moreover how the identified need of having a Nested RDBMS for DataStage Engine would be satisfied by a normal RDBMS, say DB2?Yes.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The "relative merits" question is entirely hypothetical - it's done, and they're not going back. In version 8.0 the structure of the common Repository is far more complex than what is required for DataStage alone - it is closer to the MetaStage hub in design. To achieve first normal form, there are many more tables, with many more parent-child relationships. But, in addition, the locator must form part of the key - even if it's just for DataStage there is one Repository across all projects rather than one per project.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 11
- Joined: Tue Jun 22, 2004 1:06 pm
One repository
ray.wurlod wrote:The "relative merits" question is entirely hypothetical - it's done, and they're not going back. In version 8.0 the structure of the common Repository is far more complex than what is required for DataStage alone - it is closer to the MetaStage hub in design. To achieve first normal form, there are many more tables, with many more parent-child relationships. But, in addition, the locator must form part of the key - even if it's just for DataStage there is one Repository across all projects rather than one per project.
My understanding was that for the executable components - Universe is stil used. Is this correct? Why not migrate out of the Pick OS - isn't it about time? I'm starting to head to an upgrade cycle (from 7.5) - my biggest beef with this product has been no real "change of behavior" notes indicating, for instance, what has changed in the Transformer stage so that I can go and survey my transformers as implemented and say - these twenty percent are going to need to be gone over with a fine comb - it is always "put the code in there, compile, and test to see IF the stuff still works the same" - or is there something I've been missing. Enlighten me.
Hello all!
If it helps at all, I am in the process of moving a large project from 7.5.1a to Version 8.0.1 right now. My first attempt at this with some older code went very well and I had no errors, but my testing was not extensive. It was just to have a little something to play with and learn the new features.
Earlier this week, I removed all that code so that I could load my release ready code to perform testing on our V8 Development box. During the import process I have received thousands of errors that were not there the first time I imported code. I am going to experiment with this today and see if I can get around it. I still have more searching of DSXchange to see if anyone else has had this issue yet. I will create another post in the appropriate Server Edition forum for that error.
I will make sure to add any lessons learned to this FAQ if appropriate.
If it helps at all, I am in the process of moving a large project from 7.5.1a to Version 8.0.1 right now. My first attempt at this with some older code went very well and I had no errors, but my testing was not extensive. It was just to have a little something to play with and learn the new features.
Earlier this week, I removed all that code so that I could load my release ready code to perform testing on our V8 Development box. During the import process I have received thousands of errors that were not there the first time I imported code. I am going to experiment with this today and see if I can get around it. I still have more searching of DSXchange to see if anyone else has had this issue yet. I will create another post in the appropriate Server Edition forum for that error.
I will make sure to add any lessons learned to this FAQ if appropriate.
Bestest!
John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services
"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services
"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
Here is an update on my project's migration from 7.5.1a to 8.0.1.
Performance: So far, the only performance information I can give deals with the development tools and response times dealing with those. We have tried local installs of the clients, as well as our standard Citrix virtual machine installs to access the software. Working through Citrix was abysmal at first, with just switching screens taking up to 20 minutes. Local client install was marginally better, requiring only 12 minutes to change screens. IBM then told us to not even try running the clients without at least 3 GB of RAM available. I was running 1 GB and onboard video with shared memory. I put in a video card and boosted RAM to 4GB. Performance working with the tools was then acceptable, though still generally slower than V7. With this experience, our admins tweaked the memory settings on the Citrix configuration to assign more memory, which improved performance some as well. Then, we got a fix pack from IBM to fix a problem at the WebSphere/InformationServer side of things and things improved tremendously. We are now at acceptable levels.
Running jobs: So far, this part has been a nightmare. On Dev, all jobs imported directly from V7 with no changes and compiled with no problems. However, nothing runs reliably. The very same code run over and over for the same data set and databases breaks at random places for no apparent reason. In Prod, the same jobs exported from V8 Dev (as opposed to V7 Dev) will not even compile (viewtopic.php?t=122916&highlight=.
I am opening another ticket with IBM support to see what they come up with for this one. So far the only suggestions I have had for this issue is to drop the project and rebuild it all.
I don't like that option...
I will update as we resolve and relate the experiences...
Performance: So far, the only performance information I can give deals with the development tools and response times dealing with those. We have tried local installs of the clients, as well as our standard Citrix virtual machine installs to access the software. Working through Citrix was abysmal at first, with just switching screens taking up to 20 minutes. Local client install was marginally better, requiring only 12 minutes to change screens. IBM then told us to not even try running the clients without at least 3 GB of RAM available. I was running 1 GB and onboard video with shared memory. I put in a video card and boosted RAM to 4GB. Performance working with the tools was then acceptable, though still generally slower than V7. With this experience, our admins tweaked the memory settings on the Citrix configuration to assign more memory, which improved performance some as well. Then, we got a fix pack from IBM to fix a problem at the WebSphere/InformationServer side of things and things improved tremendously. We are now at acceptable levels.
Running jobs: So far, this part has been a nightmare. On Dev, all jobs imported directly from V7 with no changes and compiled with no problems. However, nothing runs reliably. The very same code run over and over for the same data set and databases breaks at random places for no apparent reason. In Prod, the same jobs exported from V8 Dev (as opposed to V7 Dev) will not even compile (viewtopic.php?t=122916&highlight=.
I am opening another ticket with IBM support to see what they come up with for this one. So far the only suggestions I have had for this issue is to drop the project and rebuild it all.
I don't like that option...
![Evil or Very Mad :evil:](./images/smilies/icon_evil.gif)
I will update as we resolve and relate the experiences...
Bestest!
John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services
"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services
"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
Bug found with multi-instanced job and log autopurge
Update of my migration progress:
Having been pounding on V8's bug with multi-instanced jobs and log autopurge for a couple of weeks now, here is what we have learned at my company thus far:
1. There is a bug in V8 that appears when you have multi-instanced jobs and autopurge turned on for the logs. They have issued a couple of patches and tweaks to fix this issue, but it still does not appear to have worked completely. A quick way to see if your jobs are affected after running and getting odd errors: go to the Director to the job that failed. If it is multi-instanced and shows a status of 'Compiled', guess what? You too are a winner and have the bug. See below...
2. There is a variable you should set to allay some of the problem. Add 'DS_NO_INSTANCE_PURGING=1' to your .profile on Unix (I'm not sure where it goes on Windows). This supposedly gets rid of the problem, but my mileage has varied thus far.
3. I have found a potential workaround but I have more testing to do yet before I'll swear to it. It may be that it will only work for my project due to the way I have it set up (maximum parametization, invocation_id used for all job calls including the .csv files for job control, single base code for 19 companies). I recompiled all my jobs and then modified my .csv file to remove the invocation_id for the run. I did NOT change my .ini with regards to number of instances allowed. I set autopurge to retain 5 days worth of logs instead of the previous day settings I had before. I ran the job control with those settings. Then I re-added the invocation_id to my .csv file and ran again, and again, and so forth, alternating between clearing the database or not each time just to make sure a lot of data and log entries were made in different ways. Basically, I was trying to make it break. So far, doing that has been successful, but I reserve the right to be pessimistic until it continues to run for weeks on end without intervention.
This workaround may be functional, but it really should not be necessary if IBM had truly fixed the bug with the last patch we installed.
IBM is still looking at the problem and we are waiting for responses.
Thus concludes today's sermon,
Having been pounding on V8's bug with multi-instanced jobs and log autopurge for a couple of weeks now, here is what we have learned at my company thus far:
1. There is a bug in V8 that appears when you have multi-instanced jobs and autopurge turned on for the logs. They have issued a couple of patches and tweaks to fix this issue, but it still does not appear to have worked completely. A quick way to see if your jobs are affected after running and getting odd errors: go to the Director to the job that failed. If it is multi-instanced and shows a status of 'Compiled', guess what? You too are a winner and have the bug. See below...
2. There is a variable you should set to allay some of the problem. Add 'DS_NO_INSTANCE_PURGING=1' to your .profile on Unix (I'm not sure where it goes on Windows). This supposedly gets rid of the problem, but my mileage has varied thus far.
3. I have found a potential workaround but I have more testing to do yet before I'll swear to it. It may be that it will only work for my project due to the way I have it set up (maximum parametization, invocation_id used for all job calls including the .csv files for job control, single base code for 19 companies). I recompiled all my jobs and then modified my .csv file to remove the invocation_id for the run. I did NOT change my .ini with regards to number of instances allowed. I set autopurge to retain 5 days worth of logs instead of the previous day settings I had before. I ran the job control with those settings. Then I re-added the invocation_id to my .csv file and ran again, and again, and so forth, alternating between clearing the database or not each time just to make sure a lot of data and log entries were made in different ways. Basically, I was trying to make it break. So far, doing that has been successful, but I reserve the right to be pessimistic until it continues to run for weeks on end without intervention.
This workaround may be functional, but it really should not be necessary if IBM had truly fixed the bug with the last patch we installed.
IBM is still looking at the problem and we are waiting for responses.
Thus concludes today's sermon,
Bestest!
John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services
"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"
John Miceli
System Specialist, MCP, MCDBA
Berkley Technology Services
"Good Morning. This is God. I will be handling all your problems today. I will not need your help. So have a great day!"