Migration to Version 8

ray.wurlod · Post by **ray.wurlod** » Tue Mar 18, 2008 5:24 pm

There is, in my opinion, scope for an FAQ on the do's, don'ts, traps, pitfalls (and, just in case, successes) of migrating to version 8.

I have not done this upgrade yet. I have only installed version 8 "clean".

ArndW · Post by **ArndW** » Wed Mar 19, 2008 2:14 am

The same in my case, all of my V8 sites have been "greenfield" installations. I would guess that is most likely still the most common scenario as existing implementations at V7 will try to wait a bit longer before migrating up to V8.

kumar_s · Post by **kumar_s** » Wed Mar 19, 2008 9:51 am

Does a migration really required?
I was under the impression that, V8 can be installed 'over' the v7 installation. And hence no/less change in the jobs would require.
Or is this called as migration

ArndW · Post by **ArndW** » Wed Mar 19, 2008 10:47 am

Kumar - that would be a migration. But as well all know, sometimes when the marketing department says "100% compatible" they don't really mean that...

kumar_s · Post by **kumar_s** » Wed Mar 19, 2008 4:14 pm

Is it mandate to move the Repository from Universe to DB2 / Oracle / SQLserver?

ray.wurlod · Post by **ray.wurlod** » Wed Mar 19, 2008 4:59 pm

kcbland · Post by **kcbland** » Sun Mar 23, 2008 10:06 pm

This question doesn't seem to get answered cleanly so I'm going to ask it:

Given Server jobs developed:
1. under previous versions let's say 6.? and 7.?
2. according to standard use, no hacker tricks or references to undocumented API's, no Transformer bugs like cascading reference key expressions,etc
3. using no special functions or Universe tricks

(Basically you wrote jobs using the product and didn't do anything "special"

1. Can those jobs simply be imported using a .dsx into an 8.0 project and be compiled and runnable?
2. Are there any job conversion efforts (parameter blocks versus explicit parameters, etc).

The impetus is to weed out the confusion. I've heard so many answers on this topic I don't know which is most accurate. There's always due diligence when doing an onsite upgrade. There's always testing your jobs to make sure everything still works. I have a hard time believing there's actually significant coding and conversion efforts involved to make 7.5 Server jobs work under 8.0.

chulett · Post by **chulett** » Sun Mar 23, 2008 11:40 pm

kcbland wrote:I have a hard time believing there's actually significant coding and conversion efforts involved to make 7.5 Server jobs work under 8.0.

I have a hard time believing there's any. They should just... work.

ag_ram · Post by **ag_ram** » Thu Apr 03, 2008 4:11 pm

Ray,

kumar_s wrote:

Is it mandate to move the Repository from Universe to DB2 / Oracle / SQLserver?

ray.wurlod wrote:

Yes.

Can you expand your answer by explaining the key neccesities of keeping a change in the the best-adaptable multivalue RDBMS Universe to an usual RDBMS such as DB2 and Oracle for the storage of Repository items in DataStage? And moreover how the identified need of having a Nested RDBMS for DataStage Engine would be satisfied by a normal RDBMS, say DB2?

ray.wurlod · Post by **ray.wurlod** » Thu Apr 03, 2008 4:40 pm

The "relative merits" question is entirely hypothetical - it's done, and they're not going back. In version 8.0 the structure of the common Repository is far more complex than what is required for DataStage alone - it is closer to the MetaStage hub in design. To achieve first normal form, there are many more tables, with many more parent-child relationships. But, in addition, the locator must form part of the key - even if it's just for DataStage there is one Repository across all projects rather than one per project.

paulhill20 · Post by **paulhill20** » Tue Apr 22, 2008 8:16 pm

ray.wurlod wrote:The "relative merits" question is entirely hypothetical - it's done, and they're not going back. In version 8.0 the structure of the common Repository is far more complex than what is required for DataStage alone - it is closer to the MetaStage hub in design. To achieve first normal form, there are many more tables, with many more parent-child relationships. But, in addition, the locator must form part of the key - even if it's just for DataStage there is one Repository across all projects rather than one per project.

My understanding was that for the executable components - Universe is stil used. Is this correct? Why not migrate out of the Pick OS - isn't it about time? I'm starting to head to an upgrade cycle (from 7.5) - my biggest beef with this product has been no real "change of behavior" notes indicating, for instance, what has changed in the Transformer stage so that I can go and survey my transformers as implemented and say - these twenty percent are going to need to be gone over with a fine comb - it is always "put the code in there, compile, and test to see IF the stuff still works the same" - or is there something I've been missing. Enlighten me.

jdmiceli · Post by **jdmiceli** » Fri Sep 26, 2008 7:43 am

Hello all!

If it helps at all, I am in the process of moving a large project from 7.5.1a to Version 8.0.1 right now. My first attempt at this with some older code went very well and I had no errors, but my testing was not extensive. It was just to have a little something to play with and learn the new features.

Earlier this week, I removed all that code so that I could load my release ready code to perform testing on our V8 Development box. During the import process I have received thousands of errors that were not there the first time I imported code. I am going to experiment with this today and see if I can get around it. I still have more searching of DSXchange to see if anyone else has had this issue yet. I will create another post in the appropriate Server Edition forum for that error.

I will make sure to add any lessons learned to this FAQ if appropriate.

jdmiceli · Post by **jdmiceli** » Fri Oct 17, 2008 11:23 am

Here is an update on my project's migration from 7.5.1a to 8.0.1.

Performance: So far, the only performance information I can give deals with the development tools and response times dealing with those. We have tried local installs of the clients, as well as our standard Citrix virtual machine installs to access the software. Working through Citrix was abysmal at first, with just switching screens taking up to 20 minutes. Local client install was marginally better, requiring only 12 minutes to change screens. IBM then told us to not even try running the clients without at least 3 GB of RAM available. I was running 1 GB and onboard video with shared memory. I put in a video card and boosted RAM to 4GB. Performance working with the tools was then acceptable, though still generally slower than V7. With this experience, our admins tweaked the memory settings on the Citrix configuration to assign more memory, which improved performance some as well. Then, we got a fix pack from IBM to fix a problem at the WebSphere/InformationServer side of things and things improved tremendously. We are now at acceptable levels.

Running jobs: So far, this part has been a nightmare. On Dev, all jobs imported directly from V7 with no changes and compiled with no problems. However, nothing runs reliably. The very same code run over and over for the same data set and databases breaks at random places for no apparent reason. In Prod, the same jobs exported from V8 Dev (as opposed to V7 Dev) will not even compile (viewtopic.php?t=122916&highlight=.
I am opening another ticket with IBM support to see what they come up with for this one. So far the only suggestions I have had for this issue is to drop the project and rebuild it all.

I don't like that option...

I will update as we resolve and relate the experiences...

jdmiceli · Post by **jdmiceli** » Fri Oct 24, 2008 11:22 am

Update of my migration progress:

Having been pounding on V8's bug with multi-instanced jobs and log autopurge for a couple of weeks now, here is what we have learned at my company thus far:

1. There is a bug in V8 that appears when you have multi-instanced jobs and autopurge turned on for the logs. They have issued a couple of patches and tweaks to fix this issue, but it still does not appear to have worked completely. A quick way to see if your jobs are affected after running and getting odd errors: go to the Director to the job that failed. If it is multi-instanced and shows a status of 'Compiled', guess what? You too are a winner and have the bug. See below...

2. There is a variable you should set to allay some of the problem. Add 'DS_NO_INSTANCE_PURGING=1' to your .profile on Unix (I'm not sure where it goes on Windows). This supposedly gets rid of the problem, but my mileage has varied thus far.

3. I have found a potential workaround but I have more testing to do yet before I'll swear to it. It may be that it will only work for my project due to the way I have it set up (maximum parametization, invocation_id used for all job calls including the .csv files for job control, single base code for 19 companies). I recompiled all my jobs and then modified my .csv file to remove the invocation_id for the run. I did NOT change my .ini with regards to number of instances allowed. I set autopurge to retain 5 days worth of logs instead of the previous day settings I had before. I ran the job control with those settings. Then I re-added the invocation_id to my .csv file and ran again, and again, and so forth, alternating between clearing the database or not each time just to make sure a lot of data and log entries were made in different ways. Basically, I was trying to make it break. So far, doing that has been successful, but I reserve the right to be pessimistic until it continues to run for weeks on end without intervention.

This workaround may be functional, but it really should not be necessary if IBM had truly fixed the bug with the last patch we installed.

IBM is still looking at the problem and we are waiting for responses.

Thus concludes today's sermon,

DSXchange

Migration to Version 8

Migration to Version 8

One repository

Bug found with multi-instanced job and log autopurge