Friday, April 8, 2011

Upgrading an HDR Environment from 11.50.FC3 to 11.50.FC8W2

I'm about to upgrade one of our HDR pairs from a relatively old version of 11.50 to the most recent drop to resolve a minor bug. Since an upgrade is something you may be interested in doing some day, I thought it would be a good idea to document my steps here.

Normally an upgrade from xC3 to xC8 would be trivial and low risk since we're only upgrading the minor versions, but with the addition of Compression in xC4 any upgrade from xC3 or lower must be treated as a major upgrade which includes a reinitialization of HDR.

I want to minimize downtime by doing as much of the work as I can outside of the maintenance window and I want to minimize my risk by testing the following migration and reversion steps on my test systems to ensure there are no gotchas and choreograph my upgrade in a way that always gives me a quick and safe way to abort mission and get production up and running if something goes wrong during the upgrade.

Lucky for you and me, the bulk of the upgrade work can be accomplished during normal business hours with no impact or downtime. Not only does this minimize the amount of late night/early morning maintenance window work you will have to do, it will minimize the mistakes that can happen when you are doing work during sleepy time.

A lot of my upgrade steps are based on INFORMIXDIR being a link to an actual directory, you did remember to do that didn't you? If not, you should invest the time and energy required to make INFORMIXDIR a link, you will be glad you did.

Get yourself some binaries

There are two types Informix tar balls, there is the bundle which includes everything (engine, CSDK, iConnect, etc.) which is typically downloaded from Passport Advantage or the Innovator-C download page and there are Fix Packs which are typically downloaded from Fix Central and only include the engine.

The bundle is installed via ids_install and the engine is installed via installserver if that helps.

If upgrading with a Fix Pack, you will need the bundle used for your current installation. For this upgrade I am using a Fix Pack for 11.50.FC8W2 and have the 11.50.FC3 bundle from my initial installation.

Install the new engine - Perform the following steps on the HDR Primary and HDR Secondary

Let us skip some of the boring stuff and say that I have downloaded my Fix Pack, located my initial bundle and unpacked each tar-ball into their own temporary folder.

I like to install Informix into a version specific directory, for example /opt/informix-ids.11.50.FC8W2, and use the soft link /opt/informix as my INFORMIXDIR. This allows me to pre install everything and then simply shutdown the engine, redirect the link and restart the engine when it is time to upgrade.

I will first install the FC3 bundle into the 11.50.FC8W2 directory by temporarily setting INFORMIXDIR to /opt/informix-ids.11.50.FC8W2 and running ids_install as root.
root:/tmp/ids-11.50.FC3> mkdir /opt/informix-ids.11.50.FC8W2
root:/tmp/ids-11.50.FC3> chown informix:informix /opt/informix-ids.11.50.FC8W2
root:/tmp/ids-11.50.FC3> export INFORMIXDIR=/opt/informix-ids.11.50.FC8W2
root:/tmp/ids-11.50.FC3> ./ids_install
After this is complete I will install the FC8W2 Fix Pack on top of the FC3 bundle as root.
root:/tmp/ids-11.50.FC8W2> export INFORMIXDIR=/opt/informix-ids.11.50.FC8W2
root:/tmp/ids-11.50.FC8W2> ./installserver 

Copy configuration files from current install directory to new install directory - HDR Primary and Secondary

We will need to copy some files from the current INFORMIXDIR to our soon to be new INFORMIXDIR. This is what I need to copy, your needs may vary slightly.
# definitely will need the ONCONFIG file
cp /opt/informix/etc/$ONCONFIG /opt/informix-ids.11.50.FC8W2/etc/

# and we'll need this file
cp /opt/informix/etc/oncfg_* /opt/informix-ids.11.50.FC8W2/etc/

# I setup my HDR trusts in the Informix hosts.equiv file
cp /opt/informix/etc/hosts.equiv /opt/informix-ids.11.50.FC8W2/etc

# if you are using a storage manager for backups, don't forget this guy
cp /opt/informix/etc/sm_versions /opt/informix-ids.11.50.FC8W2/etc

# if your sqlhosts file lives somewhere in INFORMIXDIR, don't forget to copy it
cp /opt/informix/etc/sqlhosts /opt/informix-ids.11.50.FC8W2/etc

# and finally the alarmprogram if you have made any changes
cp /opt/informix/etc/ /opt/informix-ids.11.50.FC8W2/etc
Just one more thing to do to make sure we are protected

Take a Level 0 backup, it is better to have it and not need it than to need it and not have it.

We are now ready to upgrade

We will break HDR by converting the Primary and Secondary nodes to standard engines and we will shut down the Secondary node and not touch it until we are 100% satisfied the upgrade was successful. This gives us a quick way to back out of our upgrade attempt if something goes wrong. If there is a problem we can restart the Secondary (which is now a writable standard engine) and direct the clients to this node while we figure out what went wrong.

Convert the Primary to a standard engine and shutdown
informix@primary> onmode -d standard
informix@primary> onmode -ky
Convert the Secondary to a standard engine and shutdown
informix@secondary> onmode -d standard
informix@secondary> onmode -ky
Relink /opt/informix on the Primary
informix@primary> rm /opt/informix
informix@primary> ln -s /opt/informix-ids.11.50.FC8W2 /opt/informix
Start the upgraded engine on the Primary
informix@primary> oninit -v
If everything goes as planned the upgrade will complete successfully and your primary engine will only be down for a few minutes and your downtime will be minimized.

You can use this time to verify that everything with the upgrade has gone as planned. Check the online.logs for any strange errors and watch your applications for any unexpected behaviour. When you are confident that the upgrade was successful you should take another Level 0 backup, upgrade the Secondary and rebuild HDR.

Relink /opt/informix on the Secondary and reinitialize HDR
informix@secondary> rm /opt/informix
informix@secondary> ln -s /opt/informix-ids.11.50.FC8W2 /opt/informix

Follow the instructions for initializing HDR from a previous post.

Oh Noes! Something went wrong. REVERT!

So you thought everything was hunky dory with the upgrade but after reinitializing HDR you realize you need to revert back to your previous version. Gross.

Thankfully Informix has a utility to help with this, the -b switch of the onmode utility.
> onmode -b --
You may only revert IBM Informix Dynamic Server to
IBM Informix-OnLine Version 7.31.UC1 or later:
    Use 'onmode -b 7.3'  to revert to 7.3x
    Use 'onmode -b 9.2'  to revert to 9.2x
    Use 'onmode -b 9.3'  to revert to 9.3x
    Use 'onmode -b 9.4'  to revert to 9.4x
    Use 'onmode -b 10.00' to revert to 10.00x
    Use 'onmode -b 11.10' to revert to 11.10x
    Use 'onmode -b 11.50.xC1' to revert to 11.50.xC1
    Use 'onmode -b 11.50.xC2' to revert to 11.50.xC2
    Use 'onmode -b 11.50.xC3' to revert to 11.50.xC3
    Use 'onmode -b 11.50.xC4' to revert to 11.50.xC4
    Use 'onmode -b 11.50.xC5' to revert to 11.50.xC5
For more information refer to Section II of IBM Informix Migration guide
Convert the Primary to a standard engine
informix@primary> onmode -d standard
Convert the Secondary to a standard engine and shutdown
informix@secondary> onmode -d standard
informix@secondary> onmode -ky
Execute the onmode -b command and revert disk structures back to 11.50.xC3

Informix will be offline after a successful onmode -b
informix@primary> onmode -b 11.50.xC3
Relink /opt/informix to the old INFORMIXDIR
informix@primary> rm /opt/informix
informix@primary> ln -s /opt/informix-ids-11.50.FC3 /opt/informix
Start the engine in Quiescent mode and verify everything is Ok
informix@primary> oninit -s
Bring the engine into online mode
informix@primary> onmode -m
Finally you will want to complete a Level 0 backup and reinitialize HDR.


  1. One remark: I always checks differences between my $ONCONFIG and $INFORMIXDIR/etc/onconfig.std that came from the new version to looking for new configuration parameters.

  2. I agree, you should also read the release notes and take care of any steps documented in the migration guide that I have not listed here.