Saturday, June 16, 2012

Oracle VM upgrade story

As I have described in my last post I stared my work with Oracle VM 3.1 but after all problems with NFS local storage I installed Oracle VM 3.0. When storage issues have been resolved it was a good time run and share information about upgrade process from version 3.0.3 to 3.1. Process itself has been split into two phases - Oracle VM upgrade and Oracle VM Manager upgrade.

Oracle VM upgrade
Oracle VM upgrade is straight forward procedure. I have booted my server from new 3.1 CD and chose upgrade option. After couple of minutes and one reboot new version was in place. Unfortunately repository based on OCFS was not mounted and none of my virtual machines started. But before I dig into it I decided to upgrade Oracle VM Manager.

Oracle VM Manager upgrade
You can download upgrade ISO image from e-delivery (V32481-01) and it has only 123 MB in size. In next step ISO image has to be mounted using loop device and upgrade script should be started.
[root@OVMiddleEarth ~]# mount -o loop /root/V32481-01.iso /mnt
[root@OVMiddleEarth ~]# cd /mnt/
[root@OVMiddleEarth mnt]# ls
components  EULA  LICENSE  runUpgrader.sh  TRANS.TBL  upgrade
[root@OVMiddleEarth mnt]# ./runUpgrader.sh
Stating OVM Manager upgrade on Thu Jun 14 13:40:16 IST 2012

Oracle VM Manager 3.1.1.305 upgrade utility
Upgrade logfile : /tmp/upgrade-2012-06-14-40.log


It is highly recommended to do a full database repository backup prior to upgrading Oracle VM Manager ...

Press any key to continue ...


Oracle VM Manager is running ...
Verifying installation status ...
Read Oracle VM Manager config file ...
Found Oracle VM Manager install files ...
Found Oracle VM Manager upgrader ...
Found Oracle WebLogic Server ...
Found Java ...
Using the following information :
  Database Host          : localhost
  Database SID           : XE
  Database LSNR          : 1521
  Oracle VM Schema       : ovs
  Oracle VM Manager UUID : 0004fb00000100000a19593edeada0d8
  Current Build ID       : 3.0.3.126
  Upgrade from version   : 3.0.3
  Upgrade to version     : 3.1.1
Using /tmp/workdir.MdYTjS6054 for backup and export location.
Using /tmp/patchdir.bGPOJ6055 for patching.
Enter password for user ovs :
Undeploying previous version of Oracle VM Manager application ...
Undeploying Oracle VM Manager help ...
Undeploying Oracle VM Manager console ...
Undeploying Oracle VM Manager core ...
Waiting for Oracle VM Manager core to fully undeploy...
Waiting...
Finished undeploying previous version ...
Exporting Oracle VM Manager repository ...
Please wait as this can take a long time ...
Oracle VM Manager repository export completed ...
Creating backup file ...
/mnt/./upgrade: line 465: /usr/bin/zip: No such file or directory
Backup of Oracle VM Manager repository failed ... Exiting upgrade process.

Aborting upgrade on Thu Jun 14 13:43:14 IST 2012 due to error
Attempt to rollback to before starting this upgrade

.... redeploy weblogic and 3.0.1 OVM Manager
Redeploying back to the 3.0.1  Oracle VM Manager core container ...
Redeploying back to the 3.0.1 Oracle VM Manager console ...
Redeploying back to the 3.0.1  Oracle VM Manager help ...
I hit a problem here - zip program could not be found. This is strange as I didn't recall this issues from Oracle VM 3.1 / Oracle VM 3.1 Manager installation. It mean that server installation process is different then server upgrade or Oracle VM Manager upgrade has different requirements. Whatever is a root cause let solve it quick. As I mentioned in my last post I have added OEL 5 repository to my Oracle VM yum configuration so I was able to run
[root@OVMiddleEarth mnt]# yum install zip
el5_latest                                                                                                                                | 1.1 kB     00:00
el5_latest/primary                                                                                                                        | 9.1 MB     00:23
el5_latest                                                                                                                                             9031/9031
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package zip.x86_64 0:2.31-2.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================================
 Package                           Arch                                 Version                                   Repository                                Size
=================================================================================================================================================================
Installing:
 zip                               x86_64                               2.31-2.el5                                el5_latest                               136 k

Transaction Summary
=================================================================================================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)

Total download size: 136 k
Is this ok [y/N]: y
Downloading Packages:
zip-2.31-2.el5.x86_64.rpm                                                                                                                 | 136 kB     00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : zip                                                                                                                                       1/1

Installed:
  zip.x86_64 0:2.31-2.el5

Complete!
[root@OVMiddleEarth mnt]#
Now I was ready to restart upgrade process
[root@OVMiddleEarth mnt]# ./runUpgrader.sh
Stating OVM Manager upgrade on Thu Jun 14 13:50:22 IST 2012

Oracle VM Manager 3.1.1.305 upgrade utility
Upgrade logfile : /tmp/upgrade-2012-06-14-50.log


It is highly recommended to do a full database repository backup prior to upgrading Oracle VM Manager ...

Press any key to continue ...


Oracle VM Manager is running ...
Verifying installation status ...
Read Oracle VM Manager config file ...
Found Oracle VM Manager install files ...
Found Oracle VM Manager upgrader ...
Found Oracle WebLogic Server ...
Found Java ...
Using the following information :
  Database Host          : localhost
  Database SID           : XE
  Database LSNR          : 1521
  Oracle VM Schema       : ovs
  Oracle VM Manager UUID : 0004fb00000100000a19593edeada0d8
  Current Build ID       : 3.0.3.126
  Upgrade from version   : 3.0.3
  Upgrade to version     : 3.1.1
Using /tmp/workdir.RLGCBY8025 for backup and export location.
Using /tmp/patchdir.aEpoE8026 for patching.
Enter password for user ovs :
Undeploying previous version of Oracle VM Manager application ...
Undeploying Oracle VM Manager help ...
Undeploying Oracle VM Manager console ...
Undeploying Oracle VM Manager core ...
Waiting for Oracle VM Manager core to fully undeploy...
Waiting...
Finished undeploying previous version ...
Exporting Oracle VM Manager repository ...
Please wait as this can take a long time ...
Oracle VM Manager repository export completed ...
Creating backup file ...
Oracle VM Manager repository backup in /tmp/ovm-manager-3-backup-2012-06-14.zip
Upgrading Oracle VM Manager ...
Backing up old files to /tmp/ovm-manager-3-backup-2012-06-14-135340...
Removing old files ...
Unpacking Oracle VM Manager 3.1.1.305
`transform_003001001000_010.xsl' -> `/tmp/patchdir.aEpoE8026/transform_003001001000_010.xsl'
`transform_003001001000_020.xsl' -> `/tmp/patchdir.aEpoE8026/transform_003001001000_020.xsl'
`deletedClasses.xml' -> `/tmp/patchdir.aEpoE8026/deletedClasses.xml'
Filtering full repository export to the selective export subset at /tmp/workdir_sel.GbPcqS8595 ...
cp: omitting directory `/tmp/workdir.RLGCBY8025/jrnl'
cp: omitting directory `/tmp/workdir.RLGCBY8025/objs'
  adding: objs/19/193.cl.xml (deflated 70%)
  adding: objs/51/519.cl.xml (deflated 72%)
  adding: objs/51/511.cl.xml (deflated 81%)
  adding: objs/10/84/10848.cl.xml (deflated 75%)
  adding: objs/10/79/10791.cl.xml (deflated 73%)
  adding: objs/42/426.cl.xml (deflated 70%)
  adding: objs/41/412.cl.xml (deflated 85%)
  adding: objs/17/07/17071.cl.xml (deflated 75%)
  adding: objs/47/472.cl.xml (deflated 74%)
  adding: objs/16/161.cl.xml (deflated 67%)
  adding: objs/9.cl.xml (deflated 93%)

Selective export is at /tmp/workdir_sel.GbPcqS8595
 11 objects selected (out of 12735) to be upgraded
Transform XSL files used:
-rw-r--r-- 1 root root 56678 Jun 14 13:53 /tmp/patchdir.aEpoE8026/transform_003001001000_010.xsl
-rw-r--r-- 1 root root 10079 Jun 14 13:53 /tmp/patchdir.aEpoE8026/transform_003001001000_020.xsl
Changed classes encountered in selective export set:

    com.oracle.ovm.mgr.api.manager.BusinessManagerDbImpl
    com.oracle.ovm.mgr.api.manager.ModelManagerDbImpl
    com.oracle.ovm.mgr.api.manager.RasManagerDbImpl
    com.oracle.ovm.mgr.api.physical.network.BondPortDbImpl
    com.oracle.ovm.mgr.api.physical.network.EthernetPortDbImpl
    com.oracle.ovm.mgr.api.physical.network.InternalPortDbImpl
    com.oracle.ovm.mgr.api.physical.ServerDbImpl
    com.oracle.ovm.mgr.api.virtual.VirtualMachineDbImpl
    com.oracle.ovm.mgr.api.virtual.VirtualMachineTemplateDbImpl
    com.oracle.ovm.mgr.api.virtual.XenHypervisorDbImpl

Upgrading Oracle VM Manager repository ...
Please wait as this can take a long time ...
Oracle VM Manager repository upgrade completed ...
Validating Oracle VM Manager repository ...
Oracle VM Manager repository validation completed ...
Refresh system-jazn-data.xml file ...
Redeploying Oracle VM Manager core container ...
Redeploying Oracle VM Manager console ...
Redeploying Oracle VM Manager help ...
Install ADF Patch ...
Completed upgrade to 3.1.1.305 ...
Writing updated config in /u01/app/oracle/ovm-manager-3/.config
Restart WebLogic ...
Stopping Oracle VM Manager                                 [  OK  ]
Starting Oracle VM Manager                                 [  OK  ]

OVM Manager upgrade finished on Thu Jun 14 13:57:31 IST 2012
[root@OVMiddleEarth mnt]#
This time it finish with successfully and I was able to login to Oracle VM Manager. 

Post upgrade changes

My first impression after login to upgraded system was that none of two OCFS file systems have been mounted. I checked system logs and looked around in system and I found that only one iSCSI target has been presented. New version of VM means new kernel and new configuration for multipath daemon. In my case second HDD (/dev/sdb) which I used as a block device to my local iSCSI server has been configured with multi path access and I had to change my iSCSI server configuration - instead of using direct path to /dev/sdb2 I need to use path presented through device mapper.
[root@OVMiddleEarth tgt]#  ls -l /dev/mapper/
total 0
brw-rw---- 1 root disk 252,   4 Jun 14 14:04 1IET_00010001
crw------- 1 root root  10, 236 Jun 14 14:04 control
brw-rw---- 1 root disk 252,   0 Jun 14 14:04 SATA_ST31000524AS_9VPCK40X
brw-rw---- 1 root disk 252,   1 Jun 14 14:04 SATA_ST31000524AS_9VPCK40Xp1
brw-rw---- 1 root disk 252,   2 Jun 14 14:04 SATA_ST31000524AS_9VPCK40Xp2
brw-rw---- 1 root disk 252,   3 Jun 14 14:04 SATA_ST31000524AS_9VPCK40Xp3
[root@OVMiddleEarth tgt]#
When I found out what is a mapper name for my device I have changed tgtd configuration and rebooted server.
[root@OVMiddleEarth ~]# vi /etc/tgt/targets.conf
...

    backing-store /dev/mapper/SATA_ST31000524AS_9VPCK40Xp2
    backing-store /etc/tgt/small_disk
    write-cache off


I helped and now it least cluster heartbeat file system has been mounted but still no repository. Solution has simple but it took me some time to find it out. I need to rescan all disks in Storage section of Oracle VM Manager and acknowledge all events in Repository section.

Lesson Learned
Event view and acknowledge of all previous errors helps in some stages and made a clean view of current state of Oracle VM.

regards,
Marcin

0 comments: