Unsafe deinstall using Oracle Univeral Installer.

Are Oracle Homes totally independent from OUI point of view? This was my impression until yesterday when I hit Oracle OUI de-installer bug in Oracle Real Application Cluster environment. I did 3 simple steps:

• I have installed new Oracle Home into new directory on both nodes
• There was timing issue between servers so I need to remove new Oracle Home using OUI de-install functionality.
• I have corrected timing issue and installed new Oracle Home once again.

All old homes have been used by working Oracle ClusterWare, ASM and database and I believed that there should not be any issues. Unfortunately a few minutes after that work I have got first error message but only for new session:

ORA-27504: IPC error creating OSD context 
ORA-27300: OS system dependent operation: IPC init failed with status:65 
ORA-27301: OS failure message: Package not installed 
ORA-27302: failure occurred at: skgxpcini 
ORA-27303: additional information: libskgxpd.so called

I was surprised and I have checked $ORACLE_HOME/lib/ directory and I have found root cause - libskgxpd.so has very recent modification date and it was date of my installation. I have checked all OUI logs immediately looking for that file and in deinstallation log I found that entry:

INFO: The output of this make operation is also available at: '/opt/app/oracle/product/10.2.0/Db_new/install/make.log'
INFO:
INFO: Start output from spawned process:
INFO: ----------------------------------
INFO:
INFO: rm -f /opt/app/oracle/product/10.2.0/Db_1/lib/libskgxp10.so
INFO: cp /opt/app/oracle/product/10.2.0/Db_1/lib//libskgxpd.so /opt/app/oracle/product/10.2.0/Db_1/lib/libskgxp10.so

But in that same log I have found following message at the end:

INFO: Current Inventory:
        Oracle Home: OraCrs10g_home
                This Oracle Home has not been changed in this session.
        Oracle Home: ASM_HOME
                This Oracle Home has not been changed in this session.
        Oracle Home: RDBMS_HOME
                This Oracle Home has not been changed in this session.
INFO: This deinstallation was successful

So OUI replaced library libskgxp10.so in existing old home (RDBMS_HOME) and at the end of work stated that this home has not been change. There are bugs related to that : 7006848, 5474623.

If you ever hit that error – there is two possible solutions:

1. Copy backup libskgxp10.so from untouched home if you have home with this same patch level (in my case it was ASM)
   
2. Rebuild Oracle using the following steps   
   
   • cd $ORACLE_HOME/rdbms/lib   
     rename the original library (if exists)
     mv libskgxp10.so libskgxp10.so.old  
   • Relink to configure UDP for IPC    
     make -f ins_rdbms.mk rac_on ipc_udp ioracle   
   • Check whether the library exists    
     ls -l $ORACLE_HOME/lib/libskgxp10.so
   • startup the instance

And good advise at the end – your testing environment should be same as production one. If you are going to do some work on RAC test it on your test RAC too.