Grid Infrastructure upgrade to 11.2.0.4 on AIX - always read log files

Oracle Restart 11.2.0.3 and same version of database has to be upgraded into 11.2.0.4. Looks like quite easy task but I hit some problems with it.

I had 31.9 GB free and OUI through error about disk space. Fair enough cause it also proposed a parameter to ignore it. Current version of GI (11.2.0.3) occupied around 25 GB so i though it will be OK to use a parameter.

-bash-3.2$ ./runInstaller -silent -responseFile /software/oracle/database/aix/grid_reponse_file_11204.rsp \
-waitforcompletion -ignoreDiskWarning
********************************************************************************

Your platform requires the root user to perform certain pre-installation
OS preparation.  The root user should run the shell script 'rootpre.sh' before
you proceed with Oracle installation.  rootpre.sh can be found at the top level
of the CD or the stage area.

Answer 'y' if root has run 'rootpre.sh' so you can proceed with Oracle
installation.
Answer 'n' to abort installation and then ask root to run 'rootpre.sh'.

********************************************************************************

Has 'rootpre.sh' been run by root on all nodes? [y/n] (n)
y

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 190 MB.   Actual 1265 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 16512 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-04-01_08-46-59AM. Please wait ...[WARNING] [INS-13014] Target environment do not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/installActions2014-04-01_08-46-59AM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/installActions2014-04-01_08-46-59AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
You can find the log of this install session at:
/u01/app/oraInventory/logs/installActions2014-04-01_08-46-59AM.log
You may not have enough space on your drive for a successful install. Do you still want to continue?
y
The installation of Oracle Grid Infrastructure 11g was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2014-04-01_08-46-59AM.log' for more details.

As a root user, execute the following script(s):
        1. /u01/app/grid/product/11.2.0/grid4/rootupgrade.sh


As install user, execute the following script to complete the configuration.
        1. /u01/app/grid/product/11.2.0/grid4/cfgtoollogs/configToolAllCommands RESPONSE_FILE=

        Note:
        1. This script must be run on the same host from where installer was run.
        2. This script needs a small password properties file for configuration assistants that require passwords (refer to install guide documentation).


Successfully Setup Software.

Actually note to myself - CHECK LOGS - especially oraInstall.err and silentInstall.log and to not trust OUI that installation was successful.
In next step I run I run rootupgrade but there was execution permissions errors for these files: rootadd_rdbms.sh, rootadd_filemap.sh, setowner.sh

/u01/app/grid/product/11.2.0/grid4/rootupgrade.sh[8]: /u01/app/grid/product/11.2.0/grid4/network/install/sqlnet/setowner.sh: 0403-006 Execute permission denied.
/u01/app/grid/product/11.2.0/grid4/rootupgrade.sh[13]: /u01/app/grid/product/11.2.0/grid4/rdbms/install/rootadd_rdbms.sh: 0403-006 Execute permission denied.
/u01/app/grid/product/11.2.0/grid4/rootupgrade.sh[15]: /u01/app/grid/product/11.2.0/grid4/rdbms/install/rootadd_filemap.sh: 0403-006 Execute permission denied.
Using configuration parameter file: /u01/app/grid/product/11.2.0/grid4/crs/install/crsconfig_params
Creating trace directory

I checked it on OS and well it was true

-bash-3.2$ ls -l /u01/app/grid/product/11.2.0/grid4/rdbms/install/rootadd_rdbms.sh
-rw-r--r--    1 grid     oinstall       2999 Apr 01 09:24 /u01/app/grid/product/11.2.0/grid4/rdbms/install/rootadd_rdbms.sh
-bash-3.2$ ls -l /u01/app/grid/product/11.2.0/grid4/rdbms/install/rootadd_filemap.sh
-rw-r--r--    1 grid     oinstall       1105 Apr 01 09:31 /u01/app/grid/product/11.2.0/grid4/rdbms/install/rootadd_filemap.sh
-bash-3.2$ ls -l /u01/app/grid/product/11.2.0/grid4/network/install/sqlnet/setowner.sh
-rw-r--r--    1 grid     oinstall        341 Apr 01 09:31 /u01/app/grid/product/11.2.0/grid4/network/install/sqlnet/setowner.sh

OK - maybe I could fix it but at this time I checked error log and it was much more errors

oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp100. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp101. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp102. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp103. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp104. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp105. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp106. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp107. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp108. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp109. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp110. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp111. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp112. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp113. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp114. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp115. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp116. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp117. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp118. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp119. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp120. Please create more space at this location and try again.
oracle.sysman.oii.oiif.oiifb.OiifbCancelException: OUI-10096:You may not have enough space to write to /tmp/OraInstall2014-04-01_08-46-59AM/dtemp121. Please create more space at this location and try again.
java.io.FileNotFoundException: /tmp/OraInstall2014-04-01_08-46-59AM/setpermslog.err (There is not enough space in the file system.)
 at java.io.FileOutputStream.open(Native Method)
 at java.io.FileOutputStream.(FileOutputStream.java:205)
 at java.io.FileOutputStream.(FileOutputStream.java:96)
 at oracle.sysman.oii.oiij.OiijFastCopyPhase.execPermissions(OiijFastCopyPhase.java:1245)
 at oracle.sysman.oii.oiij.OiijFastCopyPhase.doFastModeOperations(OiijFastCopyPhase.java:542)
 at oracle.sysman.oii.oiif.oiifw.OiifwInstallPhaseWCCE$OiifwComponentIP1WCCE.doOperation(OiifwInstallPhaseWCCE.java:1980)
 at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
 at oracle.sysman.oii.oiif.oiifw.OiifwInstallPhaseWCCE.doOperation(OiifwInstallPhaseWCCE.java:897)
 at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
 at oracle.sysman.oii.oiif.oiifw.OiifwActionsPhaseWCDE.doOperation(OiifwActionsPhaseWCDE.java:641)
 at oracle.sysman.oii.oiif.oiifb.OiifbLinearIterator.iterate(OiifbLinearIterator.java:147)
 at oracle.sysman.oii.oiic.OiicInstallAPISession$OiicAPISelCompsInstall.doOperation(OiicInstallAPISession.java:1095)
 at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
 at oracle.sysman.oii.oiic.OiicInstallAPISession.doInstallAction(OiicInstallAPISession.java:679)
 at oracle.sysman.oii.oiic.OiicInstallAPISession.access$000(OiicInstallAPISession.java:94)
 at oracle.sysman.oii.oiic.OiicInstallAPISession$OiicActionsThread.run(OiicInstallAPISession.java:971)

Strange enough there were errors related to /tmp space not a GRID HOME location - I have verified all logs files again:

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 190 MB.   Actual 1265 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 16512 MB    Passed

So far looks OK but

Free Space: itd11:/u01/app/grid/product/11.2.0/grid4: This is a prerequisite condition to test whether sufficient free space is available in the file system.
INFO: Severity:IGNORABLE
INFO: OverallStatus:SUCCESSFUL
INFO: -----------------------------------------------
INFO: Verification Result for Node:itd11
INFO: Expected Value:12GB 
INFO: Actual Value:30.5491GB 
INFO: -----------------------------------------------
INFO: *********************************************
INFO: Free Space: itd11:/tmp: This is a prerequisite condition to test whether sufficient free space is available in the file system.
INFO: Severity:IGNORABLE
INFO: OverallStatus:VERIFICATION_FAILED
INFO: -----------------------------------------------
INFO: Verification Result for Node:itd11
INFO: Expected Value:1GB 
INFO: Actual Value:915.3164MB 

OK there is 85 MB MB shortage in /tmp filesystem, cause some of space has been allocated already by OUI itself. I can't complain about disk error problem as I forced OUI to ignore it but I would really appreciate if OUI doesn't display successful feedback when something went wrong ;)



I do a proper cleanup of /tmp and re-run process


Note to myself : Remember do not trust output of command and always read logs


PS.
That remind me one guy I used to know who work as developer. By default all his programs end up with exit(0) C function so if you check error code there was not errors at all.

regards,
Marcin