Oracle Linux and ESX hot add cpu and memory

I'm still quite new in Vmware ESX environment and features which I'm using now as my lab (curiosity of VMWare Guru Program).
This time I decided to test hot add CPU and memory functionality. It looks very interesting and in past years hot cpu / memory games where restricted to sophisticated hardware only - now everybody can test is at home.

My lab is basesd on Intel 4 cores and VM are running Oracle Linux 6.3 with or without Oracle Enterprise Kernel. Here are my findings:

1. Hot add CPU / memory is disabled by default and has to be enabled when VM is down. To do so you have to edit your VM configuration and go to options.

2. When feature is enabled you can ADD CPU or memory by editing your VM configuration. Keep in mind that there is no possibility to decrease number of CPU or memory.

3. Here is a table with supported options for ESX 5i and Oracle Linux with or without UEK

I tested adding a CPU to system for RedHat and UEK kernel. As you can find in table linked above adding memory is supported for RedHat kernels only.

RedHat kernel

OEL 6.3 with RedHat kernel running with 1 CPU

[root@oraclese ~]# uname -a
Linux oraclese.localdomain 2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23 12:05:59 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@oraclese ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dts
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

New CPU has been added in vSphere client. Below is output from messages log.

Jan 13 15:17:25 oraclese kernel: SMP alternatives: switching to SMP code
Jan 13 15:17:25 oraclese kernel: Booting Node 0 Processor 1 APIC 0x1
Jan 13 15:17:25 oraclese kernel: Disabled fast string operations
Jan 13 15:17:25 oraclese kernel: mce: CPU supports 0 MCE banks
Jan 13 15:17:25 oraclese kernel: x86 PAT enabled: cpu 1, old 0x0, new 0x7010600070106
Jan 13 15:17:25 oraclese kernel: Skipped synchronization checks as TSC is reliable.
Jan 13 15:17:25 oraclese kernel: microcode: CPU1 sig=0x206a7, pf=0x1, revision=0x14
Jan 13 15:17:25 oraclese kernel: platform microcode: firmware: requesting intel-ucode/06-2a-07
Jan 13 15:17:25 oraclese firmware.sh[2070]: Cannot find  firmware file 'intel-ucode/06-2a-07'
Jan 13 15:17:25 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:17:25 oraclese kdump: failed to stop
Jan 13 15:17:25 oraclese kdump: No crashkernel parameter specified for running kernel

Now system is running with two CPU's

[root@oraclese ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dts
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dts
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Adding memory - OEL 6.3 wtih RedHat kernel - system running with 1 GB of RAM

[root@oraclese ~]# uname -a
Linux oraclese.localdomain 2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23 12:05:59 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@oraclese ~]# free
             total       used       free     shared    buffers     cached
Mem:       1016680     190384     826296          0       4736      77112
-/+ buffers/cache:     108536     908144
Swap:      2097144          0    2097144

Additional 1 GB has been added via vSphere Client. Below is output from messages log.

Jan 13 15:40:54 oraclese kernel: init_memory_mapping: 0000000040000000-0000000048000000
Jan 13 15:40:54 oraclese kernel: Built 1 zonelists in Node order, mobility grouping on.  Total pages: 274190
Jan 13 15:40:54 oraclese kernel: Policy zone: Normal
Jan 13 15:40:54 oraclese kernel: init_memory_mapping: 0000000048000000-0000000050000000
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kernel: init_memory_mapping: 0000000050000000-0000000058000000
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kernel: init_memory_mapping: 0000000058000000-0000000080000000
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel

Now Linux reconized 2 GB of RAM

[root@oraclese ~]# free
             total       used       free     shared    buffers     cached
Mem:       2065256     211840    1853416          0       4764      77328
-/+ buffers/cache:     129748    1935508
Swap:      2097144          0    2097144

Oracle UEK release 2 kernel

OEL 6.3 running with Oracle UEK release 2

[root@oraclese ~]# uname -a
Linux oraclese.localdomain 2.6.32-300.3.1.el6uek.x86_64 #1 SMP Fri Dec 9 18:57:35 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@oraclese ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

New CPU has been added in vSphere client. Below is output from messages log.

Jan 13 15:38:12 oraclese kernel: processor LNXCPU:01: registered as cooling_device1
Jan 13 15:38:12 oraclese kernel: SMP alternatives: switching to SMP code
Jan 13 15:38:12 oraclese kernel: Booting processor 1 APIC 0x1 ip 0x6000
Jan 13 15:38:12 oraclese kernel: Initializing CPU#1
Jan 13 15:38:12 oraclese kernel: CPU: Physical Processor ID: 2
Jan 13 15:38:12 oraclese kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jan 13 15:38:12 oraclese kernel: CPU: L2 cache: 256K
Jan 13 15:38:12 oraclese kernel: CPU: L3 cache: 6144K
Jan 13 15:38:12 oraclese kernel: CPU 1/0x1 -> Node 0
Jan 13 15:38:12 oraclese kernel: mce: CPU supports 0 MCE banks
Jan 13 15:38:12 oraclese kernel: x86 PAT enabled: cpu 1, old 0x0, new 0x7010600070106
Jan 13 15:38:12 oraclese kernel: CPU1: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz stepping 07
Jan 13 15:38:12 oraclese kernel: Skipping synchronization checks as TSC is reliable.
Jan 13 15:38:12 oraclese kernel: microcode: CPU1 sig=0x206a7, pf=0x1, revision=0x14
Jan 13 15:38:12 oraclese kernel: platform microcode: firmware: requesting intel-ucode/06-2a-07
Jan 13 15:38:12 oraclese firmware.sh[2074]: Cannot find  firmware file 'intel-ucode/06-2a-07'
Jan 13 15:38:12 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:38:12 oraclese kdump: failed to stop
Jan 13 15:38:12 oraclese kdump: No crashkernel parameter specified for running kernel

Now system is running with two CPU's

[root@oraclese ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

regards, Marcin

Oaktable videos

Hi,

Short update - I have upload new video from OakTable World 2012 - Brendan Gregg -- USE Method for Performance Tuning.

All videos and presentations can be found here - OakTable world

regards,
Marcin

How to rename NIC in OEL 6.X

This is a quick note about host cloning and network interface (ethX) renaming.
As I wrote in my last post I have ESX lab now and I create a template of OEL 6.3 with all my settings and I would like use it for adding new VM (including RAC nodes). It is easy but there is one small issue – every time you clone VM network devices are renamed. Well MAC address is unique so it doesn’t surprise me but I would like to keep interface names like eth0, eth1 and eth2 and I got eth0, eth3, eth4 instead. Since OEL 5 (Redhat 5) all device name are generated by udev mechanism so I start my research there.
Well it was easier than I thought – there is a file in "/udev/rules.d" directory called "70-persistent-net.rules".
And here is file content:

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:ce:65:67", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:95", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:94", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:98", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:99", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

Well this is it – all I need to do was to run "ifconfig –a" and confirm MAC addresses of existing interfaces. I have removed old entries and change interface names. My final configuration file looks like that

[root@localhost rules.d]# cat 70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:ce:65:67", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:95", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:94", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

After changes I rerun udev mechanism

[root@localhost rules.d]# start_udev
Starting udev:  

And finally I have got correct configuration. Next lesson learned.
Regards,
Marcin

New lab setup

After couple of month playing with Oracle VM I decided to install and configure VMWare. Thanks to VMware Guru Program I’m able to test ESX 5i with one year license. This is excellent opportunity for me to get more familiar with newest VMware solutions. I have been using VMware tools for years and I remember times when I deployed my first Oracle RAC 9i on Workstation 3.x and had to "hack" configuration file to change disk.locking parameter to false.

When I start instalation I have run into couple of issues as I decided to leave my current disk setup and I had only one free disk. New ESX is using GPT partition table and I couldn’t use it together with Grub to start Linux or VMware. This is when USB sticks went to play and I have installed ESX 5i on it and force my PC to boot from USB only. With other USB with Grub I have boot selection on my headless PC based on which USB key is connected at boot time. First problem solved.
Next issue appears to when I created my first VM and want to clone it – there was no such option in Windows vSphere Client. I read a manual (frankly I just googled) and I found out that I need to install vSphere 5 vCenter Server to be able to clone my VM automatically. Well I don’t have other host for it but I found out that I can download a VM image and deploy it on same ESX host. Here is a link to very nice instruction - vSphere vCenter virtual appliance quick start guide and you can find documentation here - vSphere documentation.
After I downloaded and installed vCenter and used vSphere Client to connect into I was able to use all features I was looking for. Actually it is strange for me that I need to have additional management server to clone VM. But I can’t complain Oracle VM need management server even to start VM so I would say both are quite similar in terms of management overhead for home usage. I can give VMWare stack one point more as you can stop vCenter and keep it down if you don’t need all features implemented in it.

I'm using now my new lab as Oracle server in various configurations and also as development box and additional workstation. I spend some time trying to setup remote desktop from my laptop to workstation with sound but without luck. I have tried FreeNX and NoMachine solution but setting audio transport was too much for me. After couple of days thanks to Twitter discussion with @simon_haslam peoples from ThinLinc joined it I have been told that we can use their remote desktop solution for free up to 10 connection. And that is what I was looking for  - remote desktop with video and audio support. Good work ThinLinc.

Now time for my TODO list with my little virtual server:
- install free version of Cisco Nexus V1000 switch and play with RAC NIC failovers
- Learn Hadoop and Oracle DB Hadoop integration
- Install Cloudera software (including newest Impala)
- extend day to 48h to have time to do all above ;)

regards,
Marcin

DataGuard and Oracle Restart - how to make it work

If you are going to implement Oracle Data Guard together with Oracle Restart you should be aware that there is a configuration problem in version 11.2.0.3 (and probably in other 11.2. as well). Oracle Restart is not checking what is a current database role and it is going to start standby instance in OPEN mode. This can end up with license issue if your Data Guard Broker will start applying process on standby database. If you don't have license to use Active Data Guard you just broken your license agreement.

There is couple of possible solutions:

• manually set up "mount" state as start up mode for standby database and change it after every switch- or fail-over
• disable MRP functionality on standby database and keep it open in read only mode 
• add your own script to open primary database only and keep standby in "mount" state

I want to describe last solution and share script which I have created for it.

1. First of all you have to change your Oracle Restart configuration to open both databases (standby and primary) in "mount" state using following command.  

   [oracle@testdb1 ~] srvctl modify database -d database_name -s MOUNT
   

2. In next step user script has to be created - you can use this one as an template. I have based it on CRS demo script.Script has been save in /tmp directory on both server using name "opendb"
   

   #!/bin/sh
   # These messages goes into the CRSD agent log file.
   echo " *******   `date` ********** "
   echo "Action script '$_CRS_ACTION_SCRIPT' for resource[$_CRS_NAME] called for action $1"
   #env
   #
   
   #setup database home
   export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db3
   #setup database SID
   export ORACLE_SID=testa
   DBROLE=''
   
   
   # check database role and return following values
   # OPENED if DB is open and it is primary
   # PRIMARY if DB is mounted and DB role is primary
   # PHYSICAL is DB is mounted and DB role is standby
   
   function getrole() {
   
   $ORACLE_HOME/bin/sqlplus -s / as sysdba << EOF
        spool /tmp/getrole.tmp
        set echo off feedback off head off
        select case when OPEN_MODE = 'READ WRITE' then 'OPENED' when OPEN_MODE='MOUNTED' and DATABASE_ROLE='PRIMARY' then 'PRIMARY' when OPEN_MODE='MOUNTED' and DATABASE_ROLE='PHYSICAL STANDBY' then 'PHYSICAL' end from v\$database;
        exit
   EOF
   DBROLE=`cat /tmp/getrole.tmp | sed 's/[ \t]*$//' | sed 's/^[ \t]*//' | tail -1`
   
   }
   
   
   case "$1" in
     'start')
        echo "START entry point has been called.."
        getrole
        # check role and do following actions
        case $DBROLE in
          'PHYSICAL')   echo "This is standby - do nothing" ;;
          'PRIMARY')
        $ORACLE_HOME/bin/sqlplus -s / as sysdba << EOF
        alter database open;
        exit
   EOF
        echo ;;
        esac
        exit 0
        ;;
   
     'stop')
        echo "STOP entry point has been called.."
        exit 0
        ;;
   
     'check')
       echo "CHECK entry point has been called.."
       getrole
       if [ "$DBROLE" == 'OPENED' ] || [ "$DBROLE" == 'PHYSICAL' ]; then
           echo "Check -- SUCCESS"
           exit 0
       else
           echo "Check -- FAILED"
           exit 1
       fi
       ;;
   
     'clean')
        echo "CLEAN entry point has been called.."
        exit 0
        ;;
   
   esac
   

3. Register user script in Oracle Restart and set up dependencies using following command. Ora.testa.db is my database resource name in CRS - please change to your database resource name. New resource is called "ora.opendb" and it has to be registered on both servers and started.There is hard dependency between my resource and database resource so my script will be started only when database has been started as well.
   

   [oracle@testdb1 ~]$ /u01/app/oracle/product/11.2.0/grid3/bin/crsctl add resource ora.opendb -type cluster_resource \
   > -attr "ACTION_SCRIPT=/tmp/opendb,CHECK_INTERVAL=30,RESTART_ATTEMPTS=2 \
   > ,START_DEPENDENCIES=hard(intermediate:ora.testa.db)"
   
   [oracle@testdb1 ~]$ /u01/app/oracle/product/11.2.0/grid3/bin/crsctl start resource ora.opendb
   

4. Check status of resources.

   [oracle@testdb1 ~]$ /u01/app/oracle/product/11.2.0/grid3/bin/crsctl status res -t
   --------------------------------------------------------------------------------
   NAME           TARGET  STATE        SERVER                   STATE_DETAILS
   --------------------------------------------------------------------------------
   Local Resources
   --------------------------------------------------------------------------------
   ora.LISTENER.lsnr
                  ONLINE  ONLINE       testdb1
   ora.ons
                  OFFLINE OFFLINE      testdb1
   --------------------------------------------------------------------------------
   Cluster Resources
   --------------------------------------------------------------------------------
   ora.cssd
         1        OFFLINE OFFLINE
   ora.diskmon
         1        OFFLINE OFFLINE
   ora.evmd
         1        ONLINE  ONLINE       testdb1
   ora.opendb
         1        ONLINE  ONLINE       testdb1
   ora.testa.db
         1        ONLINE  ONLINE       testdb1                  Open
   
   

When everything is deployed is it running in following way: Oracle Restart is starting both data bases in mount state and then next resource (user script) is started. Script is checking current database role and it is opening primary database and doesn't do any action on standby database. 

Disclamer:
Please check script in your development and test environment before you deploy it and change anything in production environment. This script has been created for my personal tests and there is no guarantee that it is bug free.

regards,
Marcin

 

 

Oracle Grid Cloud Control 12c BP1 on Oracle Enterprise Linux 6.1

I have installed it in my test box and I hit a problem with auto-discovery due to missing libssl.so library for nmap. Issue is very easy to fix using common Linux troubleshooting approach. First what versions I'm using:

[root@oem-server ~]# cat /etc/oracle-release
Oracle Linux Server release 6.1
[root@oem-server ~]# openssl version
OpenSSL 1.0.0-fips 29 Mar 2010
[root@oem-server ~]# yum list | grep -e "^openssl"
openssl.x86_64                           1.0.0-20.el6_2.5            @ol6_latest
openssl.i686                             1.0.0-20.el6_2.5            ol6_latest
openssl-devel.i686                       1.0.0-20.el6_2.5            ol6_latest
openssl-devel.x86_64                     1.0.0-20.el6_2.5            ol6_latest
openssl-perl.x86_64                      1.0.0-20.el6_2.5            ol6_latest
openssl-static.x86_64                    1.0.0-20.el6_2.5            ol6_latest
openssl098e.i686                         0.9.8e-17.0.1.el6_2.2       ol6_latest
openssl098e.x86_64                       0.9.8e-17.0.1.el6_2.2       ol6_latest

Now what kind of issue I had

[root@oem-server ~]# /u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap
/u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap: error while loading shared libraries: libssl.so.4: cannot open shared object file: No such file or directory

How I fixed it

[oracle@oem-server ~]$  ldd /u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap
        linux-vdso.so.1 =>  (0x00007fff447ff000)
        libssl.so.4 => not found
        libcrypto.so.4 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003d40800000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003d43c00000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003d41000000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003d43400000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003d40400000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003d40000000)

[root@oem-server ~]# ln -s /usr/lib64/libssl.so.10 /usr/lib64/libssl.so.4
[root@oem-server ~]# ln -s /usr/lib64/libcrypto.so.10 /usr/lib64/libcrypto.so.4

[oracle@oem-server ~]$  ldd /u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap
        linux-vdso.so.1 =>  (0x00007fffb0686000)
        libssl.so.4 => /usr/lib64/libssl.so.4 (0x00007fd6b1c0a000)
        libcrypto.so.4 => /usr/lib64/libcrypto.so.4 (0x00007fd6b1870000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003d40800000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003d43c00000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003d41000000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003d43400000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003d40400000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x0000003d48c00000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x0000003d46c00000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003d44000000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x0000003d47800000)
        libz.so.1 => /lib64/libz.so.1 (0x0000003d41400000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003d40000000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x0000003d46800000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003d47000000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003d42400000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003d40c00000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003d41c00000)

Now it is working

[oracle@oem-server ~]$ /u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap -v

Starting Nmap 5.51.3 ( http://nmap.org ) at 2012-07-08 19:19 IST
Unable to find nmap-services!  Resorting to /etc/services
Read data files from: /etc
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 0.03 seconds
[oracle@oem-server ~]$

After that I checked MOS and Grid Control 12c is certified with OEL 6 so this errors was not expected but then I realized that I didn't read know issues for Oracle® Enterprise Manager Cloud Control Support Notes for Linux x86 and x86-64 and I found it in point called "Host Discovery Job Displays Error While Loading Shared Libraries". Official solution is a little bit more complicated and required downloading openssl sources and compiling it. Anyway I found article above using Google and I couldn't find it on entry documentation page.

regards,
Marcin

RMAN Transportable tablespace bug

I was a long time since I last used Transportable Tablespaces and I decided to refresh my memory and setup a test environment for Streams using TTS feature. I was going to use example script from Oracle Streams Documentation which is creating necessary data files for specified point of time (no need to switch source tablespace into read only mode) and also is running Data Pump creating meta data file. I have done necessary  changes and run it.

[oracle@testdb1 oracle]$ rsp / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Mon Aug 6 12:50:04 2012
Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Session altered.

SET SERVEROUTPUT ON SIZE 1000000
DECLARE
  until_scn NUMBER;
BEGIN
  until_scn:= DBMS_FLASHBACK.GET_SYSTEM_CHANGE_NUMBER;
      DBMS_OUTPUT.PUT_LINE('Until SCN: ' || until_scn);
SYS@testa AS SYSDBA >   2    3    4    5    6  END;
  7  /
Until SCN: 6300731

PL/SQL procedure successfully completed.
SYS@testa AS SYSDBA > exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

[oracle@testdb1 oracle]$ rlwrap rman target /
Recovery Manager: Release 11.2.0.3.0 - Production on Mon Aug 6 12:50:33 2012
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
connected to target database: TESTA (DBID=243207980)

RMAN>  RUN
2>     {
        TRANSPORT TABLESPACE 'SOE_TS'
        UNTIL SCN 6300731
        TABLESPACE DESTINATION '/u01/app/oracle/dest';
        DATAPUMP DIRECTORY SOURCE_DIR
        AUXILIARY DESTINATION '/u01/app/oracle/aux_files'
        DUMP FILE 'soe_ts.dmp'
        EXPORT LOG 'soe.log' 
        IMPORT SCRIPT 'soe_ts.sql'
        TABLESPACE DESTINATION '/u01/app/oracle/dest';
       }

using target database control file instead of recovery catalog
RMAN-05026: WARNING: presuming following set of tablespaces applies to specified point-in-time

List of tablespaces expected to have UNDO segments
Tablespace SYSTEM
Tablespace UNDOTBS2
...

Unfortunately there is a RMAN bug related to export log file. Due to it first attempt end up with following error

contents of Memory Script:
{
# make read only the tablespace that will be exported
sql clone 'alter tablespace  "SOE" read only';
}
executing Memory Script

sql statement: alter tablespace  "SOE" read only

Performing export of metadata...

Removing automatic instance
shutting down automatic instance
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of transport tablespace command at 08/05/2012 23:18:47
RMAN-06136: ORACLE error from auxiliary database: ORA-01097: cannot shutdown while in a transaction - commit or rollback first
RMAN-06136: ORACLE error from auxiliary database: ORA-01460: unimplemented or unreasonable conversion requested

RMAN>

At first sight error message has nothing to do with log file but I know that RMAN error messaging is well .. different is a good word here. I search for solution and I found out that workaround is quite simple - EXPORT LOG clause has to be removed from script. So let's run it again

[oracle@testdb1 oracle]$ rlwrap rman target /
Recovery Manager: Release 11.2.0.3.0 - Production on Mon Aug 6 12:50:33 2012
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
connected to target database: TESTA (DBID=243207980)

RMAN>  RUN
2>       {
        TRANSPORT TABLESPACE 'SOE_TS'
        UNTIL SCN 6300731
        TABLESPACE DESTINATION '/u01/app/oracle/dest';
        DATAPUMP DIRECTORY SOURCE_DIR
        AUXILIARY DESTINATION '/u01/app/oracle/aux_files'
        DUMP FILE 'soe_ts.dmp'
        #EXPORT LOG 'soe.log' - bug
        IMPORT SCRIPT 'soe_ts.sql'
        TABLESPACE DESTINATION '/u01/app/oracle/dest';
      }

using target database control file instead of recovery catalog
RMAN-05026: WARNING: presuming following set of tablespaces applies to specified point-in-time

List of tablespaces expected to have UNDO segments
Tablespace SYSTEM
Tablespace UNDOTBS2
...
contents of Memory Script:
{
# make read only the tablespace that will be exported
sql clone 'alter tablespace  "SOE_TS" read only';
}
executing Memory Script

sql statement: alter tablespace  "SOE_TS" read only

Performing export of metadata...
   EXPDP> FLASHBACK automatically enabled to preserve database integrity.
   EXPDP> Starting "SYS"."TSPITR_EXP_khar":
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/PLUGTS_BLK
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/TABLE
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/INDEX/INDEX
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/INDEX/FUNCTIONAL_INDEX/INDEX
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/CONSTRAINT/CONSTRAINT
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/INDEX_STATISTICS
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/INDEX/STATISTICS/FUNCTIONAL_INDEX/INDEX_STATISTICS
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/CONSTRAINT/REF_CONSTRAINT
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/TABLE_STATISTICS
   EXPDP> Processing object type TRANSPORTABLE_EXPORT/POST_INSTANCE/PLUGTS_BLK
   EXPDP> Master table "SYS"."TSPITR_EXP_khar" successfully loaded/unloaded
   EXPDP> ******************************************************************************
   EXPDP> Dump file set for SYS.TSPITR_EXP_khar is:
   EXPDP>   /u01/app/oracle/dest/soe_ts.dmp
   EXPDP> ******************************************************************************
   EXPDP> Datafiles required for transportable tablespace SOE_TS:
   EXPDP>   /u01/app/oracle/dest/soe_ts01.dbf
   EXPDP> Job "SYS"."TSPITR_EXP_khar" successfully completed at 12:55:35
Export completed

Now looks much better - tablespace has been exported and import script has been generated as well. Now I'm ready for next steps and can continue my work with Streams.

regards,
Marcin