How dNFS database clone works – part 1

There is new feature in Oracle 11.2.0.2 called dNFS clone. It has been described by Kevin Closson on his blog post - Oracle Database 11g Direct NFS Clonedb Feature  and very good configuration description has been posted by Tim Hall on his blog - Direct NFS (DNFS) Clonedb in Oracle Database 11g Release 2 (Patchset 11.2.0.2). I have played with it just after I found both blogs but never think how Oracle implemented that feature. I came back to it when my colleague asked me if I ever use that in production environment and what are performance implications of using it. 

Oracle is using Direct NFS option as Oracle Disk Manager 

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0

(is there any other still in use ? I remember one – Veritas Quick I/O module for Veritas filesystem / volume manager). As far as I understand ODM is a library which is providing IO functionality for database. As Oracle can be used on different hardware and Oracle kernel can call ODM functions to read or write data from storage and from Oracle kernel perspective it doesn’t really matter how ODM fulfill IO request.

When I started thinking how Oracle implemented that feature, two ideas came into my mind:

• Oracle keep in-memory map of changed blocks and know where to go to find latest block – actually I never believe in it – this too costly solution 
• Oracle is just doing read from both sources (backup and NFS) and it is comparing which block is newer if necessary

But guessing work is against BAAG and I want to check what database is actually doing when we are using dnfs clone. I have started with two scenarios – read data which have been changed after clone has been established and read unchanged data. Thanks to Oracle instrumentation (trace 10046), strace and wireshark I see much more details now but still there is long way to have whole picture. Let me share my findings with you.

Read changed data in clone 

I added one row to scott.bonus table which was empty when I cloned database and I executed simple query – select * from scott.bonus – below is a mixture output from all three monitoring sources

EXEC #140195143601328:c=0,e=42,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=953918076,tim=2013-01-17 14:29:43:108500
WAIT #140195143601328: nam='SQL*Net message to client' ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=13891 tim=2013-01-17 14:29:43:108660
WAIT #140195143601328: nam='Disk file operations I/O' ela= 5 FileOperation=2 fileno=4 filetype=2 obj#=75337 tim=2013-01-17 14:29:43:108800

First WAIT line with disk OS CALL (thanks to @CaryMillsap Method R training I started to thinking about WAIT lines as I should before) has no strace nor wireshark activity and call itself is very short – 5us –so I just ignored it for time being.

statfs("/u01/app/oracle/backup/data_data_D-TESTSE_I-4169104998_TS-USERS_FNO-4_10nvidnv", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=5158559, f_bfree=2226212, f_bavail=1964171, f_files=1310720, f_ffree=1268869, f_fsid={1982750833, 2010996514}, f_namelen=255, f_frsize=4096})
open("/u01/app/oracle/backup/data_data_D-TESTSE_I-4169104998_TS-USERS_FNO-4_10nvidnv", O_RDWR|O_DSYNC) = 11
fcntl(11, F_DUPFD, 256) = 257
close(11)               = 0
fcntl(257, F_SETFD, FD_CLOEXEC) = 0
fstatfs(257, {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=5158559, f_bfree=2226212, f_bavail=1964171, f_files=1310720, f_ffree=1268869, f_fsid={1982750833, 2010996514}, f_namelen=255, f_frsize=4096}) = 0
WAIT #140195143601328: nam='Disk file operations I/O' ela= 182 FileOperation=2 fileno=4 filetype=2 obj#=75337 tim=2013-01-17 14:29:43:109050
WAIT #140195143601328: nam='Disk file operations I/O' ela= 160 FileOperation=2 fileno=4 filetype=2 obj#=75337 tim=2013-01-17 14:29:43:109150

There are next two i/o related calls and above we can see that backup file has been opened , handler has been duplicated and fstatfs call has been executed. There is another mystery here I have seen couple of system calls and then two entries from Oracle trace files – not sure why it look like this – as we are dealing here on microsecond level it is hard to match tim / ela from Oracle trace file with actuall OS calls from strace – but I will try to do it later - in next post probably. Right now we now that Oracle process opened backup file and it is ready to read from it. 
Query is using full scan on table so first of all Oracle need to read segment header

sendmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0\0\214\252\16\236\235\0\0\0\0\0\0\0\2\0\1\206\243\0\0\0\3\0\0\0\6\0\0\0\1\0\0\0000\253\315\357\253\0\0\0\24oraclese.localdomain\0\0\3241\0\0\3241\0\0\0\2\0\0\3241\0\0\3242\0\0\0\0\0\0\0\0\0\0\0$\1\0\7\1\1\240\202\3\0\0\0\0\222\364,

V3 READ Call (Reply In 44853), FH:0x8f3a5fe4 Offset:256131072 Len:8192,44846,2013-01-17 14:29:43.109264,192.168.1.60,192.168.1.20,NFS,210

poll([{fd=32, events=POLLIN}], 1, 500) = 1 ([{fd=32, revents=POLLIN}])
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0 \200\252\16\236\235\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\201\240\0\0\0\1\0\0\3241\0\0\3241\0\0\0\0\21X \0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\227\252.\312\216\16\364\222\0\0\0\0\4\207`\25P\370\n[\0\0\0\
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0 \0\0\0\0\0\0004\24\0\0\0\0\0\0!z\0\1\1\0\0\0 z\0\1!z\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0I&\1\0\0\0\0\20 z\0\1\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\

V3 READ Reply (Call In 44846) Len:8192,44853,2013-01-17 14:29:43.144922,192.168.1.20,192.168.1.60,NFS,1150

recvmsg(32, 0x7fffe701bc20, MSG_DONTWAIT|MSG_NOSIGNAL) = -1 EAGAIN (Resource temporarily unavailable)

WAIT #140195143601328: nam='db file sequential read' ela= 36202 file#=4 block#=31266 blocks=1 obj#=75337 tim=2013-01-17 14:29:43:145410

We can see that process is using sendmsg function sending NFS READ call (captured by wireshark) and it is waiting for answer. Due to my NFS and network configuration respond has been split into two recvmsg system calls but it is one NFS READ Reply – one database block ( 8kB) has been read and this action has been registered in trace file as db file sequential read. Now process is going to execute full scan and read blocks from table

sendmsg(32, {msg_name(0)=NULL, msg_iov(2)=[{"\200\0\0\214\252\16\236\237\0\0\0\0\0\0\0\2\0\1\206\243\0\0\0\3\0\0\0\6\0\0\0\1\0\0\0000\253\315\357\253\0\0\0\24oraclese.localdomain\0\0\3241\0\0\3241\0\0\0\2\0\0\3241\0\0\3242\0\0\0\0\0\0\0\0\0\0\0$\1\0\7\1\1\240\202\3\0\0\0\0\222\364,V3 READ Call (Reply In 44888), FH:0x8f3a5fe4 Offset:256139264 Len:32768  ; 
V3 READ Call (Reply In 44888), FH:0x8f3a5fe4 Offset:256139264 Len:32768
V3 READ Call (Reply In 44862), FH:0x8f3a5fe4 Offset:256172032 Len:8192
poll([{fd=32, events=POLLIN}], 1, 500) = 1 ([{fd=32, revents=POLLIN}])
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0 \200\252\16\236\240\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\201\240\0\0\0\1\0\0\3241\0\0\3241\0\0\0\0\21X \0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\227\252.\312\216\16\364\222\0\0\0\0\4\207`\25P\370\n[\0\0\0\
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0,
V3 READ Reply (Call In 44855) Len:8192,44862,2013-01-17 14:29:43.158250,192.168.1.20,192.168.1.60,NFS,1150,,,,
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0\200\200\252\16\236\237\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\201\240\0\0\0\1\0\0\3241\0\0\3241\0\0\0\0\21X \0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\227\252.\312\216\16\364\222\0\0\0\0\4\207`\25P\370\n[\0\0
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
V3 READ Reply (Call In 44855) Len:32768,44888,2013-01-17 14:29:43.159262,192.168.1.20,192.168.1.60,NFS,1110,,,,
recvmsg(32, 0x7fffe7016aa0, MSG_DONTWAIT|MSG_NOSIGNAL) = -1 EAGAIN (Resource temporarily unavailable)
WAIT #140195143601328: nam='db file scattered read' ela= 13870 file#=4 block#=31267 blocks=5 obj#=75337 tim=2013-01-17 14:29:43:159580

There are two NFS READ calls and two NFS READ Reply and 5 blocks has been read – it exactly match Oracle trace file entry – db file scattered read – with blocks equal to 5. So this scenario is straight forward – Oracle is going directly to NFS to read changed blocks – but still I can’t be sure how it decide to go there. Let’s check now what will happen when we try to read unchanged blocks.

Read unchanged blocks.  

Oracle is executing query – select * from scott.salgrade – and here is a trace / strace / wireshark output. Process is executing table full scan so it need to read segment header first

sendmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0\0\214\252\16\236\245\0\0\0\0\0\0\0\2\0\1\206\243\0\0\0\3\0\0\0\6\0\0\0\1\0\0\0000\253\315\357\253\0\0\0\24oraclese.localdomain\0\0\3241\0\0\3241\0\0\0\2\0\0\3241\0\0\3242\0\0\0\0\0\0\0\0\0\0\0$\1\0\7\1\1\240\202\3\0\0\0\0\222\364\16\216\312.\252\227\0\0\0\0\0\0\0\0\25`\ 
V3 READ Call (Reply In 44938), FH:0x8f3a5fe4 Offset:1327104 Len:8192,44930,2013-01-17 14:29:55.162780,192.168.1.60,192.168.1.20,NFS,210    
sendmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0\0X\252\16\236\246\0\0\0\0\0\0\0\2\0\1\206\243\0\0\0\3\0\0\0\0\0\0\0\1\0\0\0000\253\315\357\253\0\0\0\24oraclese.localdomain\0\0\3241\0\0\3241\0\0\0\2\0\0\3241\0\0\3242\0\0\0\0\0\0\0\0", 92}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 92
poll([{fd=32, events=POLLIN}], 1, 500) = 1 ([{fd=32, revents=POLLIN}])
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0 \200\252\16\236\245\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\201\240\0\0\0\1\0\0\3241\0\0\3241\0\0\0\0\21X \0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\227\252.\312\216\16\364\222\0\0\0\0\4\207`\25P\370\n[\0\0\0\0P\370\nZ\0\0\0\0P\370\nZP\366\241\37\0\0
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 7996}], msg_con
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0\0\30\252\16\236\246\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\201\240\0\0\0\1\0\0\3241\0\0\3241\0\0\0\0\21X \0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\227\252.\312\216\16\364\222\0\0\0\0\4\207`\25P\370\n[\0\0\0\0P\370\nZ\0\0\0\0P\370\nZP\366\241\37\0\0
V3 READ Reply (Call In 44930) Len:8192 44938 2013-01-17 14:29:55.163827,192.168.1.20,192.168.1.60,NFS,1178
recvmsg(32, 0x7fffe701bc20, MSG_DONTWAIT|MSG_NOSIGNAL) = -1 EAGAIN (Resource temporarily unavailable)         
WAIT #140195143601328: nam='db file sequential read' ela= 1323 file#=4 block#=162 blocks=1 obj#=75338 tim=2013-01-17 14:29:55:164020

At first sight nothing has been changed – Oracle is doing NFS READ call and got a response. But if you will take a look on returned data – database block is full of zeros. It isn’t a surprise as NFS files are sparse one and it should contain only changed blocks. That read activity has been recorded in trace file as db file sequential read but we still have no data - so what is going next

pread(257, "#\242\0\0\242\0\0\1\250\34\17\0\0\0\1\4\317\215\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\10\0\0\0\234\n\0\0\0\0\0\0\10\0\0\0\10\0\0\0\250\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\10\0\0\0\250\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\240\0\0\1"..., 8192, 1327104) = 8192
WAIT #140195143601328: nam='Disk file operations I/O' ela= 49 FileOperation=9 fileno=4 filetype=2 obj#=75338 tim=2013-01-17 14:29:55:164170

Disk read is performed on file with handler 257 and as we can check in first strace output this file is a backup data file. Oracle is reading segment data from here and recorded that operation in trace file as ‘Disk file operations I/O’. Based on that behaviour I think there is no map of changed blocks but process is reading NFS first and if NFS will return zeroed block it is reading backup data file. Let see rest of full scan operation.

sendmsg(32, {msg_name(0)=NULL, msg_iov(2)=[{"\200\0\0\214\252\16\236\250\0\0\0\0\0\0\0\2\0\1\206\243\0\0\0\3\0\0\0\6\0\0\0\1\0\0\0000\253\315\357\253\0\0\0\24oraclese.localdomain\0\0\3241\0\0\3241\0\0\0\2\0\0\3241\0\0\3242\0\0\0\0\0\0\0\0\0\0\0$\1\0\7\1\1\240\202\3\0\0\0\0\222\364\16\216\312.\252\227\0\0\0\0\0\0\0\0\25`\ 
V3 READ Call (Reply In 44972), FH:0x8f3a5fe4 Offset:1335296 Len:32768  ; V3 READ Call (Reply In 44978), FH:0x8f3a5fe4 Offset:1368064 Len:8192
poll([{fd=32, events=POLLIN}], 1, 500) = 1 ([{fd=32, revents=POLLIN}])
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0\200\200\252\16\236\250\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\201\240\0\0\0\1\0\0\3241\0\0\3241\0\0\0\0\21X \0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\227\252.\312\216\16\364\222\0\0\0\0\4\207`\25P\370\n[\0\0\0\0P\370\nZ\0\0\0\0P\370\nZP\366\241\37\
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 32572}], msg_co
V3 READ Reply (Call In 44940) Len:32768 44972,2013-01-17 14:29:55.168776,192.168.1.20,192.168.1.60,NFS,1514    
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\200\0 \200\252\16\236\251\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\201\240\0\0\0\1\0\0\3241\0\0\3241\0\0\0\0\21X \0\0\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\227\252.\312\216\16\364\222\0\0\0\0\4\207`\25P\370\n[\0\0\0\0P\370\nZ\0\0\0\0P\370\nZP\366\241\37\0\0
recvmsg(32, {msg_name(0)=NULL, msg_iov(1)=[{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 7996}], msg_con
V3 READ Reply (Call In 44940) Len:8192,44978,2013-01-17 14:29:55.168790,192.168.1.20,192.168.1.60,NFS,746    
recvmsg(32, 0x7fffe7016aa0, MSG_DONTWAIT|MSG_NOSIGNAL) = -1 EAGAIN (Resource temporarily unavailable)           
WAIT #140195143601328: nam='db file scattered read' ela= 4762 file#=4 block#=163 blocks=5 obj#=75338 tim=2013-01-17 14:29:55:169040

Once again process send to NFS READ calls (even if segement header wasn’t on NFS) and again only zeroed blocks has been send back. Whole operation is recorded as db file scattered read but actually process still has no data.

pread(257, "\6\242\0\0\243\0\0\1\250\34\17\0\0\0\1\4\202\0\0\0\1\0\0\0J&\1\0\247\34\17\0\0\0\0\0\2\0002\0\240\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\377\377\16\0\230\37\212\37\212\37\0\0\0\0@\0\6\0\0\0`\37\0\0"..., 40960, 1335296) = 40 
WAIT #140195143601328: nam='Disk file operations I/O' ela= 6883 FileOperation=9 fileno=4 filetype=2 obj#=75338 tim=2013-01-17 14:29:55:176020      

Again process is reading file 257 (backup data file) and it is recording that activity as "Disk file operations I/O". Based on that observation I can say that Oracle is looking for data first on NFS and then in backup file. 

But it there any caching / learning algorithm in place ? I don’t think so cause when I run this query select * from dba_source where line like 'marcin' twice with flushing buffer cache between runs and execution pattern looks similar – there was NFS activity fallowed by reading backup disk file. 

This is end of part one and I will try to publish next post about same topic soon.

regards,
Marcin 

Oracle Linux and ESX hot add cpu and memory

I'm still quite new in Vmware ESX environment and features which I'm using now as my lab (curiosity of VMWare Guru Program).
This time I decided to test hot add CPU and memory functionality. It looks very interesting and in past years hot cpu / memory games where restricted to sophisticated hardware only - now everybody can test is at home.

My lab is basesd on Intel 4 cores and VM are running Oracle Linux 6.3 with or without Oracle Enterprise Kernel. Here are my findings:

1. Hot add CPU / memory is disabled by default and has to be enabled when VM is down. To do so you have to edit your VM configuration and go to options.

2. When feature is enabled you can ADD CPU or memory by editing your VM configuration. Keep in mind that there is no possibility to decrease number of CPU or memory.

3. Here is a table with supported options for ESX 5i and Oracle Linux with or without UEK

I tested adding a CPU to system for RedHat and UEK kernel. As you can find in table linked above adding memory is supported for RedHat kernels only.

RedHat kernel

OEL 6.3 with RedHat kernel running with 1 CPU

[root@oraclese ~]# uname -a
Linux oraclese.localdomain 2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23 12:05:59 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@oraclese ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dts
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

New CPU has been added in vSphere client. Below is output from messages log.

Jan 13 15:17:25 oraclese kernel: SMP alternatives: switching to SMP code
Jan 13 15:17:25 oraclese kernel: Booting Node 0 Processor 1 APIC 0x1
Jan 13 15:17:25 oraclese kernel: Disabled fast string operations
Jan 13 15:17:25 oraclese kernel: mce: CPU supports 0 MCE banks
Jan 13 15:17:25 oraclese kernel: x86 PAT enabled: cpu 1, old 0x0, new 0x7010600070106
Jan 13 15:17:25 oraclese kernel: Skipped synchronization checks as TSC is reliable.
Jan 13 15:17:25 oraclese kernel: microcode: CPU1 sig=0x206a7, pf=0x1, revision=0x14
Jan 13 15:17:25 oraclese kernel: platform microcode: firmware: requesting intel-ucode/06-2a-07
Jan 13 15:17:25 oraclese firmware.sh[2070]: Cannot find  firmware file 'intel-ucode/06-2a-07'
Jan 13 15:17:25 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:17:25 oraclese kdump: failed to stop
Jan 13 15:17:25 oraclese kdump: No crashkernel parameter specified for running kernel

Now system is running with two CPU's

[root@oraclese ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dts
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat epb xsaveopt pln pts dts
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Adding memory - OEL 6.3 wtih RedHat kernel - system running with 1 GB of RAM

[root@oraclese ~]# uname -a
Linux oraclese.localdomain 2.6.32-279.5.2.el6.x86_64 #1 SMP Thu Aug 23 12:05:59 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@oraclese ~]# free
             total       used       free     shared    buffers     cached
Mem:       1016680     190384     826296          0       4736      77112
-/+ buffers/cache:     108536     908144
Swap:      2097144          0    2097144

Additional 1 GB has been added via vSphere Client. Below is output from messages log.

Jan 13 15:40:54 oraclese kernel: init_memory_mapping: 0000000040000000-0000000048000000
Jan 13 15:40:54 oraclese kernel: Built 1 zonelists in Node order, mobility grouping on.  Total pages: 274190
Jan 13 15:40:54 oraclese kernel: Policy zone: Normal
Jan 13 15:40:54 oraclese kernel: init_memory_mapping: 0000000048000000-0000000050000000
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kernel: init_memory_mapping: 0000000050000000-0000000058000000
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kernel: init_memory_mapping: 0000000058000000-0000000080000000
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: failed to stop
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel
Jan 13 15:40:54 oraclese kdump: No crashkernel parameter specified for running kernel

Now Linux reconized 2 GB of RAM

[root@oraclese ~]# free
             total       used       free     shared    buffers     cached
Mem:       2065256     211840    1853416          0       4764      77328
-/+ buffers/cache:     129748    1935508
Swap:      2097144          0    2097144

Oracle UEK release 2 kernel

OEL 6.3 running with Oracle UEK release 2

[root@oraclese ~]# uname -a
Linux oraclese.localdomain 2.6.32-300.3.1.el6uek.x86_64 #1 SMP Fri Dec 9 18:57:35 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@oraclese ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

New CPU has been added in vSphere client. Below is output from messages log.

Jan 13 15:38:12 oraclese kernel: processor LNXCPU:01: registered as cooling_device1
Jan 13 15:38:12 oraclese kernel: SMP alternatives: switching to SMP code
Jan 13 15:38:12 oraclese kernel: Booting processor 1 APIC 0x1 ip 0x6000
Jan 13 15:38:12 oraclese kernel: Initializing CPU#1
Jan 13 15:38:12 oraclese kernel: CPU: Physical Processor ID: 2
Jan 13 15:38:12 oraclese kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jan 13 15:38:12 oraclese kernel: CPU: L2 cache: 256K
Jan 13 15:38:12 oraclese kernel: CPU: L3 cache: 6144K
Jan 13 15:38:12 oraclese kernel: CPU 1/0x1 -> Node 0
Jan 13 15:38:12 oraclese kernel: mce: CPU supports 0 MCE banks
Jan 13 15:38:12 oraclese kernel: x86 PAT enabled: cpu 1, old 0x0, new 0x7010600070106
Jan 13 15:38:12 oraclese kernel: CPU1: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz stepping 07
Jan 13 15:38:12 oraclese kernel: Skipping synchronization checks as TSC is reliable.
Jan 13 15:38:12 oraclese kernel: microcode: CPU1 sig=0x206a7, pf=0x1, revision=0x14
Jan 13 15:38:12 oraclese kernel: platform microcode: firmware: requesting intel-ucode/06-2a-07
Jan 13 15:38:12 oraclese firmware.sh[2074]: Cannot find  firmware file 'intel-ucode/06-2a-07'
Jan 13 15:38:12 oraclese kdump: kexec: failed to unload kdump kernel
Jan 13 15:38:12 oraclese kdump: failed to stop
Jan 13 15:38:12 oraclese kdump: No crashkernel parameter specified for running kernel

Now system is running with two CPU's

[root@oraclese ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
stepping        : 7
cpu MHz         : 3110.469
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm ida arat
bogomips        : 6220.93
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

regards, Marcin

Oaktable videos

Hi,

Short update - I have upload new video from OakTable World 2012 - Brendan Gregg -- USE Method for Performance Tuning.

All videos and presentations can be found here - OakTable world

regards,
Marcin

How to rename NIC in OEL 6.X

This is a quick note about host cloning and network interface (ethX) renaming.
As I wrote in my last post I have ESX lab now and I create a template of OEL 6.3 with all my settings and I would like use it for adding new VM (including RAC nodes). It is easy but there is one small issue – every time you clone VM network devices are renamed. Well MAC address is unique so it doesn’t surprise me but I would like to keep interface names like eth0, eth1 and eth2 and I got eth0, eth3, eth4 instead. Since OEL 5 (Redhat 5) all device name are generated by udev mechanism so I start my research there.
Well it was easier than I thought – there is a file in "/udev/rules.d" directory called "70-persistent-net.rules".
And here is file content:

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:ce:65:67", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:95", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:94", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:98", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:99", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

Well this is it – all I need to do was to run "ifconfig –a" and confirm MAC addresses of existing interfaces. I have removed old entries and change interface names. My final configuration file looks like that

[root@localhost rules.d]# cat 70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:ce:65:67", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:95", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:22:94", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

After changes I rerun udev mechanism

[root@localhost rules.d]# start_udev
Starting udev:  

And finally I have got correct configuration. Next lesson learned.
Regards,
Marcin

New lab setup

After couple of month playing with Oracle VM I decided to install and configure VMWare. Thanks to VMware Guru Program I’m able to test ESX 5i with one year license. This is excellent opportunity for me to get more familiar with newest VMware solutions. I have been using VMware tools for years and I remember times when I deployed my first Oracle RAC 9i on Workstation 3.x and had to "hack" configuration file to change disk.locking parameter to false.

When I start instalation I have run into couple of issues as I decided to leave my current disk setup and I had only one free disk. New ESX is using GPT partition table and I couldn’t use it together with Grub to start Linux or VMware. This is when USB sticks went to play and I have installed ESX 5i on it and force my PC to boot from USB only. With other USB with Grub I have boot selection on my headless PC based on which USB key is connected at boot time. First problem solved.
Next issue appears to when I created my first VM and want to clone it – there was no such option in Windows vSphere Client. I read a manual (frankly I just googled) and I found out that I need to install vSphere 5 vCenter Server to be able to clone my VM automatically. Well I don’t have other host for it but I found out that I can download a VM image and deploy it on same ESX host. Here is a link to very nice instruction - vSphere vCenter virtual appliance quick start guide and you can find documentation here - vSphere documentation.
After I downloaded and installed vCenter and used vSphere Client to connect into I was able to use all features I was looking for. Actually it is strange for me that I need to have additional management server to clone VM. But I can’t complain Oracle VM need management server even to start VM so I would say both are quite similar in terms of management overhead for home usage. I can give VMWare stack one point more as you can stop vCenter and keep it down if you don’t need all features implemented in it.

I'm using now my new lab as Oracle server in various configurations and also as development box and additional workstation. I spend some time trying to setup remote desktop from my laptop to workstation with sound but without luck. I have tried FreeNX and NoMachine solution but setting audio transport was too much for me. After couple of days thanks to Twitter discussion with @simon_haslam peoples from ThinLinc joined it I have been told that we can use their remote desktop solution for free up to 10 connection. And that is what I was looking for  - remote desktop with video and audio support. Good work ThinLinc.

Now time for my TODO list with my little virtual server:
- install free version of Cisco Nexus V1000 switch and play with RAC NIC failovers
- Learn Hadoop and Oracle DB Hadoop integration
- Install Cloudera software (including newest Impala)
- extend day to 48h to have time to do all above ;)

regards,
Marcin

DataGuard and Oracle Restart - how to make it work

If you are going to implement Oracle Data Guard together with Oracle Restart you should be aware that there is a configuration problem in version 11.2.0.3 (and probably in other 11.2. as well). Oracle Restart is not checking what is a current database role and it is going to start standby instance in OPEN mode. This can end up with license issue if your Data Guard Broker will start applying process on standby database. If you don't have license to use Active Data Guard you just broken your license agreement.

There is couple of possible solutions:

• manually set up "mount" state as start up mode for standby database and change it after every switch- or fail-over
• disable MRP functionality on standby database and keep it open in read only mode 
• add your own script to open primary database only and keep standby in "mount" state

I want to describe last solution and share script which I have created for it.

1. First of all you have to change your Oracle Restart configuration to open both databases (standby and primary) in "mount" state using following command.  

   [oracle@testdb1 ~] srvctl modify database -d database_name -s MOUNT
   

2. In next step user script has to be created - you can use this one as an template. I have based it on CRS demo script.Script has been save in /tmp directory on both server using name "opendb"
   

   #!/bin/sh
   # These messages goes into the CRSD agent log file.
   echo " *******   `date` ********** "
   echo "Action script '$_CRS_ACTION_SCRIPT' for resource[$_CRS_NAME] called for action $1"
   #env
   #
   
   #setup database home
   export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db3
   #setup database SID
   export ORACLE_SID=testa
   DBROLE=''
   
   
   # check database role and return following values
   # OPENED if DB is open and it is primary
   # PRIMARY if DB is mounted and DB role is primary
   # PHYSICAL is DB is mounted and DB role is standby
   
   function getrole() {
   
   $ORACLE_HOME/bin/sqlplus -s / as sysdba << EOF
        spool /tmp/getrole.tmp
        set echo off feedback off head off
        select case when OPEN_MODE = 'READ WRITE' then 'OPENED' when OPEN_MODE='MOUNTED' and DATABASE_ROLE='PRIMARY' then 'PRIMARY' when OPEN_MODE='MOUNTED' and DATABASE_ROLE='PHYSICAL STANDBY' then 'PHYSICAL' end from v\$database;
        exit
   EOF
   DBROLE=`cat /tmp/getrole.tmp | sed 's/[ \t]*$//' | sed 's/^[ \t]*//' | tail -1`
   
   }
   
   
   case "$1" in
     'start')
        echo "START entry point has been called.."
        getrole
        # check role and do following actions
        case $DBROLE in
          'PHYSICAL')   echo "This is standby - do nothing" ;;
          'PRIMARY')
        $ORACLE_HOME/bin/sqlplus -s / as sysdba << EOF
        alter database open;
        exit
   EOF
        echo ;;
        esac
        exit 0
        ;;
   
     'stop')
        echo "STOP entry point has been called.."
        exit 0
        ;;
   
     'check')
       echo "CHECK entry point has been called.."
       getrole
       if [ "$DBROLE" == 'OPENED' ] || [ "$DBROLE" == 'PHYSICAL' ]; then
           echo "Check -- SUCCESS"
           exit 0
       else
           echo "Check -- FAILED"
           exit 1
       fi
       ;;
   
     'clean')
        echo "CLEAN entry point has been called.."
        exit 0
        ;;
   
   esac
   

3. Register user script in Oracle Restart and set up dependencies using following command. Ora.testa.db is my database resource name in CRS - please change to your database resource name. New resource is called "ora.opendb" and it has to be registered on both servers and started.There is hard dependency between my resource and database resource so my script will be started only when database has been started as well.
   

   [oracle@testdb1 ~]$ /u01/app/oracle/product/11.2.0/grid3/bin/crsctl add resource ora.opendb -type cluster_resource \
   > -attr "ACTION_SCRIPT=/tmp/opendb,CHECK_INTERVAL=30,RESTART_ATTEMPTS=2 \
   > ,START_DEPENDENCIES=hard(intermediate:ora.testa.db)"
   
   [oracle@testdb1 ~]$ /u01/app/oracle/product/11.2.0/grid3/bin/crsctl start resource ora.opendb
   

4. Check status of resources.

   [oracle@testdb1 ~]$ /u01/app/oracle/product/11.2.0/grid3/bin/crsctl status res -t
   --------------------------------------------------------------------------------
   NAME           TARGET  STATE        SERVER                   STATE_DETAILS
   --------------------------------------------------------------------------------
   Local Resources
   --------------------------------------------------------------------------------
   ora.LISTENER.lsnr
                  ONLINE  ONLINE       testdb1
   ora.ons
                  OFFLINE OFFLINE      testdb1
   --------------------------------------------------------------------------------
   Cluster Resources
   --------------------------------------------------------------------------------
   ora.cssd
         1        OFFLINE OFFLINE
   ora.diskmon
         1        OFFLINE OFFLINE
   ora.evmd
         1        ONLINE  ONLINE       testdb1
   ora.opendb
         1        ONLINE  ONLINE       testdb1
   ora.testa.db
         1        ONLINE  ONLINE       testdb1                  Open
   
   

When everything is deployed is it running in following way: Oracle Restart is starting both data bases in mount state and then next resource (user script) is started. Script is checking current database role and it is opening primary database and doesn't do any action on standby database. 

Disclamer:
Please check script in your development and test environment before you deploy it and change anything in production environment. This script has been created for my personal tests and there is no guarantee that it is bug free.

regards,
Marcin

 

 

Oracle Grid Cloud Control 12c BP1 on Oracle Enterprise Linux 6.1

I have installed it in my test box and I hit a problem with auto-discovery due to missing libssl.so library for nmap. Issue is very easy to fix using common Linux troubleshooting approach. First what versions I'm using:

[root@oem-server ~]# cat /etc/oracle-release
Oracle Linux Server release 6.1
[root@oem-server ~]# openssl version
OpenSSL 1.0.0-fips 29 Mar 2010
[root@oem-server ~]# yum list | grep -e "^openssl"
openssl.x86_64                           1.0.0-20.el6_2.5            @ol6_latest
openssl.i686                             1.0.0-20.el6_2.5            ol6_latest
openssl-devel.i686                       1.0.0-20.el6_2.5            ol6_latest
openssl-devel.x86_64                     1.0.0-20.el6_2.5            ol6_latest
openssl-perl.x86_64                      1.0.0-20.el6_2.5            ol6_latest
openssl-static.x86_64                    1.0.0-20.el6_2.5            ol6_latest
openssl098e.i686                         0.9.8e-17.0.1.el6_2.2       ol6_latest
openssl098e.x86_64                       0.9.8e-17.0.1.el6_2.2       ol6_latest

Now what kind of issue I had

[root@oem-server ~]# /u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap
/u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap: error while loading shared libraries: libssl.so.4: cannot open shared object file: No such file or directory

How I fixed it

[oracle@oem-server ~]$  ldd /u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap
        linux-vdso.so.1 =>  (0x00007fff447ff000)
        libssl.so.4 => not found
        libcrypto.so.4 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003d40800000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003d43c00000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003d41000000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003d43400000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003d40400000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003d40000000)

[root@oem-server ~]# ln -s /usr/lib64/libssl.so.10 /usr/lib64/libssl.so.4
[root@oem-server ~]# ln -s /usr/lib64/libcrypto.so.10 /usr/lib64/libcrypto.so.4

[oracle@oem-server ~]$  ldd /u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap
        linux-vdso.so.1 =>  (0x00007fffb0686000)
        libssl.so.4 => /usr/lib64/libssl.so.4 (0x00007fd6b1c0a000)
        libcrypto.so.4 => /usr/lib64/libcrypto.so.4 (0x00007fd6b1870000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003d40800000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003d43c00000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003d41000000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003d43400000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003d40400000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x0000003d48c00000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x0000003d46c00000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003d44000000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x0000003d47800000)
        libz.so.1 => /lib64/libz.so.1 (0x0000003d41400000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003d40000000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x0000003d46800000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003d47000000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003d42400000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003d40c00000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003d41c00000)

Now it is working

[oracle@oem-server ~]$ /u01/Middleware/agent/agent_inst/discovery/nmap/bin/nmap -v

Starting Nmap 5.51.3 ( http://nmap.org ) at 2012-07-08 19:19 IST
Unable to find nmap-services!  Resorting to /etc/services
Read data files from: /etc
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 0.03 seconds
[oracle@oem-server ~]$

After that I checked MOS and Grid Control 12c is certified with OEL 6 so this errors was not expected but then I realized that I didn't read know issues for Oracle® Enterprise Manager Cloud Control Support Notes for Linux x86 and x86-64 and I found it in point called "Host Discovery Job Displays Error While Loading Shared Libraries". Official solution is a little bit more complicated and required downloading openssl sources and compiling it. Anyway I found article above using Google and I couldn't find it on entry documentation page.

regards,
Marcin