Today I found a thread on Oracle forum about a slow connection and I recall that I solved it in past.
Now I have found a few notes on Metalink (ex. 803838.1) which confirm my investigation that Oracle 11g is using DNS to resolve names instead of /etc/hosts file on Unix box. If your server is not registered in DNS or if there is no DNS server which was specified in /etc/resolv.conf you can have a connection problem.
A fastest way to solve it on Linux is to remove "nameserver" entry from /etc/resolv.conf file if you don't need a DNS on the server otherwise you have to be sure that your Oracle server is registered in DNS.
On Solaris a solution is a little bit different - change line in /etc/nsswitch.conf into
ipnodes: files [NOTFOUND=continue] dns
and do not delete line starting with hosts.