Promoting Oracle DB in Enterprise Manager Fails with IO Error

A friend recently asked me about promoting their Oracle Database in Enterprise Manager and getting a strange IO Error. The full error was “IO Error: Got minus one from a read call, connect lapse 1ms, Authentication lapse 0ms”.

I’d never seen this before so we started digging around. He showed me he could connect with the same connect string on the DB host, but not from his other host. In the listener log, we found we were getting a TNS-00516: Permission denied error when testing the connection.

05-FEB-2020 11:48:40 * 12546
TNS-12546: TNS:permission denied
 TNS-12560: TNS:protocol adapter error
  TNS-00516: Permission denied

Turns out, as some point he had enabled valid node checking in the sqlnet.ora but only added the database server to the configuration file.

TCP.VALIDNODE_CHECKING = YES
TCP.EXCLUDED_NODES= (<IP ADDRESS>)
TCP.INVITED_NODES=(<IP ADDRESS, <HOSTNAME>)

He was assuming all communication goes through the Enterprise Manager Agent which resides on this host, so it should have gone through. However, not ALL communication goes through the agent. Many activities establish a direct connection from the OMS server/s to the database listener. This includes target discovery/promotion, performance hub and administration tasks. The metric collection and job execution is done by the agent directly.

We added the OMS hostname to the tcp.invited_nodes list and testing completed successfully and database was promoted and successfully monitored in Enterprise Manager. Keep in mind the OMS requires communication to all monitored hosts on their target specific port (in this case whatever the listener is running on).