Fixing: java.rmi.ConnectException: Connection refused to host: 127.0.0.1

While setting up Zabbix and JBoss JMX monitoring in my testing environment I encountered the following messages in the zabbix-java-gateway.log file when trying to get values from a remote server using JMX.

The zabbix-java-gateway would try to connect to the JMX server using 127.0.0.1 instead of the IP number of the JBoss server:

2014-01-07 17:33:05.129 [pool-1-thread-1] WARN com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused
at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:100) ~[zabbix-java-gateway-2.2.1.jar:na]
at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:63) ~[zabbix-java-gateway-2.2.1.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_28]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_28]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_28]

My environment consists of 2 servers, a zabbix and jboss server. Zabbix is supposed to monitor the jboss machine using JMX remotely, for which it is using the zabbix-java-gateway component on the zabbix server.

The error on the zabbix server says it can’t connect to the RMI server on 127.0.0.1. Which is quite correct, there isn’t one running on the zabbix server. But why does it want to connect to 127.0.0.1 instead of the JBoss server’s IP?

Doing a search on this problem on Google gives a lot of posts stating it is because of a misconfiguration on the Linux JBoss server. Resolving the hostname would return 127.0.0.1 instead of the external IP number. But this was not the case in my situation:

[root@jboss ~]# hostname
jboss.kanbier.lan
[root@jboss ~]# ping $(hostname)
PING jboss.kanbier.lan (10.37.129.4) 56(84) bytes of data.
64 bytes from jboss.kanbier.lan (10.37.129.4): icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from jboss.kanbier.lan (10.37.129.4): icmp_seq=2 ttl=64 time=0.031 ms

Resolving looking good, it doesn’t look like this is the issue. However, I do agree on the Zabbix Java Gateway getting the value of 127.0.0.1 for the RMI server from jboss.kanbier.lan. Taking a look at the JBoss starting output:

Using RMI server codebase: http://127.0.0.1:8083/

I would like the value of 10.37.129.4 there instead of 127.0.0.1, but being a testing environment I never spend a lot of time properly configuring JBoss to do so. It turns out starting JBoss with the proper argument will let it know it’s IP number should be 10.37.129.4 instead of 127.0.0.1:

[root@jboss ~]# ./run.sh -b 10.37.129.4

Let’s take a look at the output now:

Using RMI server codebase: http://10.37.129.4:8083/

Looking good from this end now. Zabbix is starting to receive JMX data from the JBoss server as well:

DEBUG com.zabbix.gateway.JMXItemChecker - connecting to JMX agent at service:jmx:rmi:///jndi/rmi://10.37.129.4:9999/jmxrmi
DEBUG com.zabbix.gateway.ItemChecker - getting value for item 'jmx["java.lang:type=MemoryPool,name=Tenured Gen",Usage.used]'

Long story short:

Receiving the message “java.rmi.ConnectException: Connection refused to host: 127.0.0.1” means the RMI on the server end says it can be reached on IP number 127.0.0.1, even to remote hosts, which of course will not work.

2 thoughts on “Fixing: java.rmi.ConnectException: Connection refused to host: 127.0.0.1

  1. i also have this type of error when trying to monitor tomcat using zabbix.

    2014-03-31 15:05:35.654 [main] INFO com.zabbix.gateway.JavaGateway – Zabbix Java Gateway 2.2.1 (revision 40808) has started
    2014-03-31 15:05:35.671 [main] INFO com.zabbix.gateway.JavaGateway – listening on 0.0.0.0/0.0.0.0:10052
    2014-03-31 15:06:01.889 [pool-1-thread-1] WARN com.zabbix.gateway.SocketProcessor – error processing request
    com.zabbix.gateway.ZabbixException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.0.162; nested exception is:
    java.net.ConnectException: Connection refused]
    at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:100) ~[zabbix-java-gateway-2.2.1.jar:na]
    at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:63) ~[zabbix-java-gateway-2.2.1.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
    Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.0.162; nested exception is:
    java.net.ConnectException: Connection refused]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.7.0_45]
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268) ~[na:1.7.0_45]
    at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:92) ~[zabbix-java-gateway-2.2.1.jar:na]
    … 4 common frames omitted

    • A few questions:

      1. Is 192.168.0.162 the IP number of the host you want to monitor?
      2. What port number did you configure for the JMX interface?
      3. Can you telnet to that port from the host running the Zabbix Java Gateway?

      Things that come to mind:

      Did you allow access through a firewall if present?
      Did you enable remote monitoring in tomcat?

      Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *