While setting up Zabbix and JBoss JMX monitoring in my testing environment I encountered the following messages in the zabbix-java-gateway.log file when trying to get values from a remote server using JMX.
The zabbix-java-gateway would try to connect to the JMX server using 127.0.0.1 instead of the IP number of the JBoss server:
2014-01-07 17:33:05.129 [pool-1-thread-1] WARN com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused
at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:100) ~[zabbix-java-gateway-2.2.1.jar:na]
at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:63) ~[zabbix-java-gateway-2.2.1.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_28]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_28]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_28]
My environment consists of 2 servers, a zabbix and jboss server. Zabbix is supposed to monitor the jboss machine using JMX remotely, for which it is using the zabbix-java-gateway component on the zabbix server.
The error on the zabbix server says it can’t connect to the RMI server on 127.0.0.1. Which is quite correct, there isn’t one running on the zabbix server. But why does it want to connect to 127.0.0.1 instead of the JBoss server’s IP?
Doing a search on this problem on Google gives a lot of posts stating it is because of a misconfiguration on the Linux JBoss server. Resolving the hostname would return 127.0.0.1 instead of the external IP number. But this was not the case in my situation:
[root@jboss ~]# hostname
[root@jboss ~]# ping $(hostname)
PING jboss.kanbier.lan (10.37.129.4) 56(84) bytes of data.
64 bytes from jboss.kanbier.lan (10.37.129.4): icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from jboss.kanbier.lan (10.37.129.4): icmp_seq=2 ttl=64 time=0.031 ms
Resolving looking good, it doesn’t look like this is the issue. However, I do agree on the Zabbix Java Gateway getting the value of 127.0.0.1 for the RMI server from jboss.kanbier.lan. Taking a look at the JBoss starting output:
Using RMI server codebase: http://127.0.0.1:8083/
I would like the value of 10.37.129.4 there instead of 127.0.0.1, but being a testing environment I never spend a lot of time properly configuring JBoss to do so. It turns out starting JBoss with the proper argument will let it know it’s IP number should be 10.37.129.4 instead of 127.0.0.1:
[root@jboss ~]# ./run.sh -b 10.37.129.4
Let’s take a look at the output now:
Using RMI server codebase: http://10.37.129.4:8083/
Looking good from this end now. Zabbix is starting to receive JMX data from the JBoss server as well:
DEBUG com.zabbix.gateway.JMXItemChecker - connecting to JMX agent at service:jmx:rmi:///jndi/rmi://10.37.129.4:9999/jmxrmi
DEBUG com.zabbix.gateway.ItemChecker - getting value for item 'jmx["java.lang:type=MemoryPool,name=Tenured Gen",Usage.used]'
Long story short:
Receiving the message “java.rmi.ConnectException: Connection refused to host: 127.0.0.1” means the RMI on the server end says it can be reached on IP number 127.0.0.1, even to remote hosts, which of course will not work.