JBoss EAP 6 monitoring using remoting-jmx and Zabbix

Setting up Zabbix monitoring for JBoss EAP 6 turns out te be quite different as for JBoss EAP 5. This is because the method of fetching JMX data in EAP 5 is no longer available in EAP 6.

In this post I’ll describe a method on how to enable JMX monitoring for JBoss EAP 5 and JBoss EAP 6 within the same Zabbix installation.

Update 22-01-2014: You can download a custom zabbix-java-gateway RPM file for Zabbix 2.2.1 on RHEL/CentOS6 64-bit from my site here. The process of creating the RPM can be found here.

Please note: You still need to add the remoting-libraries from your EAP installation to /usr/sbin/zabbix_java/lib if you use my RPM.

Problem:

JBoss EAP 5 supports JMX monitoring using RMI, where JBoss EAP 6 does not. EAP 6 uses “remoting-jmx” instead of “rmi”. This is a problem because the Zabbix Java Gateway only supports RMI for the JMX interface, preventing us from monitoring JBoss EAP 6 using JMX monitoring.

Goal:

My goal in this blogpost is to enable JMX monitoring for JBoss EAP 5 and JBoss EAP 6 on the same Zabbix Server.

Steps:

  1. Modify source code of the Zabbix Java Gateway to enable both RMI and remoting-jmx
  2. Add the required libraries to the Zabbix Java Gateway
  3. Enable remote management in JBoss EAP 6
  4. Configure a JBoss EAP 5 and JBoss EAP 6 host to test the new configuration
  5. Using JConsole to identify items you want to monitor in Zabbix

Let’s get started.

1. Modify source code of the Zabbix Java Gateway to enable both RMI and remoting-jmx

Like stated before, the Zabbix Java Gateway component only supports JMX over RMI. This is because the service URL it is using to connect to the remote server is hardcoded in JMXItemChecker.java:

url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");

At the time of writing the most current version of Zabbix is 2.2.1, and there is no way to customise this JMX url. JBoss EAP 6 doesn’t support RMI and expects remoting-jmx.

Replacing the url with this one will enable remoting-jmx, at the cost of losing RMI monitoring:

url = new JMXServiceURL("service:jmx:remoting-jmx://" + conn + ":" + port);

In my environment I want to monitor both JBoss EAP 5 and EAP 6, and I don’t want to add a second zabbix server to do so. So while searching for a way to do this with minimal changes to the zabbix source code I had the idea of using a different port number for different JBoss versions.

It’s far from ideal and just a workaround until the Zabbix people come up with a real solution, but it requires only a tiny bit of modifying the source code.

Here are my notes of modifying the source code and building new binaries on a clean CentOS 6.5 machine.

Preparing the build server:

 # yum -y update
 # reboot
 # yum groupinstall "Development Tools"
 # yum -y install mysql-server httpd php mysql-devel php-mysql libxml2-devel libcurl-devel php-mbstring php-bcmath php-gd php-xml

Getting the software:

 # cd /opt
 # wget http://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/2.2.1/zabbix-2.2.1.tar.gz/download
 # tar -xvzf zabbix-2.2.1.tar.gz

Modify the JMXItemChecker.java:

# vi zabbix-2.2.1/src/zabbix_java/src/com/zabbix/gateway/JMXItemChecker.java

Change:To:
This enables remoting-jmx for hosts with a JMX Interface on port 7777 only, while keeping RMI for any other port. Of course you can change this port to be whatever you need, just be sure to match the port you configure in JBoss.

That is all there is to it, let’s compile  the binaries and install our modified version of Zabbix:

# cd zabbix-2.2.1
# ./configure --enable-server --enable-agent --with-mysql --with-libcurl --with-libxml2 --enable-java
# make install

Configure the Java Pollers:

# vi /usr/local/etc/zabbix_server.conf

Add these values and save:

JavaGateway=10.37.129.2
StartJavaPollers=5
JavaGatewayPort=10052

3. Add the required libraries to the Zabbix Java Gateway

Before we can start the java gateway we need to tell it how to use remoting-jmx. We can use the jar files from our JBoss EAP 6 installation to do this.

Find the following jar files in your <EAP_6_HOME>/modules/system/layers/base/<sub_dirs>:

$ cat needed_modules.txt
jboss-as-remoting
jboss-logging
jboss-logmanager
jboss-marshalling
jboss-marshalling-river
jboss-remoting
jboss-sasl
jcl-over-slf4j
jul-to-slf4j-stub
log4j-jboss-logmanager
remoting-jmx
slf4j-api
xnio-api
xnio-nio

Place these in the java gateway lib directory:

$ for i in $(cat needed_modules.txt); do find /opt/jboss-eap-6.0 -iname ${i}*.jar -exec cp {} /usr/local/sbin/zabbix_java/lib/ \; ; done

If you’re reading this post I assume setting up the Zabbix database and web frontend is trivial so I won’t cover it here. Let’s start the java gateway and zabbix server and add a JBoss EAP 5 and JBoss EAP 6 host to test our modifications.

3. Enable remote management in JBoss EAP 6

I’m using a .zip deployment of JBoss EAP 6, unzipped in /opt/jboss-eap-6.0 and running on a host with an external IP number of 10.37.129.5.

In this version of JBoss a user is required to do remote monitoring. Fortunately JBoss comes with the add-user.sh script to do this for you:

[root@jboss6 bin]# cd /opt/jboss-eap-6.0/bin/
[root@jboss6 bin]# ./add-user.sh

What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a): 

Enter the details of the new user to add.
Realm (ManagementRealm) :
Username : zabbix
Password : zabbix_1
Re-enter Password : zabbix_1
About to add user 'zabbix' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'zabbix' to file '/opt/jboss-eap-6.0/standalone/configuration/mgmt-users.properties'
Added user 'zabbix' to file '/opt/jboss-eap-6.0/domain/configuration/mgmt-users.properties'
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? no

We can now use the user “zabbix” with a password of “zabbix_1” for remote management. We also need to configure the management interface of JBoss to listen on port 7777.

I’m going to run my JBoss instance in standalone mode with the standalone-full profile. To configure the management port, open:

/opt/jboss-eap-6.0/standalone/configuration/standalone-full.xml

Search for this line:

<socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>

Change the port number to 7777:

<socket-binding name="management-native" interface="management" port="${jboss.management.native.port:7777}"/>

Save the file and start the JBoss instance:

/opt/jboss-eap-6.0/bin/standalone.sh --server-config=standalone-full.xml -b 10.37.129.5 -Djboss.bind.address.management=10.37.129.5

 

4. Configure a JBoss EAP 5 and JBoss EAP 6 host to test the new configuration

In the Zabbix web interface go to: Configuration -> Hosts -> Create host.

The first host I will add is a JBoss EAP 5 host, so it will require JMX RMI monitoring:

Hostname: jboss.kanbier.lan
Groups: Linux servers
JMX Interfaces: jboss.kanbier.lan DNS 9999

Screen Shot 2014-01-18 at 00.14.15
Click “Save”.

Let’s add a JMX item for testing to this host. Click “Items” in the jboss.kanbier.lan row and click “Create Item”.

Name: State of DefaultDS
Type: JMX agent
Key: jmx["jboss.jca:service=ManagedConnectionPool,name=DefaultDS",StateString]
Type of information: Text
New application: JBoss EAP 5

Screen Shot 2014-01-18 at 00.19.08

Click “Save”.

Now for the EAP 6 host that will be using remoting-jmx because we’ll configure the JMX interface to use port 7777.

Create a host:

Host name: jboss6.kanbier.lan
Groups: Linux servers
JMX Interfaces: jboss6.kanbier.lan DNS port 7777

Screen Shot 2014-01-18 at 00.24.14

Click “Save”.

Let’s create a JMX item for this EAP 6 host for testing:

Name: JNDI name for ExampleDS
Type: JMX agent
Key: jmx["jboss.as:subsystem=datasources,data-source=ExampleDS","jndiName"]
Host interface: <your_to_monitor_host>:7777
User name: zabbix
Password: zabbix_1
Type of information: Text
New application: JBoss EAP 6 JNDI String

Screenshot 2014-02-17 13.50.11

Click “Save”.

Now under Monitoring -> Latest data you can see the values being pulled in by Zabbix:

Screen Shot 2014-01-18 at 00.30.34

5. Using JConsole to identify items you want to monitor in Zabbix

There are at least 2 ways to get the JMX keys fron JBoss so you can add them to Zabbix. In my example for JBoss EAP 6 I use the key:

"jboss.as:subsystem=datasources,data-source=ExampleDS","jndiName"

This fetches the “jndiName” from the ExampleDS data-source. How to construct this key? You can easily browse the items that are available for monitoring using either JConsole or jboss-cli.sh.  In this example I’ll be using JConsole.

JConsole is a Java GUI application that connects to the management interface of JBoss in very much the same way as the Zabbix Java Gateway. It is shipped with JBoss and located in <JBOSS_HOME>/bin.

Before you start it, make sure your JBoss EAP 6 instance is running using the configuration described in the instructions above.

These are the steps I took to get it started on jboss6.kanbier.lan with external ip 10.37.129.5.

$ export PATH=/usr/java/latest/bin:$PATH
$ export JAVA_HOME=/usr/java/latest
$ /opt/jboss-eap-6.0/bin/jconsole.sh

This should start a GUI if you have an X session available. Fill in:

Check: Remote Process
service:jmx:remoting-jmx://10.37.129.5:7777
Username: zabbix
Password: zabbix_1

Screen Shot 2014-02-18 at 09.01.58

Click connect, and you should connect to the JBoss management interface. You now have a GUI interface to access metrics on memory, threads and pretty much anything you would like to know about your JVM:

Screen Shot 2014-02-18 at 09.07.20

But there is one downside, the data available is only realtime. So no history unless you keep JConsole running forever. This is where Zabbix comes in. You can pull the data from the management interface just like JConsole does and store it in Zabbix so you can build up a history, analyse trends and all things Zabbix does best.

Let’s say you want to monitor the HeapMemoryUsage in Zabbix. This requires some knowledge of JBoss, but here is how to find it in JConsole.

Click on the MBeans tab -> java.lang -> Memory -> Attributes -> HeapMemoryUsage:

Screen Shot 2014-02-18 at 10.59.30

This attribute contains a CompositeDataSupport, from which we can extract the data we want. In the Descriptor field the OpenType tells us what we can extract from the CompositeData:

javax.management.openmbean.CompositeType(name=java.lang.management.MemoryUsage,
items=((itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),
(itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long))))

Now to convert this to Zabbix item keys, just follow the path you used in JConsole:

jmx["java.lang:type=Memory","HeapMemoryUsage.used"]

 

Create an item in Zabbix for our jboss6 host using this key:

Name: Used heap memory
Type: JMX agent
Key: jmx["java.lang:type=Memory","HeapMemoryUsage.used"]
User name: zabbix
Password: zabbix_1
Type of information: Numeric
Data type: Decimal
Units: Bytes

Screen Shot 2014-02-18 at 11.16.36

Save the item. Zabbix should now start monitoring the heap memory size:

Screen Shot 2014-02-18 at 11.21.50

Happy monitoring!

29 thoughts on “JBoss EAP 6 monitoring using remoting-jmx and Zabbix

          • Dennis,
            I have a question, you say to change the management port from 9999 to 7777,

            I have Jconsole working with the default port for Remoting 4447, why the management port and not the Remoting port?

            I have made the coding changes and rebuilt the gateway, and forced updated that rpm. But not having much luck getting it to work so far.

          • I must be missing one more thing….. the call is failing zabbix is reporting
            CANNOT OPEN RECEIVED JSON.

            Any clue?

          • Hey Vito, I’m sorry I’m a bit busy as of late. Can you get me the complete logs? And maybe run the java-gateway in debug mode, to see what actually gets returned.

  1. I’m testing this procedure with JBoss 7 and the following error occurred:
      1972:20140206:105419.790 Item [Russian: jmx [“java.lang: type = Runtime” VMName]] error: “javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed”

    already added the user and configured the item master in Zabbix.

    Any suggestions?

    • Can you successfully use jconsole using the remoting-jmx protocol? As explained here?

      If so, in what mode is JBoss running? I haven’t tested this in domain mode yet.

      Also, is it possible your JBoss server requires secure connections using SSL?

      One last question, do you see any authentication denials on the JBoss server side?

    • Then your problem is with JBoss and management interface, not so much with Zabbix. Let’s see if I can help you further.

      Can you post the exact command you use to start the JBoss server?

      Can you post the exact service-url you use within jconsole?

      Do you see the management interface being started on the expected port?

      • 1- Ok

        2- standalone.sh -b 0.0.0.0
        I’m just testing for monitoring with Zabbix. I’m not using in production environment.

        3- service:jmx:remoting-jmx://127.0.0.1:9999
        I tested on port 4447, but can not connect

        4- yes. I see. Look:
        08:59:36,937 INFO [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on /0.0.0.0:4447
        08:59:36,937 INFO [org.jboss.as.remoting] (MSC service thread 1-8) JBAS017100: Listening on /0.0.0.0:9999

        • I don’t think you need to enable SSL to test this, in my testing environment I haven’t enabled SSL at all. In production environments I would recommend to actually use SSL, but I haven’t got time to test this fully yet.

          I think you might have to load the standalone-full.xml file instead of the standard standalone.

          I start my server like this:

          ./standalone.sh -b 192.168.1.2 –server-config=standalone-full.xml -Djboss.bind.address.management=192.168.1.2

          • Access with jconsole now.
            Before I was using OpenJDK. I tried with the Oracle JDK and it worked. Keeps giving error when Zabbix will collect, but I will prepare a VM with Zabbix using the Oracle JDK to view solves the problem.

        • Hi Janssen, did you get it working?

          I went over your comments again, are you sure you configured the JMX interface in Zabbix for the server you want to monitor correctly? The jmx interface port should match the one jboss uses.

          If you followed my post Zabbix will connect to your host on port 7777, so it’s important the management interface in jboss also uses 7777 and not 9999.

          You can change this in the standalone-full.xml, and be sure you start jboss with –server-config=standalone-full.xml

          I’ll update the post with this information soon, after reviewing it this is not explained at all.

  2. I can connect to jboss thru jmx-remoting using jconsole. Also the Zabbix Status for the monitored maschine is green.
    unfortunately the jmx symbol is red and says :

    java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi

    on jboss log i have:

    Remote connection failed: java.io.IOException: Received an invalid message length of 1246907721

    and

    WARN [org.jboss.remotingjmx.protocol.v2.ServerProxy] (pool-2-thread-1) Unable to send notification to listener 1

    I followed your tutorial step by step. Any suggestions ?

  3. Pingback: Monitorar JBoss EAP 6.x Domain Mode | Infradev

  4. Thanks a lot for the great tutorial, followed, and works like charm, I can collect the Mamory usage f.e.

    Unfortunately I need some data from jboss nodes working in a jboss domain.

    Your example of Datasource Status doesn’t work, since there’s no such object and member using AS 7.1.1. domain mode, but if I try to collect the value for let’s say jmx[“jboss.as:subsystem=web,connector=http”,bytesSent], java agent just reports “Not supported”.

    I have tried several objects, with/out/ “.intValue, .longValue() etc.”, they are still “Not supported”.

    What am I doing wrong?

  5. We run several jboss instances on the same server. Listen IP addresses are shifted, not the ports – but it is irrelevant at the moment.

    As far as I understand at the moment, jmx metrics (“Key” value in the Item) must be unique for the host, so I cannot monitor more than one HeapMemoryUsage…

    Is it really the case? It would be quite a pity for me.

    • I was too quick, as it happened many times before… 🙂

      I have created separate hosts in zabbix for each of the jboss instances. There is only jmx interface for theese hosts (no agent), and they share a common template with the jmx metrics we need.

  6. Dennis,

    Hi, thank you for the great instructions! I’m trying to configure Zabbix 3.0.1 with JBoss EAP 6.4, but I know almost nothing of compiling java.

    Do you have a newest RPM of zabbix-java-gateway? I’m using Zabbix 3.0.1 on RHEL 7.

    I tried… 😉

  7. Dennis,

    I am running JBoss 6.4.0 and zabbix 3.2.1 and it is a domain and ssl turned on. Has anyone successfully made this work with this combination?

Leave a Reply

Your email address will not be published. Required fields are marked *