JBoss monitoring and Zabbix

For the last few days I’ve been busy getting my JBoss environment into the Zabbix monitoring tool. While I found some good information on the internet, I ended up creating something from scratch and thought it would be nice to share it with you.

Update 27-01-2014: This post is only valid for JBoss EAP 5 and will not work for EAP 6. Please see this post for JBoss Enterprise Application Platform 6.

First things first

First, let me explain my environment and goals. I’ll be setting up monitoring for JBoss EAP 5.2.0, running on Red Hat Enterprise Linux 6.5 with a Zabbix Agent installed. The used version of Zabbix is 2.2.1.

  • Zabbix is running on zabbix.kanbier.lan
  • JBoss on jboss.kanbier.lan

The Zabbix Agent is active on the JBoss server and functional.

JBoss is running and functional, and remote JMX monitoring is enabled.

My primary goal in this post is to monitor all created queue’s in JBoss using Zabbix. And I don’t want to add *all* my queues by hand.

There a a few things we need to do to make this possible:

  • We need to know what queues are available for monitoring
  • We need to have remote access to JMX to get values from the queues

Let’s start by getting a list from JBoss containing all possible queues, and add them to Zabbix.

JBoss’s list…

Zabbix comes with a feature to auto discover items for a host. This is pre configured for filesystems for example. But they also added the possibility to use custom discover rules.

Before we dig into the rule, let’s take a look at what we actually want to know. We want to know what Queues are present in JBoss. How can we tell?

We could of course look at the jmx-console web interface at http://<your_jboss_instance>:8080/jmx-console/:

Screen Shot 2014-01-13 at 15.07.22

 

While this presentation is easy on the eye, it’s not something Zabbix is willing to convert to host items. It would be a good idea to have this data available from the command line, so we can easily convert it to JSON which Zabbix will happily process.

JBoss comes with a tool called twiddle.sh. Twiddle.sh is a command line interface into JBoss, just what we need.

Since we’re going to need twiddle.sh later on, I’ll configure it at the appropriate place. If this doesn’t seem logical to you right now you’re probably right, but I’ll make better sense in a bit.

Twiddle.sh is found in the bin directory of your JBoss installation:

# find /opt -name twiddle.sh
/opt/jbossas/jboss-as/bin/twiddle.sh

Be sure to check out the help output of twiddle.sh to see what it does. For now it’s important to know it also accepts a parameter file to use for authentication.

I’ll be configuring it for use with zabbix, so let’s create a directory for that:

# mkdir -p /opt/zabbix/twiddle
# chown -R zabbix:zabbix /opt/zabbix

Create a parameter file for twiddle.sh so you don’t have to keep entering your username/password. Open /opt/zabbix/twiddle/twiddle.properties and place the following contents in it. Replace the values with the proper onces for your environment of course:

twiddle.user=admin
twiddle.password=adminpasswd
twiddle.server=ip_of_jboss_server

Now twiddle uses a jar file which has it’s log4j parameters compiled in. I’m not totally sure why, but if you’re not using root to run twiddle.sh  you’ll get a “permission denied” message because it can’t write to the logfile. We’re going to run twiddle.sh as the ‘zabbix’ user, so we’ll override the log4j parameters using yet another property file.

Open /opt/zabbix/twiddle/log4j.properties and place the following contents:

log4j.rootCategory=OFF, CONSOLE
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Target=System.out
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ABSOLUTE} %-5p [%c{1}] %m%n

Now with the two property files in place, let’s run twiddle to see if we can get a list of message queues (this is one line):

# export JAVA_OPTS=”-Dlog4j.configuration=file:/opt/zabbix/twiddle/log4j.properties”;/opt/jbossas/jboss-as/bin/twiddle.sh -P /opt/zabbix/twiddle/twiddle.properties query jboss.messaging.destination:*

I got the object name filter “jboss.messaging.destination:” from the jmx-console web interface. Using the above command you just query the jboss.messaging.destionation for everything (*).

The result:

jboss.messaging.destination:service=Queue,name=DLQ
jboss.messaging.destination:service=Queue,name=ExpiryQueue
jboss.messaging.destination:service=Queue,name=MyAwesomeQueue

Great! We now have a list of messaging queues in our JBoss instance. But it’s not in JSON format so Zabbix won’t accept it as input for LLD.

JBoss and queryJBoss.pl

We have the output we need, just not in JSON format. Wouldn’t it be nice if there was something to easily query for this kind of data, convert the output to JSON and have Zabbix process it.

Even beter, let the Zabbix Agent do the querying so you can just have a discovery rule in your JBoss Template. Add the template to a host and your queues (or any other data you wish) automagically get added to the host.

I’ve created a Perl script that just does that: queryJBoss.pl.

The script queryJBoss.pl accepts two arguments:

  • an object name filter (i.e. jboss.messaging.destination) (required)
  • a Zabbix macro name (optional)

Let’s download the script and set it up:

# cd /opt/zabbix/
# wget https://raw2.github.com/dkanbier/zabbix-jboss/master/queryJBoss.pl
# chown zabbix:zabbix queryJBoss.pl
# chmod +x queryJBoss.pl

There is one thing you probably need to modify in the script, the location of the twiddle.sh script. We used a find command earlier:

 # find /opt -name twiddle.sh
/opt/jbossas/jboss-as/bin/twiddle.sh

Be sure this matches the $_twiddle parameter under “# Options” in the queryJBoss.pl script. That’s probably the only option you need to modify.  Let’s try running the script:

# /opt/zabbix/queryJBoss.pl jboss.messaging.destination

Result:

“data”:[

{
“{#JBOSSMESSAGINGDESTINATION_JNDI}”:”jboss.messaging.destination:service=Queue,name=DLQ”,
“{#JBOSSMESSAGINGDESTINATION_SERVICE}”:”Queue”,
“{#JBOSSMESSAGINGDESTINATION_NAME}”:”DLQ”
}
,
{
“{#JBOSSMESSAGINGDESTINATION_JNDI}”:”jboss.messaging.destination:service=Queue,name=ExpiryQueue”,
“{#JBOSSMESSAGINGDESTINATION_SERVICE}”:”Queue”,
“{#JBOSSMESSAGINGDESTINATION_NAME}”:”ExpiryQueue”
}
,
{
“{#JBOSSMESSAGINGDESTINATION_JNDI}”:”jboss.messaging.destination:service=Queue,name=MyAwesomeQueue”,
“{#JBOSSMESSAGINGDESTINATION_SERVICE}”:”Queue”,
“{#JBOSSMESSAGINGDESTINATION_NAME}”:”MyAwesomeQueue”
}

]
}

The script used twiddle.sh and made the output to be in JSON format. There is just one “problem”, easily spotted by lazy admins like yourself. The Zabbix macros ({#JBOSSMESSAGINGDESTINATION_SERVICE}) are a bit on the long side. And since you’ll need to type these once you create a Zabbix template, I’d prefer a name like {#QUEUE_SERVICE}.

This is what the second optional argument for the queueJBoss.pl script does. The Zabbix macro name should be unique per template so we need to add something to them once we create a new discovery rule. ( Don’t worry we’ll get to creating the discovery rule next… )  Running the script like this:

# /opt/zabbix/queryJBoss.pl jboss.messaging.destination QUEUE

Returns:

{
“data”:[

{
“{#QUEUE_JNDI}”:”jboss.messaging.destination:service=Queue,name=DLQ”,
“{#QUEUE_SERVICE}”:”Queue”,
“{#QUEUE_NAME}”:”DLQ”
}
,
{
“{#QUEUE_JNDI}”:”jboss.messaging.destination:service=Queue,name=ExpiryQueue”,
“{#QUEUE_SERVICE}”:”Queue”,
“{#QUEUE_NAME}”:”ExpiryQueue”
}
,
{
“{#QUEUE_JNDI}”:”jboss.messaging.destination:service=Queue,name=MyAwesomeQueue”,
“{#QUEUE_SERVICE}”:”Queue”,
“{#QUEUE_NAME}”:”MyAwesomeQueue”
}

]
}

Much better looking, and less typing. All right, so we’ve got the data we want and it’s in the correct format. Let’s set Zabbix to work!

Zabbix Low-Level Discovery (LLD) 

First we configure the Zabbix Agent on the JBoss server. We’ll want the agent to run queryJBoss.pl to generate a list of discovered queues so Zabbix can add them to the host.

Edit your zabbix_agentd.conf file and add a UserParameter for the queryJBoss.pl script:

UserParameter=jboss.queue.discovery,/opt/zabbix/queryJBoss.pl jboss.messaging.destination QUEUE

This tells Zabbix there is a key available to query for discovery: “jboss.queue.disovery”.  When Zabbix asks for this key, the agent will run “/opt/zabbix/queryJBoss.pl jboss.messaging.destination QUEUE”.  Like we’ve seen earlier, this returns a list of queues in JSON format.

Don’t forget to restart the Zabbix Agent after changing the configuration file!

So things are looking good on the JBoss side of things. Let’s configure a “Discovery rule” in Zabbix.

Open up your Zabbix console and go to: Configuration -> Templates and click the “Create template” button and fill in:

“Template name” : Template JBoss Generic
“Groups”: add “Templates”

Screen Shot 2014-01-13 at 16.21.50

Click Save.

Now in the Templates view click “Discovery” in the row of the “Template JBoss Generic” template. Click “Create discovery rule” in the next screen and fill in:

Name: JBoss queue discovery
Type: Zabbix agent
Key: jboss.queue.discovery 

Screen Shot 2014-01-13 at 16.29.32
Leave the rest  on default, make sure “Enabled” is checked and click Save.

Just a quick note before we continue, did you notice the value of “Key” is the same as the value we configured earlier in zabbix_agentd.conf?

Also note the Update interval. This is set to query for new queues every 30 seconds. A bit on the high side if you ask me, I’d say once an hour is plenty.

All right, let’s continue. We now have the ability to add items to the host using this discovery rule. The items to add using a discovery rule are called “Item prototypes”. Let’s take a look. In the “Discovery rules” screen, click “Item Prototypes”. Click “Create item prototype” .

I’m interested in the current message count of my queues, so I’m going to create an item for them.

NOTE: Remember I assume remote JMX monitoring is working.

Fill in:

Name: Message count for {#QUEUE_NAME}
Type: JMX agent
Key: jmx[{#QUEUE_JNDI},MessageCount]
New Application: JBoss Queues

Screen Shot 2014-01-13 at 16.35.55

Click Save.

Just another quick note on what happens here with the Zabbix macros ({#QUEUE_NAME}). Remember earlier where I explained you can use the second argument to queryJBoss.pl to set a custom Zabbix macro name? This is where the macros are being used by Zabbix.

Per queue that is found by the Zabbix Agent, it sends it’s data to Zabbix with these names. So for every queue there will be an item created using the values that are stored in the Zabbix macros.

We’re almost done. We just need to link the template to the JBoss host. In Zabbix, click Configuration -> Hosts -> JBoss Host. Click the “Templates” tab in the Host screen and add the “Template JBoss Generic” to the Host.

Click Save.

Now if everything went well, after 30 seconds you should see the discovered items appearing in Zabbix. Go to Configuration -> Hosts -> JBoss Host -> Items:

Screen Shot 2014-01-13 at 16.51.50

And there we go. The “message count” parameter is now being monitored by Zabbix for all my queues.

Please notice that this blogpost only covers the “queues”, but you can monitor anything you can see in jmx-console the very same way. For example, let’s say you want to monitor your Managed Connection Pools you only need to add a discovery rule to the agent configuration:

 UserParameter=jboss.pool.discovery,/opt/zabbix/queryJBoss.pl jboss.jca POOL

Don’t forget to restart zabbix_agentd 😉

Run the queryJBoss.pl tool once so you know which Zabbix macro’s you need to configure in your discovery rule:

# ./queryJBoss.pl jboss.jca POOL

Result:

{“{#POOL_JNDI}”:”jboss.jca:service=ManagedConnectionPool,name=JmsXA”,
“{#POOL_SERVICE}”:”ManagedConnectionPool”,
“{#POOL_NAME}”:”JmsXA”
}

With this information you can create a discovery rule for {#POOL_NAME} just like we did for {#QUEUE_NAME}.

NOTE: The output of the last example contains more information and returns not only ManagedConnectionPools but also ConnectionFactories and other stuff. You can filter these out with the “Macro” and “Regex” parameters in the Discovery Rule creation page.

Happy monitoring!

Leave a Reply

Your email address will not be published. Required fields are marked *