Out-Of-The-Box JMX over Firewall in Hotspot 7

Finally! A minor feature with big impact made it into the Hotspot 7 - probably as a result of the JRockit and Hotspot convergence efforts (as JRockit has had this feature for ages). The absence of this feature has been a big pain in the past, at least for people like me that are a bit obsessed with runtime monitoring, and effectively prevented one from using JMX in production-like environments.

The problem that all hotspot JVM versions until later Java7 releases had, was that it was not possible to define the RMI port used in the stubs returned by the JMX registry. It was possible (and still is) to define the JMX registry port via command-line arguments (-Dcom.sun.management.jmxremote.port). This allows you to connect to the JMX registry, lookup the actual RMI stub and download it (done by the JVM for you). The stub unfortunately was referring to a remote object running on a second port (aka ephemeral JMX port) which was not configurable and, to make things worse, was randomly picked upon JVM startup. This made it next to impossible (unless you open your firewall completely for inbound TCP connections to ports > 1024) to actually use JMX over a firewall. This (minor) problem forced you to either write your own JMX socket factory which needs to be initialized when the JVM process was started (e.g. using weblogic startup classes, special tomcat connectors, etc.) or forget about JMX in production.

In recent versions of the JVM (at least starting from 7u25) it is also possible to configure the second (ephemeral) port. You can (and probably would want to) set it to the same port as the registry thus requiring a firewall rule for one single TCP port. The new JVM argument is -Dcom.sun.management.jmxremote.rmi.port. To enable anonymous and unencrypted JMX connections over a firewall you need the following startup arguments (note how both ports are set and also set to the same value):

-Dcom.sun.management.jmxremote.port=2811 -Dcom.sun.management.jmxremote.rmi.port=2811 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

and an inbound firewall rule for TCP port 2811.

Have fun!

PS. In case you are also interested in open-source and very flexible production monitoring of your JVMs you can have a look at graphite and metrics-sampler.

Java Performance Pitfalls - DatatypeFactory

The DatatypeFactory class available in JSE since 1.5 is used to create javax.xml.datatype for mapping between XML and java objects. It allows for easy creation of dates and timestamps to use in a JAXB context.

One typical use case of the factory is when you need to create a XML-serializable timestamp with time zone information. In such cases we usually first create an instance of the factory and then use this instance to create the timestamp. It looks as innocent as that:

DatatypeFactory.newInstance().newXMLGregorianCalendar(GregorianCalendar());

As such code fragments might appear quite often in code that needs lots of timestamps, you might have already defined this in a static utility method somewhere.

Although the code looks pretty harmless, it has a performance issue - the creation of datatype factory instances is a pretty expensive operation. It involves going through your class-loaders and trying to determine which class it should instantiate. Not only is that costly in terms of CPU performance but it also involves some class-loader internal synchronization (in our case in the Weblogic classloader) which greatly affects concurrency. Unfortunately, the result of this resolution is not cached internally so the whole process is done each time you call the newInstance() method. The main issue IMHO is that the java API documentation does not clearly state how expensive the operation is (although the javadocs describe the implementation resolution mechanism which might make you suspicious). As a result developers usually end up creating new instances all the time in critical paths. In normal low concurrency scenarios this is not an issue but becomes noticeable once you have a lot of threads trying to create datatype factory instances at the same time and consuming a lot of CPU and/or blocking due to synchronization in the classloader.

The solution to the problem is pretty simple - reuse previously created datatype factory instances. The javadocs do not state whether the implementation is thread-safe (which in our case it was) so you might need to store an instance of the factory in a thread-local variable in your utility class and reuse it when needed. This will make quite an impact in high-concurrency scenarios.