With the release of Java 1.5, JMX is built-in and readily usable in even the smallest of applications. It's now a simple task to instrument nearly any application and expose important monitoring metrics. Custom applications, involving many processes and multiple middleware components, may be effectively managed from a remote console. As a result, there's been an explosion of data exposed through JMX and available for analysis and presentation.

As the use of JMX expands, one would expect that mistakes be made and lessons learned the hard way. Software developers are usually much more familiar with their own application domain and are often not experts in monitoring and management tools.

The SL Corporation and I have over 20 years of experience in monitoring and visualization applications, with particular expertise in Java. The company's Enterprise RTView product has been specially adapted to deal with real-time data produced by JMX-enabled applications and has features to compensate for many overlooked requirements.

This article discusses errors that have been repeatedly seen in JMX implementations regarding the content and design of data structures known as MBeans. Common JMX best practice knowledge is briefly reviewed to provide some initial context. This is followed by a detailed discussion about issues that arise when custom application monitoring requirements grow to include aggregation, analysis, and visualization in real-time. Recommendations are then offered that may help users make optimal use of JMX in these situations.

Common Best Practices
JMX is a very general solution framework and doesn't define specific monitoring or management data structures. This puts the burden on developers to establish conventions themselves to extract and process information consistently so it can be analyzed and visualized.

Best practice documents for JMX typically suggest adherence to common standards regarding naming conventions, data types, deployment constraints, portability, and so on. These suggestions fall into several categories:

MBean Usage Conventions
Specific conventions should be applied to MBean names and JMX data types to implement MBeans that are portable, i.e., commonly available JMX client applications can handle them without issue. Generally, this means one should follow the guidelines for using standard domain and key names (such as type= and name=) and OpenMBean data types instead of custom Java classes (for ease of deployment). These suggestions are presented in depth in Sun's discussion of JMX best practices.

MBean names should be predictable and used in a consistent manner when representing a hierarchy. Most importantly, one should define the same attribute schema for all beans of the same type or at the same level in a hierarchy. It's obvious that a violation of this principle will result in complications for client applications.

Use Standard MBeans wherever possible; these are the simplest to implement and the easiest to maintain, since JMX implicitly understands data types and run-time behavior simply from the Java source directives. There are situations where Dynamic and/or Model MBeans may be necessary, but their use should be kept to a minimum.

Follow Established Design Patterns
Use well-established design patterns for defining MBeans. Several of these are described in Design Patterns for JMX and Application Manageability. Often, the motive behind these patterns is to shield users of JMX MBeans from the details of their implementation or even their attribute structure. This lets the developer change the MBean's details without affecting client applications.

Other patterns are useful in maintaining separation between the monitoring and management code contained in the JMX MBean and the business logic in the application. There are significant advantages in terms of maintainability if the application being monitored has little or no knowledge of how the monitoring is done.

Much has been written about Aggregator beans as a way to minimize the number of MBeans that a client has to connect to and query. When the number of beans reaches into the thousands, performance issues come into play. The Aggregator pattern is one way to improve performance by minimizing MBean access.

Model After Established Systems
The most popular J2EE application servers make extensive use of JMX as a tool for managing their internal functions. These very large systems provide solid ground for testing implementation techniques for JMX MBeans. As such, they provide excellent examples to follow.

In the administration system for the BEA WebLogic Server, there can be over a thousand different MBean instances. MBeans have been developed to address just about every variation of monitoring and management problem. By exploring the techniques used in this system, much can be learned and applied to your own requirements. The BEA JMX system is especially complete when it comes to the use of notifications for monitoring system performance.

The J2EE Management Specification JSR-77 defines a Management Model and provides useful guidelines for data types and implementation patterns. Effective management and monitoring involves states, statistics, metrics, relationships, and more. Understanding these information structures can be helpful in developing quality systems.

Monitoring, Analysis & Visualization Issues
The best practices outlined above are useful, but they don't fully address issues that come up in large-scale monitoring applications involving visualization and, in particular, dynamic analysis of real-time data. Requirements, such as trending or slice-and-dice data analysis, are often afterthoughts. It's important to understand these issues upfront and develop an implementation plan that minimizes the work required to present the data. Often, this can influence the design of the MBeans developed for such an application.

The best way to illustrate this is with a concrete example. Consider developing a monitoring system for a sample application, a simple message switching system. This example is representative of many applications. Similar data structures are relevant for monitoring the performance of routers, message boxes, caching systems, object databases, even CEP engines. Shared among them is the desire to use JMX for instrumentation.

Our sample device supports multiple channels for message traffic and collects metrics and statistics about its operation in real-time, so performance can be optimized. Table 1 shows a simple data structure containing information that might be exposed via a JMX MBean for a single channel.

The Total Msgs Sent/Rcvd counts would typically be represented as a long integer since the number could grow larger over time. The memory usage could be stored as an integer since the Java memory limit on many machines is 1GB, so 32-bits will handle that effectively. The other data - Channel ID and Statistics - are stored as strings.