XVM Heartbeats

Configure periodic XVM-level statistics emission from your XVM.

Overview

XVM heartbeats provide periodic snapshots of XVM-level statistics including system metrics, thread statistics, pool usage, engine stats, and application-defined stats. Heartbeats can be emitted via messaging, logged to binary transaction logs, or traced for debugging.

Statistics collection must be balanced against performance impact. While many statistics have minimal overhead, enabling all collection options can impact latency-sensitive applications.

Heartbeat Configuration

Heartbeats for an XVM can be enabled via DDL using the <heartbeats> element:

<xvms>
  <xvm name="my-xvm">
    <heartbeats enabled="true" interval="5">
      <collectNonZGStats>true</collectNonZGStats>
      <collectIndividualThreadStats>true</collectIndividualThreadStats>
      <collectSeriesStats>true</collectSeriesStats>
      <collectSeriesDatapoints>false</collectSeriesDatapoints>
      <maxTrackableSeriesValue>100000000</maxTrackableSeriesValue>
      <includeMessageTypeStats>false</includeMessageTypeStats>
      <collectPoolStats>true</collectPoolStats>
      <poolDepletionThreshold>1.0</poolDepletionThreshold>
      <logging enabled="true"></logging>
      <tracing enabled="true"></tracing>
    </heartbeats>
  </xvm>
</xvms>

Configuration Settings

Configuration Setting	Default	Description
enabled	false	Enable or disable XVM stats collection and heartbeat emission. Performance Impact: Collection of stats and emission of heartbeats can impact application performance from both a latency and throughput standpoint. For applications that are particularly sensitive to performance, it is a good idea to compare performance with and without heartbeats enabled to understand the overhead that is incurred by enabling heartbeats.
interval	1000	The interval in seconds at which XVM stats will be collected and emitted
collectNonZGStats	true	Some statistics collected by the stats collection thread require creating a small amount of garbage. This can be set to false to suppress collection of these stats
collectIndividualThreadStats	true	Indicates whether heartbeats will contain stats for each active thread in the JVM. Individual thread stats are useful for identifying busy threads and thread affinitization candidates
collectSeriesStats	true	Indicates whether or not series stats should be included in heartbeats
collectSeriesDatapoints	false	Indicates whether or not series stats should report the data points captured for a series statistic. Warning: Enabling this value includes each datapoint collected in a series in heartbeats which can make emitted heartbeats very large and slow down their collection. It is not recommended that this be run in production.
maxTrackableSeriesValue	10 minutes	The maximum value (in microseconds) that can be tracked for reported series histogram timings. Datapoints above this value will be downsampled to this value, but will be reflected in the max value reported in an interval
includeMessageTypeStats	false	Sets whether or not message type stats are included in heartbeats (when enabled for the app). When captureMessageTypeStats is enabled for an app, the AEP engine will record select statistics on a per message type basis. Because inclusion of per message type stats can significantly increase the size of heartbeats, inclusion in heartbeats is disabled by default. Tip: For message type stats to be included in heartbeats, both captureMessageTypeStats for the app must be set to true (capture is disabled by default because recording them is costly), and includeMessageTypeStats must be set to true (inclusion is disabled by default because emitting them is costly).

| | collectPoolStats | true | Indicates whether or not pool stats are collected by the XVM | | poolDepletionThreshold | 1.0 | Configuration property used to set the percentage decrement at which a preallocated pool must drop to be included in an XVM heartbeat. Setting this to a value greater than 100 or less than or equal to 0 disables depletion threshold reporting. This gives monitoring applications advanced warning if it looks like a preallocated pool may soon be exhausted. By default the depletion threshold is set to trigger inclusion in heartbeats at every 1% depletion of the preallocated count. This can be changed by specifying the configuration property nv.server.stats.pool.depletionThreshold to a float value between 0 and 100. Example: If a pool is preallocated with 1000 items and this property is set to 10, pool stats will be emitted for the pool each time a heartbeat occurs and the pool has dropped below a 10% threshold of the preallocated size (e.g., at 900, 800, 700, until its size reaches 0). | | logging | See below | Configures binary logging of heartbeats. Binary heartbeat logging provides a means by which heartbeat data can be captured in a zero garbage fashion. Collection of such heartbeats can be useful in diagnosing performance issues in running apps | | tracing | See below | Configures trace logging of heartbeats. Enabling textual tracing of heartbeats is a useful way to quickly capture data from XVM heartbeats for applications that aren't monitoring XVM heartbeats remotely. Textual trace of heartbeats is not zero garbage and is therefore not suitable for applications that are latency sensitive |

Trace Output Configuration

By default all XVM statistics tracers are disabled as trace logging is not zero garbage and introduces CPU overhead. While tracing heartbeats isn't recommended in production, enabling XVM statistics trace output can be useful for debugging and performance tuning.

<xvms>
  <xvm name="my-xvm">
    <heartbeats enabled="true" interval="5">
      <tracing enabled="true">
        <traceSysStats>true</traceSysStats>
        <traceThreadStats>true</traceThreadStats>
        <tracePoolStats>true</tracePoolStats>
        <traceAppStats>true</traceAppStats>
        <traceUserStats>true</traceUserStats>
      </tracing>
    </heartbeats>
  </xvm>
</xvms>

Binary Logging Configuration

Applications that are latency sensitive might prefer to leave all tracers disabled to avoid unnecessary allocations and the associated GC activity. As an alternative, it's possible to enable logging of zero-garbage heartbeat messages to a binary transaction log:

<xvms>
  <xvm name="my-xvm">
    <heartbeats enabled="true" interval="5">
      <logging enabled="true">
         <storeRoot>/path/to/heartbeat/log/directory</storeRoot>
      </logging>
    </heartbeats>
  </xvm>
</xvms>

When a storeRoot is not set, an XVM will log heartbeats to {XRuntime.getDataDirectory}/server-heartbeats/<xvm-name>-heartbeats.log, which can then be queried and traced from a separate process using the Stats Dump Tool.

Warning: At this time binary heartbeat logs do not support rolling collection. Consequently this mechanism is not suitable for long running application instances.

Related Topics

• XVM Stats and Heartbeats - Viewing and interpreting heartbeat statistics at runtime

• Engine Statistics - Configure engine-level statistics collection

• Stats Dump Tool - Analyzing binary heartbeat logs

• Operating Model - Monitoring architecture overview

Next Steps

1. Enable heartbeats with appropriate collection settings

2. Test performance impact under load

3. Configure binary logging for zero-garbage collection or tracing for debugging

4. Set up monitoring tools to consume heartbeats

5. Balance operational visibility against performance requirements