Memory Statistics
Overview
Talon tracks detailed memory usage statistics covering JVM heap, off-heap (native) memory, IO buffers, and ADM entity lifecycle. These statistics are available in XVM heartbeats and can also be output independently via standalone trace logging. This page describes the statistics that are collected and the format in which they are traced.
Configuring Memory Statistics
Memory stats collection is disabled by default. For complete configuration details including enablement, per-entity-type breakdowns, and standalone trace output, see:
Configuring Memory Statistics - Complete memory statistics configuration guide
This page focuses on understanding and interpreting the statistics once they are collected.
Memory Statistics Categories
Memory statistics are collected by the MemoryStats singleton and cover four areas:
Host Memory — System-level physical memory, swap, and committed virtual memory
Process Memory — Physical memory footprint of the process (RSS)
JVM Memory — Standard JVM heap and non-heap memory pools (init, used, committed, max)
App Memory — Off-heap (native) memory, IO buffer lifecycle, and ADM entity object management
Host Memory
The following system-level memory fields are included in the heartbeat's SrvMonSysMemoryStats:
totalPhysicalMemorySize
Total physical memory on the host in bytes
freePhysicalMemorySize
Available physical memory in bytes. On Linux this reports MemAvailable (free + reclaimable cache/buffers) rather than raw MemFree, providing a more accurate measure of usable memory
committedVirtualMemorySize
Virtual memory guaranteed to be available to the process in bytes
totalSwapSpaceSize
Total swap space on the host in bytes
freeSwapSpaceSize
Free swap space in bytes
Note: Prior to 3.16.x, freePhysicalMemorySize reported raw MemFree on Linux, which excludes memory used by the kernel for page cache and buffers. The updated behavior reports MemAvailable, which accounts for reclaimable cache and provides a much more accurate picture of whether the host is running low on memory. On a typical Linux server, MemFree might report 500MB while MemAvailable reports 12GB.
Process Memory
processRssBytes
The process RSS (Resident Set Size) in bytes — the actual physical memory currently occupied by the process, including code, stack, heap, and memory-mapped regions. Sourced via native JNI calls to platform-specific APIs (/proc/self/stat on Linux, mach_task_basic_info on macOS). Returns 0 if native runtime is not available.
Heap Usage (SrvMonSysMemoryUsage)
SrvMonSysMemoryUsage)Standard JVM heap memory usage from MemoryMXBean.getHeapMemoryUsage():
init
Initial heap memory requested from the OS
used
Heap memory currently in use
committed
Heap memory guaranteed to be available
max
Maximum heap memory that can be used
Non-Heap Usage (SrvMonSysMemoryUsage)
SrvMonSysMemoryUsage)JVM non-heap memory usage from MemoryMXBean.getNonHeapMemoryUsage():
init
Initial non-heap memory (e.g., metaspace)
used
Non-heap memory currently in use
committed
Non-heap memory committed by the JVM
max
Maximum non-heap memory (0 if unbounded)
Off-Heap Usage (SrvMonSysMemoryOffHeapUsage)
SrvMonSysMemoryOffHeapUsage)Aggregated off-heap memory usage from MemoryStats:
usedLive
Total native + direct IO buffer memory currently in active use
usedPooled
Direct IO buffer memory held in pools available for reuse
Native Memory Counters (SrvMonSysMemoryNativeCounters)
SrvMonSysMemoryNativeCounters)Raw counters for native (non-JVM) memory operations:
allocCount
Number of native memory allocations performed
allocBytes
Cumulative bytes allocated via native allocations
reallocCount
Number of native memory reallocations performed
reallocBytes
Cumulative bytes reallocated
freeCount
Number of native memory frees performed
freeBytes
Cumulative bytes freed
IO Buffer Counters (SrvMonSysMemoryIOBufCounters)
SrvMonSysMemoryIOBufCounters)Counters for IO buffer lifecycle operations. IO buffers are used internally for message serialization, packet construction, and data transfer:
bufferAllocCount
Number of IO buffers created via new allocation (not from pool)
bufferAllocBytes
Cumulative bytes from new IO buffer allocations
bufferPoolAllocCount
Number of IO buffers obtained from the pool
bufferPoolAllocBytes
Cumulative bytes obtained from the IO buffer pool
bufferReallocCount
Number of IO buffers that were reallocated due to a size change
bufferReallocBytes
Cumulative bytes from IO buffer reallocations
bufferForkCount
Number of IO buffers that were forked (copy-on-write)
bufferForkBytes
Cumulative bytes from forked IO buffers
bufferGCDisposeCount
Number of IO buffers disposed to the garbage collector
bufferGCDisposeBytes
Cumulative bytes of IO buffers disposed to GC
bufferPoolDisposeCount
Number of IO buffers returned to the pool
bufferPoolDisposeBytes
Cumulative bytes returned to the IO buffer pool
bufferPoolLeakCount
Number of IO buffers evicted from the pool and leaked to GC
bufferPoolLeakBytes
Cumulative bytes leaked from the IO buffer pool
Entity Type Counters (SrvMonSysMemoryEntityTypeCounters)
SrvMonSysMemoryEntityTypeCounters)Entity lifecycle counters are included for each of four categories: messages, state entities, embedded entities, and collections. Each category includes both aggregate counters and optional per-type breakdowns (when nv.memory.stats.type.enable=true).
liveCount
Number of entity instances currently in active use
pooledCount
Number of entity instances currently in pools
createAllocCount
Number of entities created via new allocation
poolAllocCount
Number of entities obtained from a pool
gCDisposeCount
Number of entities disposed to the garbage collector
poolDisposeCount
Number of entities returned to a pool
poolLeakCount
Number of entities evicted from a pool and leaked to GC
pojoLiveCount
POJO wrappers currently in active use
pojoPooledCount
POJO wrappers currently in pools
pojoCreateAllocCount
POJOs created via new allocation
pojoPoolAllocCount
POJOs obtained from a pool
pojoGCDisposeCount
POJOs disposed to GC
pojoPoolDisposeCount
POJOs returned to a pool
pojoPoolLeakCount
POJOs leaked from a pool
serializerLiveCount
Serializers currently in active use
serializerPooledCount
Serializers currently in pools
serializerCreateAllocCount
Serializers created via new allocation
serializerPoolAllocCount
Serializers obtained from a pool
serializerGCDisposeCount
Serializers disposed to GC
serializerPoolDisposeCount
Serializers returned to a pool
serializerPoolLeakCount
Serializers leaked from a pool
deserializerLiveCount
Deserializers currently in active use
deserializerPooledCount
Deserializers currently in pools
deserializerCreateAllocCount
Deserializers created via new allocation
deserializerPoolAllocCount
Deserializers obtained from a pool
deserializerGCDisposeCount
Deserializers disposed to GC
deserializerPoolDisposeCount
Deserializers returned to a pool
deserializerPoolLeakCount
Deserializers leaked from a pool
metadataLiveCount
Metadata buffers currently in active use
metadataLiveBytes
Bytes of metadata buffers currently in active use
metadataAllocCount
Metadata buffers allocated
metadataAllocBytes
Cumulative bytes of metadata allocated
metadataGCDisposeCount
Metadata buffers disposed to GC
metadataGCDisposeBytes
Cumulative bytes of metadata disposed to GC
metadataPoolDisposeCount
Metadata buffers returned to a pool
metadataPoolDisposeBytes
Cumulative bytes of metadata returned to a pool
dataLiveCount
Data buffers currently in active use
dataLiveBytes
Bytes of data buffers currently in active use
dataAllocCount
Data buffers allocated
dataAllocBytes
Cumulative bytes of data allocated
dataGCDisposeCount
Data buffers disposed to GC
dataGCDisposeBytes
Cumulative bytes of data disposed to GC
dataPoolDisposeCount
Data buffers returned to a pool
dataPoolDisposeBytes
Cumulative bytes of data returned to a pool
When per-entity-type breakdown is enabled, each counter set also includes:
entityName
Fully qualified class name of the entity type
entityShortName
Simple class name of the entity type
Memory Stats in XVM Heartbeats
When XVM heartbeats are enabled, memory statistics are automatically included in the system stats section of each heartbeat. The heartbeat trace output shows system and process memory:
In addition to the textual trace, the heartbeat message (SrvMonHeartbeatMessage) carries structured memory statistics that can be consumed programmatically. These are organized into the sub-messages described in the sections above.
Standalone Memory Stats Trace
When standalone trace output is enabled (see Configuring Memory Statistics), memory stats are periodically traced to the nv.memory.stats logger.
Trace Output Format
The standalone trace output includes the following sections:
When per-entity-type tracking is enabled (nv.memory.stats.type.enable=true), each entity category is followed by per-type breakdowns prefixed with ...:
Interpreting the Trace Output
Host
totalPhysicalMemory
Total physical memory on the host
availableMemory
Available physical memory on the host. On Linux, reports MemAvailable (free + reclaimable cache/buffers) rather than raw MemFree
Process
rssBytes
Process RSS (Resident Set Size) — actual physical memory occupied by the process
Heap
init
The initial amount of heap memory the JVM requested from the OS at startup
used
The amount of heap memory currently in use
committed
The amount of heap memory guaranteed to be available to the JVM
max
The maximum amount of heap memory that can be used
Off Heap
usedLive
Total off-heap memory currently in live use. This is the sum of native memory in use (alloc - free + realloc) plus direct IO buffer memory in use (alloc + poolAlloc - poolDispose - gcDispose)
usedPooled
Off-heap memory currently held in IO buffer pools, available for reuse (poolDispose - poolAlloc - poolLeak)
nativeAlloc
Cumulative bytes allocated via native memory allocations
nativeFree
Cumulative bytes freed via native memory frees
nativeRealloc
Cumulative bytes reallocated via native memory reallocations
directAlloc
Cumulative bytes allocated by creating new direct IO buffers (not from pool)
directPoolAlloc
Cumulative bytes obtained from the direct IO buffer pool
directPoolDispose
Cumulative bytes returned to the direct IO buffer pool
directGCDispose
Cumulative bytes of direct IO buffers disposed to the garbage collector
Entity Stats
Each entity category (Message Entities, Embedded Entities, State Entities, State Collections) reports the following fields:
live
Number of entity instances currently in use (createAlloc + poolAlloc - gcDispose - poolDispose)
pooled
Number of entity instances currently held in pools (poolDispose - poolAlloc - poolLeak)
Po
POJO (Plain Old Java Object) wrapper counts — {live, pooled}
Se
Serializer counts — {live, pooled}
De
Deserializer counts — {live, pooled}
Me
Metadata buffer bytes currently in live use (allocBytes - poolDisposeBytes - gcDisposeBytes)
Da
Data buffer bytes currently in live use (allocBytes - poolDisposeBytes - gcDisposeBytes)
Tip: If the live count for messages or entities grows continuously, it may indicate a leak where objects are being allocated but never disposed or returned to the pool. Compare the live and pooled counts across intervals to identify trends.
Related Topics
Configuring Memory Statistics - Enable and configure memory stats collection
XVM Heartbeats - Interpreting heartbeat output
Configuring Monitoring - Heartbeat and statistics configuration
Trace Logging - General trace logging configuration
Last updated