For the complete documentation index, see llms.txt. This page is also available as Markdown.

Memory Statistics

Overview

Talon tracks detailed memory usage statistics covering JVM heap, off-heap (native) memory, IO buffers, and ADM entity lifecycle. These statistics are available in XVM heartbeats and can also be output independently via standalone trace logging. This page describes the statistics that are collected and the format in which they are traced.

Configuring Memory Statistics

Memory stats collection is disabled by default. For complete configuration details including enablement, per-entity-type breakdowns, and standalone trace output, see:

This page focuses on understanding and interpreting the statistics once they are collected.

Memory Statistics Categories

Memory statistics are collected by the MemoryStats singleton and cover four areas:

  • Host Memory — System-level physical memory, swap, and committed virtual memory

  • Process Memory — Physical memory footprint of the process (RSS)

  • JVM Memory — Standard JVM heap and non-heap memory pools (init, used, committed, max)

  • App Memory — Off-heap (native) memory, IO buffer lifecycle, and ADM entity object management

Host Memory

The following system-level memory fields are included in the heartbeat's SrvMonSysMemoryStats:

Field
Description

totalPhysicalMemorySize

Total physical memory on the host in bytes

freePhysicalMemorySize

Available physical memory in bytes. On Linux this reports MemAvailable (free + reclaimable cache/buffers) rather than raw MemFree, providing a more accurate measure of usable memory

committedVirtualMemorySize

Virtual memory guaranteed to be available to the process in bytes

totalSwapSpaceSize

Total swap space on the host in bytes

freeSwapSpaceSize

Free swap space in bytes

Note: Prior to 3.16.x, freePhysicalMemorySize reported raw MemFree on Linux, which excludes memory used by the kernel for page cache and buffers. The updated behavior reports MemAvailable, which accounts for reclaimable cache and provides a much more accurate picture of whether the host is running low on memory. On a typical Linux server, MemFree might report 500MB while MemAvailable reports 12GB.

Process Memory

Field
Description

processRssBytes

The process RSS (Resident Set Size) in bytes — the actual physical memory currently occupied by the process, including code, stack, heap, and memory-mapped regions. Sourced via native JNI calls to platform-specific APIs (/proc/self/stat on Linux, mach_task_basic_info on macOS). Returns 0 if native runtime is not available.

Heap Usage (SrvMonSysMemoryUsage)

Standard JVM heap memory usage from MemoryMXBean.getHeapMemoryUsage():

Field
Description

init

Initial heap memory requested from the OS

used

Heap memory currently in use

committed

Heap memory guaranteed to be available

max

Maximum heap memory that can be used

Non-Heap Usage (SrvMonSysMemoryUsage)

JVM non-heap memory usage from MemoryMXBean.getNonHeapMemoryUsage():

Field
Description

init

Initial non-heap memory (e.g., metaspace)

used

Non-heap memory currently in use

committed

Non-heap memory committed by the JVM

max

Maximum non-heap memory (0 if unbounded)

Off-Heap Usage (SrvMonSysMemoryOffHeapUsage)

Aggregated off-heap memory usage from MemoryStats:

Field
Description

usedLive

Total native + direct IO buffer memory currently in active use

usedPooled

Direct IO buffer memory held in pools available for reuse

Native Memory Counters (SrvMonSysMemoryNativeCounters)

Raw counters for native (non-JVM) memory operations:

Field
Description

allocCount

Number of native memory allocations performed

allocBytes

Cumulative bytes allocated via native allocations

reallocCount

Number of native memory reallocations performed

reallocBytes

Cumulative bytes reallocated

freeCount

Number of native memory frees performed

freeBytes

Cumulative bytes freed

IO Buffer Counters (SrvMonSysMemoryIOBufCounters)

Counters for IO buffer lifecycle operations. IO buffers are used internally for message serialization, packet construction, and data transfer:

Field
Description

bufferAllocCount

Number of IO buffers created via new allocation (not from pool)

bufferAllocBytes

Cumulative bytes from new IO buffer allocations

bufferPoolAllocCount

Number of IO buffers obtained from the pool

bufferPoolAllocBytes

Cumulative bytes obtained from the IO buffer pool

bufferReallocCount

Number of IO buffers that were reallocated due to a size change

bufferReallocBytes

Cumulative bytes from IO buffer reallocations

bufferForkCount

Number of IO buffers that were forked (copy-on-write)

bufferForkBytes

Cumulative bytes from forked IO buffers

bufferGCDisposeCount

Number of IO buffers disposed to the garbage collector

bufferGCDisposeBytes

Cumulative bytes of IO buffers disposed to GC

bufferPoolDisposeCount

Number of IO buffers returned to the pool

bufferPoolDisposeBytes

Cumulative bytes returned to the IO buffer pool

bufferPoolLeakCount

Number of IO buffers evicted from the pool and leaked to GC

bufferPoolLeakBytes

Cumulative bytes leaked from the IO buffer pool

Entity Type Counters (SrvMonSysMemoryEntityTypeCounters)

Entity lifecycle counters are included for each of four categories: messages, state entities, embedded entities, and collections. Each category includes both aggregate counters and optional per-type breakdowns (when nv.memory.stats.type.enable=true).

Field
Description

liveCount

Number of entity instances currently in active use

pooledCount

Number of entity instances currently in pools

createAllocCount

Number of entities created via new allocation

poolAllocCount

Number of entities obtained from a pool

gCDisposeCount

Number of entities disposed to the garbage collector

poolDisposeCount

Number of entities returned to a pool

poolLeakCount

Number of entities evicted from a pool and leaked to GC

pojoLiveCount

POJO wrappers currently in active use

pojoPooledCount

POJO wrappers currently in pools

pojoCreateAllocCount

POJOs created via new allocation

pojoPoolAllocCount

POJOs obtained from a pool

pojoGCDisposeCount

POJOs disposed to GC

pojoPoolDisposeCount

POJOs returned to a pool

pojoPoolLeakCount

POJOs leaked from a pool

serializerLiveCount

Serializers currently in active use

serializerPooledCount

Serializers currently in pools

serializerCreateAllocCount

Serializers created via new allocation

serializerPoolAllocCount

Serializers obtained from a pool

serializerGCDisposeCount

Serializers disposed to GC

serializerPoolDisposeCount

Serializers returned to a pool

serializerPoolLeakCount

Serializers leaked from a pool

deserializerLiveCount

Deserializers currently in active use

deserializerPooledCount

Deserializers currently in pools

deserializerCreateAllocCount

Deserializers created via new allocation

deserializerPoolAllocCount

Deserializers obtained from a pool

deserializerGCDisposeCount

Deserializers disposed to GC

deserializerPoolDisposeCount

Deserializers returned to a pool

deserializerPoolLeakCount

Deserializers leaked from a pool

metadataLiveCount

Metadata buffers currently in active use

metadataLiveBytes

Bytes of metadata buffers currently in active use

metadataAllocCount

Metadata buffers allocated

metadataAllocBytes

Cumulative bytes of metadata allocated

metadataGCDisposeCount

Metadata buffers disposed to GC

metadataGCDisposeBytes

Cumulative bytes of metadata disposed to GC

metadataPoolDisposeCount

Metadata buffers returned to a pool

metadataPoolDisposeBytes

Cumulative bytes of metadata returned to a pool

dataLiveCount

Data buffers currently in active use

dataLiveBytes

Bytes of data buffers currently in active use

dataAllocCount

Data buffers allocated

dataAllocBytes

Cumulative bytes of data allocated

dataGCDisposeCount

Data buffers disposed to GC

dataGCDisposeBytes

Cumulative bytes of data disposed to GC

dataPoolDisposeCount

Data buffers returned to a pool

dataPoolDisposeBytes

Cumulative bytes of data returned to a pool

When per-entity-type breakdown is enabled, each counter set also includes:

Field
Description

entityName

Fully qualified class name of the entity type

entityShortName

Simple class name of the entity type

Memory Stats in XVM Heartbeats

When XVM heartbeats are enabled, memory statistics are automatically included in the system stats section of each heartbeat. The heartbeat trace output shows system and process memory:

In addition to the textual trace, the heartbeat message (SrvMonHeartbeatMessage) carries structured memory statistics that can be consumed programmatically. These are organized into the sub-messages described in the sections above.

Standalone Memory Stats Trace

When standalone trace output is enabled (see Configuring Memory Statistics), memory stats are periodically traced to the nv.memory.stats logger.

Trace Output Format

The standalone trace output includes the following sections:

When per-entity-type tracking is enabled (nv.memory.stats.type.enable=true), each entity category is followed by per-type breakdowns prefixed with ...:

Interpreting the Trace Output

Host

Field
Description

totalPhysicalMemory

Total physical memory on the host

availableMemory

Available physical memory on the host. On Linux, reports MemAvailable (free + reclaimable cache/buffers) rather than raw MemFree

Process

Field
Description

rssBytes

Process RSS (Resident Set Size) — actual physical memory occupied by the process

Heap

Field
Description

init

The initial amount of heap memory the JVM requested from the OS at startup

used

The amount of heap memory currently in use

committed

The amount of heap memory guaranteed to be available to the JVM

max

The maximum amount of heap memory that can be used

Off Heap

Field
Description

usedLive

Total off-heap memory currently in live use. This is the sum of native memory in use (alloc - free + realloc) plus direct IO buffer memory in use (alloc + poolAlloc - poolDispose - gcDispose)

usedPooled

Off-heap memory currently held in IO buffer pools, available for reuse (poolDispose - poolAlloc - poolLeak)

nativeAlloc

Cumulative bytes allocated via native memory allocations

nativeFree

Cumulative bytes freed via native memory frees

nativeRealloc

Cumulative bytes reallocated via native memory reallocations

directAlloc

Cumulative bytes allocated by creating new direct IO buffers (not from pool)

directPoolAlloc

Cumulative bytes obtained from the direct IO buffer pool

directPoolDispose

Cumulative bytes returned to the direct IO buffer pool

directGCDispose

Cumulative bytes of direct IO buffers disposed to the garbage collector

Entity Stats

Each entity category (Message Entities, Embedded Entities, State Entities, State Collections) reports the following fields:

Field
Description

live

Number of entity instances currently in use (createAlloc + poolAlloc - gcDispose - poolDispose)

pooled

Number of entity instances currently held in pools (poolDispose - poolAlloc - poolLeak)

Po

POJO (Plain Old Java Object) wrapper counts — {live, pooled}

Se

Serializer counts — {live, pooled}

De

Deserializer counts — {live, pooled}

Me

Metadata buffer bytes currently in live use (allocBytes - poolDisposeBytes - gcDisposeBytes)

Da

Data buffer bytes currently in live use (allocBytes - poolDisposeBytes - gcDisposeBytes)

Tip: If the live count for messages or entities grows continuously, it may indicate a leak where objects are being allocated but never disposed or returned to the pool. Compare the live and pooled counts across intervals to identify trends.

Last updated