> For the complete documentation index, see [llms.txt](https://docs.xplatform.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.xplatform.com/talon/operating-applications/monitoring/memory-statistics.md). # Memory Statistics ## Overview Talon tracks detailed memory usage statistics covering JVM heap, off-heap (native) memory, IO buffers, and ADM entity lifecycle. These statistics are available in XVM heartbeats and can also be output independently via standalone trace logging. This page describes the statistics that are collected and the format in which they are traced. ## Configuring Memory Statistics Memory stats collection is disabled by default. For complete configuration details including enablement, per-entity-type breakdowns, and standalone trace output, see: * [Configuring Memory Statistics](/talon/developing-applications/configuring-the-runtime/monitoring/memory-statistics.md) - Complete memory statistics configuration guide This page focuses on understanding and interpreting the statistics once they are collected. ## Memory Statistics Categories Memory statistics are collected by the `MemoryStats` singleton and cover four areas: * **Host Memory** — System-level physical memory, swap, and committed virtual memory * **Process Memory** — Physical memory footprint of the process (RSS) * **JVM Memory** — Standard JVM heap and non-heap memory pools (init, used, committed, max) * **App Memory** — Off-heap (native) memory, IO buffer lifecycle, and ADM entity object management ### Host Memory The following system-level memory fields are included in the heartbeat's `SrvMonSysMemoryStats`: | Field | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `totalPhysicalMemorySize` | Total physical memory on the host in bytes | | `freePhysicalMemorySize` | Available physical memory in bytes. On Linux this reports MemAvailable (free + reclaimable cache/buffers) rather than raw MemFree, providing a more accurate measure of usable memory | | `committedVirtualMemorySize` | Virtual memory guaranteed to be available to the process in bytes | | `totalSwapSpaceSize` | Total swap space on the host in bytes | | `freeSwapSpaceSize` | Free swap space in bytes | {% hint style="info" %} **Note**: Prior to 3.16.x, `freePhysicalMemorySize` reported raw MemFree on Linux, which excludes memory used by the kernel for page cache and buffers. The updated behavior reports MemAvailable, which accounts for reclaimable cache and provides a much more accurate picture of whether the host is running low on memory. On a typical Linux server, MemFree might report 500MB while MemAvailable reports 12GB. {% endhint %} ### Process Memory | Field | Description | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `processRssBytes` | The process RSS (Resident Set Size) in bytes — the actual physical memory currently occupied by the process, including code, stack, heap, and memory-mapped regions. Sourced via native JNI calls to platform-specific APIs (`/proc/self/stat` on Linux, `mach_task_basic_info` on macOS). Returns 0 if native runtime is not available. | ### Heap Usage (`SrvMonSysMemoryUsage`) Standard JVM heap memory usage from `MemoryMXBean.getHeapMemoryUsage()`: | Field | Description | | ----------- | ----------------------------------------- | | `init` | Initial heap memory requested from the OS | | `used` | Heap memory currently in use | | `committed` | Heap memory guaranteed to be available | | `max` | Maximum heap memory that can be used | ### Non-Heap Usage (`SrvMonSysMemoryUsage`) JVM non-heap memory usage from `MemoryMXBean.getNonHeapMemoryUsage()`: | Field | Description | | ----------- | ----------------------------------------- | | `init` | Initial non-heap memory (e.g., metaspace) | | `used` | Non-heap memory currently in use | | `committed` | Non-heap memory committed by the JVM | | `max` | Maximum non-heap memory (0 if unbounded) | ### Off-Heap Usage (`SrvMonSysMemoryOffHeapUsage`) Aggregated off-heap memory usage from `MemoryStats`: | Field | Description | | ------------ | -------------------------------------------------------------- | | `usedLive` | Total native + direct IO buffer memory currently in active use | | `usedPooled` | Direct IO buffer memory held in pools available for reuse | ### Native Memory Counters (`SrvMonSysMemoryNativeCounters`) Raw counters for native (non-JVM) memory operations: | Field | Description | | -------------- | ------------------------------------------------- | | `allocCount` | Number of native memory allocations performed | | `allocBytes` | Cumulative bytes allocated via native allocations | | `reallocCount` | Number of native memory reallocations performed | | `reallocBytes` | Cumulative bytes reallocated | | `freeCount` | Number of native memory frees performed | | `freeBytes` | Cumulative bytes freed | ### IO Buffer Counters (`SrvMonSysMemoryIOBufCounters`) Counters for IO buffer lifecycle operations. IO buffers are used internally for message serialization, packet construction, and data transfer: | Field | Description | | ------------------------ | --------------------------------------------------------------- | | `bufferAllocCount` | Number of IO buffers created via new allocation (not from pool) | | `bufferAllocBytes` | Cumulative bytes from new IO buffer allocations | | `bufferPoolAllocCount` | Number of IO buffers obtained from the pool | | `bufferPoolAllocBytes` | Cumulative bytes obtained from the IO buffer pool | | `bufferReallocCount` | Number of IO buffers that were reallocated due to a size change | | `bufferReallocBytes` | Cumulative bytes from IO buffer reallocations | | `bufferForkCount` | Number of IO buffers that were forked (copy-on-write) | | `bufferForkBytes` | Cumulative bytes from forked IO buffers | | `bufferGCDisposeCount` | Number of IO buffers disposed to the garbage collector | | `bufferGCDisposeBytes` | Cumulative bytes of IO buffers disposed to GC | | `bufferPoolDisposeCount` | Number of IO buffers returned to the pool | | `bufferPoolDisposeBytes` | Cumulative bytes returned to the IO buffer pool | | `bufferPoolLeakCount` | Number of IO buffers evicted from the pool and leaked to GC | | `bufferPoolLeakBytes` | Cumulative bytes leaked from the IO buffer pool | ### Entity Type Counters (`SrvMonSysMemoryEntityTypeCounters`) Entity lifecycle counters are included for each of four categories: **messages**, **state entities**, **embedded entities**, and **collections**. Each category includes both aggregate counters and optional per-type breakdowns (when `nv.memory.stats.type.enable=true`). | Field | Description | | ------------------------------ | ------------------------------------------------------- | | `liveCount` | Number of entity instances currently in active use | | `pooledCount` | Number of entity instances currently in pools | | `createAllocCount` | Number of entities created via new allocation | | `poolAllocCount` | Number of entities obtained from a pool | | `gCDisposeCount` | Number of entities disposed to the garbage collector | | `poolDisposeCount` | Number of entities returned to a pool | | `poolLeakCount` | Number of entities evicted from a pool and leaked to GC | | `pojoLiveCount` | POJO wrappers currently in active use | | `pojoPooledCount` | POJO wrappers currently in pools | | `pojoCreateAllocCount` | POJOs created via new allocation | | `pojoPoolAllocCount` | POJOs obtained from a pool | | `pojoGCDisposeCount` | POJOs disposed to GC | | `pojoPoolDisposeCount` | POJOs returned to a pool | | `pojoPoolLeakCount` | POJOs leaked from a pool | | `serializerLiveCount` | Serializers currently in active use | | `serializerPooledCount` | Serializers currently in pools | | `serializerCreateAllocCount` | Serializers created via new allocation | | `serializerPoolAllocCount` | Serializers obtained from a pool | | `serializerGCDisposeCount` | Serializers disposed to GC | | `serializerPoolDisposeCount` | Serializers returned to a pool | | `serializerPoolLeakCount` | Serializers leaked from a pool | | `deserializerLiveCount` | Deserializers currently in active use | | `deserializerPooledCount` | Deserializers currently in pools | | `deserializerCreateAllocCount` | Deserializers created via new allocation | | `deserializerPoolAllocCount` | Deserializers obtained from a pool | | `deserializerGCDisposeCount` | Deserializers disposed to GC | | `deserializerPoolDisposeCount` | Deserializers returned to a pool | | `deserializerPoolLeakCount` | Deserializers leaked from a pool | | `metadataLiveCount` | Metadata buffers currently in active use | | `metadataLiveBytes` | Bytes of metadata buffers currently in active use | | `metadataAllocCount` | Metadata buffers allocated | | `metadataAllocBytes` | Cumulative bytes of metadata allocated | | `metadataGCDisposeCount` | Metadata buffers disposed to GC | | `metadataGCDisposeBytes` | Cumulative bytes of metadata disposed to GC | | `metadataPoolDisposeCount` | Metadata buffers returned to a pool | | `metadataPoolDisposeBytes` | Cumulative bytes of metadata returned to a pool | | `dataLiveCount` | Data buffers currently in active use | | `dataLiveBytes` | Bytes of data buffers currently in active use | | `dataAllocCount` | Data buffers allocated | | `dataAllocBytes` | Cumulative bytes of data allocated | | `dataGCDisposeCount` | Data buffers disposed to GC | | `dataGCDisposeBytes` | Cumulative bytes of data disposed to GC | | `dataPoolDisposeCount` | Data buffers returned to a pool | | `dataPoolDisposeBytes` | Cumulative bytes of data returned to a pool | When per-entity-type breakdown is enabled, each counter set also includes: | Field | Description | | ----------------- | --------------------------------------------- | | `entityName` | Fully qualified class name of the entity type | | `entityShortName` | Simple class name of the entity type | ### Memory Stats in XVM Heartbeats When XVM heartbeats are enabled, memory statistics are automatically included in the system stats section of each heartbeat. The heartbeat trace output shows system and process memory: ``` Memory (system): 94.4G total, 89.8G avail, 5.5G committed (Swap: 96.6G total, 96.6G free) Memory (proc): RSS 1.2G HEAP 1.5G init, 522M used, 1.5G commit, 1.5G max NON-HEAP 2M init, 47M used, 48M commit, 0K max OFF-HEAP 1.6G live, 0.8G pooled ``` In addition to the textual trace, the heartbeat message (`SrvMonHeartbeatMessage`) carries structured memory statistics that can be consumed programmatically. These are organized into the sub-messages described in the sections above. ## Standalone Memory Stats Trace When standalone trace output is enabled (see [Configuring Memory Statistics](/talon/developing-applications/configuring-the-runtime/monitoring/memory-statistics.md)), memory stats are periodically traced to the `nv.memory.stats` logger. ### Trace Output Format The standalone trace output includes the following sections: ``` Host {, } Process {} Heap {, , , } Off Heap {, [, , , , , , ]} Message Entities {, Po {, } Se {, } De {, } Me {} Da {}} Embedded Entities {, Po {, } Se {, } De {, } Me {} Da {}} State Entities {, Po {, } Se {, } De {, } Me {} Da {}} State Collections {, Po {, } Se {, } De {, } Me {} Da {}} ``` When per-entity-type tracking is enabled (`nv.memory.stats.type.enable=true`), each entity category is followed by per-type breakdowns prefixed with `...`: ``` Message Entities {1, 0 Po {1, 0} Se {0, 0} De {0, 0} Me {0} Da {0}} ...com.example.MyMessage {1, 0 Po {1, 0} Se {0, 0} De {0, 0} Me {0} Da {0}} ``` ### Interpreting the Trace Output #### Host | Field | Description | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **totalPhysicalMemory** | Total physical memory on the host | | **availableMemory** | Available physical memory on the host. On Linux, reports MemAvailable (free + reclaimable cache/buffers) rather than raw MemFree | #### Process | Field | Description | | ------------ | -------------------------------------------------------------------------------- | | **rssBytes** | Process RSS (Resident Set Size) — actual physical memory occupied by the process | #### Heap | Field | Description | | ------------- | -------------------------------------------------------------------------- | | **init** | The initial amount of heap memory the JVM requested from the OS at startup | | **used** | The amount of heap memory currently in use | | **committed** | The amount of heap memory guaranteed to be available to the JVM | | **max** | The maximum amount of heap memory that can be used | #### Off Heap | Field | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **usedLive** | Total off-heap memory currently in live use. This is the sum of native memory in use (alloc - free + realloc) plus direct IO buffer memory in use (alloc + poolAlloc - poolDispose - gcDispose) | | **usedPooled** | Off-heap memory currently held in IO buffer pools, available for reuse (poolDispose - poolAlloc - poolLeak) | | **nativeAlloc** | Cumulative bytes allocated via native memory allocations | | **nativeFree** | Cumulative bytes freed via native memory frees | | **nativeRealloc** | Cumulative bytes reallocated via native memory reallocations | | **directAlloc** | Cumulative bytes allocated by creating new direct IO buffers (not from pool) | | **directPoolAlloc** | Cumulative bytes obtained from the direct IO buffer pool | | **directPoolDispose** | Cumulative bytes returned to the direct IO buffer pool | | **directGCDispose** | Cumulative bytes of direct IO buffers disposed to the garbage collector | #### Entity Stats Each entity category (Message Entities, Embedded Entities, State Entities, State Collections) reports the following fields: | Field | Description | | ---------- | ----------------------------------------------------------------------------------------------- | | **live** | Number of entity instances currently in use (createAlloc + poolAlloc - gcDispose - poolDispose) | | **pooled** | Number of entity instances currently held in pools (poolDispose - poolAlloc - poolLeak) | | **Po** | POJO (Plain Old Java Object) wrapper counts — `{live, pooled}` | | **Se** | Serializer counts — `{live, pooled}` | | **De** | Deserializer counts — `{live, pooled}` | | **Me** | Metadata buffer bytes currently in live use (allocBytes - poolDisposeBytes - gcDisposeBytes) | | **Da** | Data buffer bytes currently in live use (allocBytes - poolDisposeBytes - gcDisposeBytes) | {% hint style="info" %} **Tip**: If the **live** count for messages or entities grows continuously, it may indicate a leak where objects are being allocated but never disposed or returned to the pool. Compare the **live** and **pooled** counts across intervals to identify trends. {% endhint %} ## Related Topics * [Configuring Memory Statistics](/talon/developing-applications/configuring-the-runtime/monitoring/memory-statistics.md) - Enable and configure memory stats collection * [XVM Heartbeats](/talon/operating-applications/monitoring/xvm-heartbeats.md) - Interpreting heartbeat output * [Configuring Monitoring](/talon/developing-applications/configuring-the-runtime/monitoring.md) - Heartbeat and statistics configuration * [Trace Logging](/talon/operating-applications/analysis-and-troubleshooting/trace-logging.md) - General trace logging configuration