Persistence Module

The Persistence module benchmarks X Platform's persistence mechanisms for messages and data.

Overview

X Platform provides multiple persistence mechanisms:

• Message Logger: Persists individual messages

• Packet Logger: Persists message packets

• Store Logger: Persists transactional state changes

This benchmark measures the throughput and latency of each persistence mechanism.

Test Programs

MessageLogger

Class: com.neeve.perf.persist.MessageLogger

Benchmarks single-message persistence performance.

PacketLogger

Class: com.neeve.perf.persist.PacketLogger

Benchmarks message packet persistence performance.

StoreLogger

Class: com.neeve.perf.persist.StoreLogger

Benchmarks transactional state persistence performance.

Running Benchmarks

Message Logger Test

$JAVA_HOME/bin/java -cp "libs/*" com.neeve.perf.persist.MessageLogger \
  --logDir ./logs \
  --count 1000000 \
  --messageSize 256 \
  --flushOnWrite false

Store Logger Test

$JAVA_HOME/bin/java -cp "libs/*" com.neeve.perf.persist.StoreLogger \
  --logDir ./logs \
  --count 1000000 \
  --flushOnWrite false

Key Parameters

flushOnWrite

Controls durability vs performance tradeoff:

false (default):

• Higher throughput

• Lower latency

• Data buffered in OS cache

• Risk of data loss on system crash

true:

• Guaranteed durability

• Lower throughput

• Higher latency

• All data flushed to physical media

writeBufferSize

Size of write buffer (default: 8192 bytes):

--writeBufferSize 65536  # Larger buffer = better throughput

Interpreting Results

Typical Results (Linux x86-64, NVME SSD)

Operation	flushOnWrite=false	flushOnWrite=true
Message Write	~5-10µs	~100-200µs
Packet Write	~2-5µs	~80-150µs
Store Write	~3-8µs	~90-180µs

Performance Characteristics

1. Buffered Writes (flushOnWrite=false):

   • High throughput (100K-200K writes/sec)

   • Low latency (few microseconds)

   • Suitable when durability can be relaxed

2. Synchronous Writes (flushOnWrite=true):

   • Lower throughput (5K-10K writes/sec)

   • Higher latency (hundreds of microseconds)

   • Guaranteed durability

   • Required for critical data

3. Storage Media Impact:

   • NVME SSD: ~100µs sync write latency

   • SATA SSD: ~200-500µs sync write latency

   • HDD: ~5-10ms sync write latency

Comparison with AEP Module

The AEP Module uses asynchronous persistence:

• Persistence Module: Measures pure persistence overhead

• AEP Module: Persistence runs concurrently with replication

• AEP Impact: Persistence overlaps with network latency

In AEP benchmark, persistence adds minimal latency due to pipelining

Best Practices

For High Throughput

1. Use buffered writes (flushOnWrite=false)

2. Increase write buffer size

3. Use NVME storage

4. Consider packet-based logging for batch operations

For Guaranteed Durability

1. Enable flushOnWrite=true

2. Use NVME or high-performance SSD

3. Consider battery-backed write cache

4. Accept throughput/latency tradeoff

Next Steps

• Review AEP Module to see persistence in end-to-end context

• Review Storage Module for object store benchmarks

• Return to Modules Overview for other modules