Thread Reference

Complete reference of all threads created by Talon microservices, including configuration examples for affinitizing critical threads.

Prerequisites: Before diving into configuration, review the Threading Model page to understand the architectural concepts and design rationale behind Talon's threading architecture.

Thread Reference

Talon Microservice Threads

Thread	Name	Critical Path	Description
AEP Engine Input Multiplexer	X-STEMux-<appName>-<instanceid>	Yes	The engine thread that dequeues and dispatches microservice messages and events. This is the main microservice thread on which microservice events are dispatched. They are suffixed with a global counter to allow differentiating between stats emitted by multiple instances of the same microservice running in the same JVM. The detached threads described below can offload work from this thread which can improve throughput and latencies in your microservice. Note: When running in an XVM you will see AEP engine threads created for the XVM's admin microservice which is used to handle XVM admin requests.
	X-EventMultiplexer-Wakeup-<appName>	No	A timer thread used to wake up and dispatch events scheduled via the engine's input queue.
Detached Inbound Message Logger	X-ODS-StoreLog-<appName>.in	No	When the microservice is configured with a detached inbound message logger, this thread offloads the work of writing to disk from the engine's input multiplexer which can serve as a buffer against disk I/O spikes. As inbound message loggers aren't used for HA purposes they are not on the critical path. Tip: If your microservice's inbound message load is not high, a detached inbound message logger may not be needed. The tleg3 transaction latency statistic covers inbound message logging. High values or spikes are an indicator that a detached inbound message logger can help.
Detached Outbound Message Logger	X-ODS-StoreLog-<appName>.out	No	When the microservice is configured with a detached outbound message logger, this thread offloads the work of writing to disk from the engine's input multiplexer which can serve as a buffer against disk I/O spikes. Tip: If your microservice's outbound message load is not high, a detached outbound message logger may not be needed. The tleg3 transaction latency statistic covers outbound message logging. High values or spikes are an indicator that a detached outbound message logger can help.
Per Transaction Stats Logger	X-ODS-StoreLog-<appName>.txnstats	No	When the microservice is configured with a detached per transaction stats logger, this thread offloads the work of writing to disk from the engine multiplexer which can serve as a buffer against disk I/O spikes. Tip: If your microservice's transaction load is not high, a detached per transaction stats logger may not be needed. The cepilo (commit epilogue) transaction latency statistic covers per transaction stats logging costs. High values or spikes in cepilo are an indicator that a detached per transaction stats logger can help.
Detached Bus Send Thread	X-AEP-BusManager-IO-<appName>.<busName>	Yes	When the bus is configured for detached send, this thread offloads the work of serialization and writing of outbound messages from the engine's input multiplexer which serves as a buffer against spikes caused by message bus flow control. Tip: High values in the o2p, s, s2w, ws message bus binding stats are indicators that a detached bus sender can improve performance.
Bus Binding Opener	X-AEP-BusManager-BindingOpener-<appName>.<busName>	No	Each bus configured for your microservice is managed by a Bus Manager internal to the AEP engine. Handles establishment of the bus connections and reconnects.
Store Reader Thread	X-ODS-StoreReplicatorLinkReader-<storeName>-<memberName>	Yes	The IO thread for the store which is used to read replication traffic from cluster peers.
Detached Store Persister	X-ODS-StoreLog-<storeName>-<instanceid>	No	When the store is configured for detached persistence, this thread offloads the work of writing recovery data to disk from the engine's input multiplexer which can serve as a buffer against disk I/O spikes. Tip: High values in the pers (persistence) store binding stats are an indicator that a detached store sender can improve performance.
Detached ICR Sender	X-ODS-Store-ICR-Sender-<storeName>-<instanceid>	No	When the store is configured for detached inter-cluster replication, this thread offloads the work of writing recovery data to the receiver from the engine's input multiplexer which can serve as a buffer against disk I/O spikes. Tip: High values in the icr (inter-cluster replication) store binding stats are an indicator that a detached store sender can improve performance.
Detached Store Send Thread	X-ODS-StoreReplicatorSender-<storeName>-<memberName>	Yes	When the store is configured for detached send, this thread offloads the work of writing recovery data to the network for backup instances from the engine's input multiplexer which can serve as a buffer against network I/O spikes. Tip: High values in the s2w (serialize to wire) store binding stats are an indicator that a detached store sender can improve performance.
Detached Store Dispatch Thread	X-ODS-StoreReplicatorDispatcher-<storeName>-<memberName>	Yes	When the store is configured for detached dispatch, this thread allows the store reader thread to offload work of dispatching deserialized replication traffic to the engine for processing. This is useful in cases where the cost of deserializing replication traffic is high. Tip: A high value for the store binding deserialize stat (d) can indicate that setting this property could improve throughput or latency.
Store Acceptor Thread	X-ODS-StoreLinkAcceptor-<instanceid>	No	Each store configured for clustering will create a thread that will listen for connection requests from other store members. Once the connection is established it is handed off to the store reader thread for processing.
Stats Printer Threads	X-Stats-Printer [<statName>-<instanceid>.stats]	No	Several components can be configured to trace stats (independently) from XVM collected stats. When such stats threads are enabled a thread is created to periodically print stats. This is typically useful if a microservice is run outside of an XVM.
Scheduler	X-Scheduler-<instance-count>	No	A timer thread used for scheduling events. An AepEngine uses this to perform periodic engine health checks, for example.

Discovery Threads

Thread	Name	Critical Path	Description
Discovery Timer	X-EDP-Timer	No	Each discovery provider that is opened will create a timer thread that will periodically wake up to perform discovery broadcasts. Each discovery provider typically will create additional threads specific to the discovery provider type. For example, when using an SMA-based discovery provider, message bus binding threads will be created.

XVM Threads

Thread	Name	Critical Path	Description
XVM main thread	X-Server-<xvmName>-Main	No	This thread creates and starts microservices at startup and upon completion will drive XVM acceptors for accepting admin and direct connections to the XVM.
XVM stats collector	X-Server-<xvmName>-StatsRunner	No	Collects stats for microservices, and populates them into heartbeats that can be traced, logged, dispatched, and emitted.
XVM dedicated IO thread	X-Server-<xvmName>-IOThread-<threadNumber>	Yes	When the XVM is configured for multithreading, additional IO threads beyond the XVM main thread are created that service connections that are affinitized to it. Note: When using the direct binding the IO thread is on the critical path for received messages.

---

Critical Thread Affinity Configuration Reference

Threads that can/should be affinitized include the following:

Engine Input Multiplexer

The engine thread that dequeues and dispatches microservice messages and events. This is the main microservice thread.

The detached threads described below can offload work from this thread which can improve throughput and latencies in your microservice.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <!-- Engine Input Multiplexer -->
    <inboundEventMultiplexing>
      <queueWaitStrategy>BusySpin</queueWaitStrategy>
      <queueDrainerCpuAffinityMask>[2]</queueDrainerCpuAffinityMask>
    </inboundEventMultiplexing>
  </app>
</apps>

Bus Detached Sender Thread

Each bus configured for your microservice can optionally be configured to send committed outbound messages on a detached thread. When the bus is configured for detached send, this thread offloads the work of serialization and writing of outbound messages from the engine's input multiplexer which serves as a buffer against spikes caused by message bus flow control.

Tip: High values in the o2p, s, s2w, ws message bus binding stats are indicators that a detached bus sender can improve performance.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <!-- Bus Detached Sender Thread -->
    <messaging>
      <buses>
        <bus name="orderprocessing-bus" enabled="true">
          <detachedSend enabled="true">
            <queueWaitStrategy>BusySpin</queueWaitStrategy>
            <queueDrainerCpuAffinityMask>[1]</queueDrainerCpuAffinityMask>
          </detachedSend>
        </bus>
      </buses>
    </messaging>
  </app>
</apps>

Store Reader Thread

The IO thread for the store. On a primary instance, this is the thread that dispatches store acknowledgements back into the engine. On a backup, this is the thread that dispatches received replication traffic.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <storage enabled="true">
      <clustering>
        <linkReaderCpuAffinityMask>[4]</linkReaderCpuAffinityMask>
      </clustering>
    </storage>
  </app>
</apps>

Store Detached Send Thread

When the store is configured for detached send, this thread offloads the work of writing recovery data to the network for backup instances from the engine's input multiplexer which can serve as a buffer against network I/O spikes.

Tip: High values in the s2w (Serialize To Wire) store binding stats are an indicator that a detached store sender can improve performance.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <storage enabled="true">
      <clustering>
        <detachedSend enabled="true">
          <queueWaitStrategy>BusySpin</queueWaitStrategy>
          <queueDrainerCpuAffinityMask>[5]</queueDrainerCpuAffinityMask>
        </detachedSend>
      </clustering>
    </storage>
  </app>
</apps>

Store Detached Dispatch Thread

When the store is configured for detached dispatch, this thread allows the store reader thread to offload work of dispatching deserialized replication traffic to the engine for processing. This is useful in cases where the cost of deserializing replication traffic is high.

Tip: A high value for the store binding deserialize stat (d) can indicate that setting this property could improve throughput or latency.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <storage enabled="true">
      <clustering>
        <detachedDispatch enabled="true">
          <queueWaitStrategy>BusySpin</queueWaitStrategy>
          <queueDrainerCpuAffinityMask>[5]</queueDrainerCpuAffinityMask>
        </detachedDispatch>
      </clustering>
    </storage>
  </app>
</apps>

Store Detached Persister

When the store is configured for detached persistence, this thread offloads the work of writing recovery data to disk from the engine's input multiplexer which can serve as a buffer against disk I/O spikes.

Tip: High values in the pers (Persistence) store binding stats are an indicator that a detached store sender can improve performance.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <storage enabled="true">
      <clustering>
        <persistence enabled="true">
          <detachedPersist enabled="true">
            <queueWaitStrategy>BusySpin</queueWaitStrategy>
            <queueDrainerCpuAffinityMask>[5]</queueDrainerCpuAffinityMask>
          </detachedPersist>
        </persistence>
      </clustering>
    </storage>
  </app>
</apps>

Store Detached ICR

When the store is configured for detached Inter Cluster Replication, this thread offloads the work of writing recovery data to the ICR bus to insulate the engine's input multiplexer from spikes caused by flow control on the ICR bus.

Tip: High values in the icr (Inter Cluster Replication) store binding stats are an indicator that a detached store ICR sender can improve performance.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <storage enabled="true">
      <clustering>
        <icr enabled="true">
          <detachedSend enabled="true">
            <queueWaitStrategy>BusySpin</queueWaitStrategy>
            <queueDrainerCpuAffinityMask>[5]</queueDrainerCpuAffinityMask>
          </detachedSend>
        </icr>
      </clustering>
    </storage>
  </app>
</apps>

Detached Inbound Message Logger

When the microservice is configured with a detached inbound message logger, this thread offloads the work of writing to disk from the engine's input multiplexer which can serve as a buffer against disk I/O spikes.

Tip: If your microservice's inbound message load is not high, a detached inbound message logger may not be needed. The tleg3 transaction latency statistic covers inbound message logging. High values or spikes are an indicator that a detached inbound message logger can help.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <!-- Inbound message logging -->
    <inboundMessageLogging policy="UseDedicated">
      <detachedWrite enabled="true">
        <queueWaitStrategy>BusySpin</queueWaitStrategy>
        <queueDrainerCpuAffinityMask>[7]</queueDrainerCpuAffinityMask>
      </detachedWrite>
    </inboundMessageLogging>
  </app>
</apps>

Detached Outbound Message Logger

When the microservice is configured with a detached outbound message logger, this thread offloads the work of writing to disk from the engine's input multiplexer which can serve as a buffer against disk I/O spikes.

Tip: If your microservice's outbound message load is not high, a detached outbound message logger may not be needed. The tleg3 transaction latency statistic covers outbound message logging. High values or spikes are an indicator that a detached inbound message logger can help.

<apps>
  <app name="orderprocessing-app" mainClass="com.acme.MyApp">
    <!-- Outbound message logging -->
    <outboundMessageLogging policy="UseDedicated">
      <detachedWrite enabled="true">
        <queueWaitStrategy>BusySpin</queueWaitStrategy>
        <queueDrainerCpuAffinityMask>[6]</queueDrainerCpuAffinityMask>
      </detachedWrite>
    </outboundMessageLogging>
  </app>
</apps>

Bus-Specific Threads

In addition to the core threads above, some bus bindings also support additional threads which may be affinitized.

Solace Binding

Property	Property Name	Description
Detached Dispatch Thread	dispatcher_cpu_affinity_mask	When set to true a detached dispatch thread is created that offloads the work of deserializing messages and dispatching from the solace thread reading messages from the wire. Tip: For microservices in which deserialization and dispatch cost is high, enabling detached dispatch can improve throughput and decrease latency.
Solace Consumer Session Dispatch Thread	consumer_cpu_affinity_mask	This property attempts to affinitize the Solace client library's receiver thread for this connection's consumer session. Tip: For highly active sessions the Solace thread can take up a fair amount of CPU and it is on the critical latency path. Note: Because this thread is not under the platform's control the thread is affinitized the first time it calls into the binding.
Solace Producer Session Dispatch Thread	producer_cpu_affinity_mask	This property attempts to affinitize the Solace client library's receiver thread for this connection's producer session. Tip: Acknowledgements come in on the producer session. This thread is not on the critical latency path, so it doesn't always warrant a core of its own. Note: Because this thread is not under the platform's control the thread is affinitized the first time it calls into the binding.
Detached Send Thread	detached_sends_cpu_affinity_mask	Configures the CPU affinity mask for detached sender thread. Tip: Enabling the detached send thread is usually not needed for Talon microservices because they can use the engine's bus detached send thread described in the previous section. This property may be useful for microservices using the Solace binding outside of a Talon microservice.

Example of configuring affinities for Solace bindings:

<buses>
  <bus name="orderprocessing-bus">
    <provider>solace</provider>
    <address>192.168.1.10</address>
    <port>55555</port>
    <properties>
      <detached_sends>true</detached_sends>
      <detached_sends_cpu_affinity_mask>[1]</detached_sends_cpu_affinity_mask>

      <detached_dispatch>true</detached_dispatch>
      <dispatcher_cpu_affinity_mask>[2]</dispatcher_cpu_affinity_mask>

      <single_session>false</single_session>
      <producer_cpu_affinity_mask>0</producer_cpu_affinity_mask>
      <consumer_cpu_affinity_mask>[9]</consumer_cpu_affinity_mask>
    </properties>
  </bus>
</buses>

Direct Binding

The direct binding allows microservices to connect directly to the microservice via the XVM. When using the direct binding it is possible to set up a dedicated IOThread for accepting and servicing direct connections. The IOThread can be affinitized using the affinity attribute:

<buses>
  <bus name="orderprocessing-bus">
    <provider>direct</provider>
    <address>orderprocessing-app</address>

    <!--
      Not a cpu affinity mask, this property instructs clients
      to use a particular io thread configured in the xvm. The
      cpu affinity for the io thread is configured below.
    -->
    <threadaffinity>1</threadaffinity>
  </bus>
</buses>

<servers>
  <server name="orderprocessing-vm" group="default" template="default">
    <acceptors>
      <acceptor descriptor="${ORDERPROCESSING_ACCEPTOR::tcp://0.0.0.0:0}" />
    </acceptors>
    <multiThreading enabled="true">
      <ioThreads>
        <ioThread id="0" affinity="0" enabled="true" />

        <!--
          The affinity property below configures the cpu affinity
          mask for the IO thread handling connections made to it.
        -->
        <ioThread id="1" affinity="[1]" enabled="true" />
      </ioThreads>
    </multiThreading>
  </server>
</servers>

Related Topics

• Threading Model - Architectural concepts and design rationale

• Disruptors - Configure LMAX disruptor ring buffers

• Thread Affinitization - Pin threads to CPU cores for optimal performance

• DDL Reference - Complete DDL syntax reference

Next Steps

1. Identify which threads are in your microservice's critical path

2. Use thread statistics to determine which threads are busy-spinning

3. Review Thread Affinitization for approaches to pinning threads

4. Configure affinity masks for critical threads