Unhandled Messages

From Talon's point of view, an unhandled message is a message that is received but not processed by a message handler. This section discusses how such receipt is handled.

Unhandled messages fall into 3 categories:

1. SMA Undeserializable Messages - Messages that are received from an underlying SMA MessageBusBinding that cannot be deserialized into a message.

2. SMA Unsolicited Messages - Messages that can be successfully deserialized into a message but are on a MessageChannel that is not joined or not known to the receiving microservice.

3. AEP Unhandled Messages - Valid messages intentionally attracted to the microservice on a joined MessageChannel, but for which the microservice has no declared EventHandler.

The first two situations are trapped by SMA and are reported via an SMA UnhandledMessageEvent, the latter situation is trapped by the AepEngine and reported to the microservice as an AepUnhandledMessageEvent.

SMA Triggered Unhandled Message Events

Received messages that are not deserializable or unsolicited are trapped by SMA and reported to the AepEngine via an SMA UnhandledMessageEvent which the AepEngine dispatches to the appropriate EventHandler, if registered.

Causes

Below are some typical causes for SMA UnhandledMessageEvent events.

• Receiving a message on a channel that has not been joined: this usually indicates a configuration or design error in topics or subscriptions. The intent behind the SMA's message channel abstraction is to define named logical conduits between peers. When a message is received on an unjoined channel, it means that messages are being received by the microservice on a channel that the application designer wasn't expecting to receive traffic on.

• Receiving a message on an unknown channel: This usually indicates a problem with topic subscriptions in which two channels are overlapping and causing a microservice to attract messages from a sender on another channel.

• Receiving a message that has no channel or metadata: Messages sent to X microservice are typically expected to have SMA MessageMetadata that can be used identify the message's transmission channel, type and factory. Receipt of a message without metadata for a bus that expects will thus result in an error.

• Deserialization Errors: The SMA contract is to pass the message in deserialized MessageView (POJO) form to the AepEngine. If the message's view factory cannot be found or the data is corrupted, it will result in an error.

UnhandledMessageEvent

The following fields are available on the UnhandledMessageEvent.

Field	Description
BackingMessage	When running in a Talon XVM, this will contain a SrvMonUnhandledMessageMessage. This is a monitoring alert message that contains this event in a SMA serializable form which allows the microservice to serialize the contents of this event for auditing and administrative purposes.
MessageBusBinding	The message bus binding where UnhandledMessageEvent originated, or null if not available.
MessageKey	The unhandled message's key (e.g. bus destination) on which the message was received (if available and the source binding supports transport of the key).
MessageMetadata	The unhandled message's metadata. See the MessageMetadata javadoc for a description of these fields.
MessageSMATransportMessageId	The unhandled message's SMA transport specific message id (if available). The SMA transport specific message ID may be supplied by bindings for which there is a notion of a unique identifier for a message. A null value means that either the binding doesn't support the notion of such an id or that the id wasn't available.
SerializedPayloadBlob	The serialized payload blob representing the received serialized message that was unhandled. The serialized payload is provided with the UnhandledMessageEvent to allow handlers to quarantine (save) the message for subsequent handling by administrators or by tools which allow the unhandled message to be acknowledged upstream. Note, however, that UnhandledMessageEvents are emitted in exceptional cases, and even if the serialized payload is provided, there is no guarantee that it was not corrupted during receipt. This field is not guaranteed to be available. Reasons why it may not be available include: The payload content was missing from SMA transport level message (poison/corrupt message). The SMA binding doesn't serialize messages, and consequently can provide a serialized form. There was a bug in the binding around handling the message. The encoding type of these bytes is determined by the corresponding metadata, so it is also generally necessary to ensure that that metadata is persisted along with the serialized payload to allow this serialized form to be deserialized.
SerializedMetadataBlob	The serialized metadata blob representing the metadata as received in serialized form. The serialized metadata is provided with the UnhandledMessageEvent to allow handlers to quarantine (save) the message for subsequent diagnostic handling in cases where received metadata was corrupted on the wire or during receipt in the binding. This field is not guaranteed to be available. Reasons why it may not be available include: The payload content was missing from SMA transport level message (poison/corrupt message). The SMA binding doesn't serialize messages, and consequently can provide a serialized form. There was a bug in the binding around handling the message. The encoding type of these bytes is determined by the corresponding metadata, so it is also generally necessary to ensure that the metadata is persisted along with the serialized payload to allow this serialized form to be deserialized.
UnwrappedMessage	The unwrapped (un-deserialized) message. This is the raw transported form of the message which is dependent on the source binding. For example, for a JMS binding this could be a javax.jms.Message, or a buffer when using the native Solace binding.
Reason	A Throwable object describing the reason why the message was unhandled.

Acknowledgement

For unhandled Guaranteed messages, determining whether or not to acknowledge unhandled message is an important consideration.

If the message is not acknowledged:

• The message will be redelivered if the microservice is restarted or fails over to a backup, which can be undesirable.

• If there are a large number of unacknowledged messages, it puts a burden on the messaging provider, which could hit resource limits.

• If the SMA provider doesn't support individual message acknowledgements, then failing to acknowledge the unhandled message can block subsequent acknowledgements.

But acknowledging the message:

• Is dangerous because the message will be discarded without being processed by the microservice.

Prior to the 3.2 release, whether or not the triggering message was acknowledged was left to the SMA bus binding, which in most cases would not acknowledge the message.

As of the 3.2 release, a microservice that declares an EventHandler for the UnhandledMessageEvent can control whether or not the triggering message is acknowledged. By default, the platform leans towards ensuring that the message is not discarded and will not acknowledge the message unless instructed to do so by the microservice. A microservice can control acknowledgement as discussed below.

Default Acknowledgement Behavior

Auto acknowledgement for unhandled messages can be globally set to enabled by setting the configuration environment property.

nv.sma.unhandledmessageevent.autoack=true

As of 3.2, setting the above property will cause UnhandledMessageEvents to be acknowledged regardless of whether or not there is a handler for the messages, unless a handler specifically calls setAutoAck(false) on the event. Acknowledgement is done after the event is dispatched in the Aep Engine transaction thread (i.e., the event multiplexer thread).

In the 3.1.3 patch release, the Solace binding was patched, allowing this property to control whether the receiving thread will acknowledge the message. When set, the message may be acknowledged prior to the event being dispatched to the application by the aep engine transaction thread

Prior to version 3.4.373 this property was named nv.sma.unhandledmessageevent.autoAck. This property was changed to be all lowercase for uniformity with other environment property names. The old camelcase property name is still supported for backwards compatibility, but the newer property name takes precendence. It is therefore important for applications that may specify and override this property from multiple locations (e.g. system property and ddl) use the same case everywhere.

Auto Acknowledgement

A microservice can explicitly override the default acknowledgement behavior of unhandled messages by declaring an EventHandler for UnhandledMessageEvent and setting the event's autoAck behavior.

@EventHandler
public void onUnhandledMessage(UnhandledMessageEvent event) {
  byte [] toQuarantine = event.getBackingMessage().serializeToByteArray();
  // ... quarantine the above message in some fashion.
   
  // Mark the event for auto acknowledgement, the 
  // engine will acknowledge it.
  event.setAutoAck(true);
}

Explicit Acknowledgement

t is also possible to acknowledge the event asynchronously outside of the event handler. This advanced usage is useful for a microservice that might perform a blocking operation processing the UnhandledMessageEvent.

@EventHandler
public void onUnhandledMessage(UnhandledMessageEvent event) {
  // Disable auto acknowledgement:
  event.setAutoAck(false);
  
  // Acquire the event so the platform doesn't return it
  // to a pool:
  event.acquire();
 
  // Asynchronously handle the event:
  Thread thread = new Thread(new Runnable() {
    public void run() {
      byte [] toQuarantine = event.getBackingMessage().serializeToByteArray();
      // ... quarantine the above message in some fashion that might block.
      try {
        event.acknowledge();
      }
      catch (SmaException e) {
        e.printStackTrace();
      }
      finally {
        event.dispose();
      }
    }
  }
}

AEP Triggered Unhandled Message Events

An AepUnhandledMessageEvent is emitted when a message is successfully dispatched to an AepEngine from an SMA binding, but no application EventHandler is found by the AepEngine.

Causes

• The application is incorrectly configured to join a Message Channel that it shouldn't have joined.

• The application's channel filter is not narrow enough and some message types are coming though that shouldn't be.

• A publisher is publishing messages to the channel that it shouldn't be publishing.

An AepUnhandledMessageEvent doesn't necessarily indicate a serious problem, but the presence of such events at the very least indicates inefficiencies in the type of messages that are being attracted by the applications subscriptions.

Acknowledgement

AEP unhandled messages are acknowledged like a normally handled message, except they result in an empty transaction with external effects. The message is acknowledged when the transaction is stabilized. For a microservice using EventSourcing that is using a disk based persister, such messages will end up in the microservice's recovery transaction log, and, consequently, will not be lost in the event of acknowledgement. For a StateReplication microservice or an EventSourcing microservice without a Store persister, an application should ensure that the message is appropriately quarantined prior to returning from the AepUnhandledMessageEvent event handler and is acknowledged.

Some Additional Notes:

• Because AepUnhandledMessageEvents are part of a servicd's transaction pipeline, Talon doesn't provide the capability for asynchronous acknowledgement.

• For StateReplication microservices, AEP unhandled messages will be logged in the inbound message log, but this log is not synced to disk as part of transaction commit. This means that applications should not rely on it for quarantining purposes.