Detecting Duplicates

The AEP Engine uses sequence numbers from inbound messages to detect and discard duplicate messages before they are dispatched to application handlers. This ensures that applications do not process the same message multiple times.

Overview

When an upstream microservice sends messages with sequence numbers enabled, the receiving AEP Engine can use those sequence numbers to identify duplicates. This is particularly important in scenarios involving:

Message bus retransmissions
Failover and recovery
Network issues that cause duplicate delivery

By default, duplicate detection is enabled. If an AEP Engine receives a message with a sequence number greater than 1 that is less than or equal to a previously received message's sequence number (for the same bus+channel+qos combination), the message is considered a duplicate and is not dispatched to the application.

How It Works

Duplicate detection operates on the inbound message processing pipeline:

Message arrives from message bus
Duplicate Detection - Sequence number is checked against previously received messages
If duplicate: Message is discarded (not dispatched to handlers)
If not duplicate: Message proceeds to filtering and handler dispatch

The duplicate check is performed per bus+channel+qos combination. Sequence numbers from different channels or QoS levels are tracked independently.

Sequence Number Space

Sequence number spaces start at 1. When the receiving AEP Engine receives a message with sequence number 1, it treats this as a sequence number space reset and begins tracking from this new baseline. This prevents false duplicate detection when an upstream service restarts with a fresh sequence number space.

When an upstream service's state is wiped or reinitialized, it restarts its sending stream sequence at 1, which alerts downstream applications not to consider newly lowered sequence numbers as duplicates.

Persistence Across Restarts

The AEP Engine persists the last sequence number received from each upstream sender as part of the microservice's HA state. This means:

Duplicate detection operates across service restarts
The receiving service remembers the last sequence number from each sender
After restart, the service can continue detecting duplicates based on the persisted sequence number state

This persistence ensures that duplicate detection remains effective even after failover or restart scenarios.

Duplicate detection is enabled by default. See Configuring Duplicate Detection to learn how to disable it or customize its behavior.

Monitoring

The AEP Engine tracks duplicate message statistics that can be monitored:

NumDupMsgsRcvd - Total number of duplicate messages received and discarded

This metric is always 0 when performDuplicateChecking=false.

See AEP Engine Statistics for details on monitoring engine metrics.

Prerequisites

For duplicate detection to work:

Upstream sender must enable sequence numbers - See Message Sequencing for how to configure sequence numbers on outbound messages
Message bus binding must preserve sequence numbers - The binding must transport sequence number metadata (enabled by default)