Review a transactional outbox relay

Review exercise · TypeScript

Advanced45 min

Review brief

Business transactions already write outbox rows atomically. Multiple relay instances now claim and publish those rows to a broker whose send and acknowledgement are separate phases. Review stable event identity, durable state ordering, competing ownership, failure fairness, and shutdown recovery.

Behavioral contract

Each outbox row ID is the stable business-event ID and must be used unchanged for every broker attempt. Consumers deduplicate allowed at-least-once deliveries by that ID.
`startPublish` resolves when sending has begun; only `acknowledgement` confirms broker acceptance. Mark a row published only after that acknowledgement succeeds.
`claimBatch` atomically skips active claims across relay instances. A relay must retain or renew each owner-scoped claim until it durably publishes the acknowledged row or records and releases a retryable failure.
An unacknowledged or ambiguous send remains unpublished and eventually retryable; duplicate delivery is preferable to losing the event.
A row failure increments its durable `failureCount`, schedules that row after backoff, and does not prevent later rows in the claimed batch from being attempted.
Shutdown stops new claims and allows up to ten seconds for in-flight publication to finish. On timeout, the row must remain unpublished and its claim must be released for retry before exit.
Operations telemetry must include the oldest pending-row age and visibility into repeated failure counts, in addition to throughput counters.

Review instructions

1Review the changed relay against the acknowledgement, identity, ownership, fairness, and shutdown contract.
2Trace each failure point across both the broker and durable outbox state, including competition from another relay.
3Prioritize no more than seven findings, including operational and test feedback only when it protects stated behavior.

Learning objectives

• Place durable transitions after external acknowledgement
• Preserve stable identity across retries
• Analyze competition, poison events, and shutdown

src/outbox-relay-types.tsTypeScript

@@ -0,0 +1,40 @@

1+export interface OutboxRow {

2+ id: string;

3+ topic: string;

4+ payload: string;

5+ createdAt: Date;

6+ failureCount: number;

7+}

9+export interface BrokerDelivery {

10+ acknowledgement: Promise<void>;

11+}

12+

13+export interface EventBroker {

14+ startPublish(message: {

15+ eventId: string;

16+ topic: string;

17+ payload: string;

18+ }): Promise<BrokerDelivery>;

19+}

20+

21+export interface OutboxRepository {

22+ claimBatch(ownerId: string, limit: number): Promise<OutboxRow[]>;

23+ releaseClaim(rowId: string, ownerId: string): Promise<void>;

24+ markPublished(rowId: string): Promise<void>;

25+ recordFailure(rowId: string, message: string): Promise<void>;

26+}

27+

28+export interface RelayTelemetry {

29+ claimed(count: number): void;

30+ published(count: number): void;

31+ failed(count: number): void;

32+}

33+

34+export interface OutboxRelayDependencies {

35+ broker: EventBroker;

36+ ids: { next(): string };

37+ ownerId: string;

38+ repository: OutboxRepository;

39+ telemetry: RelayTelemetry;

40+}

Reference solution

The findings below are ordered by priority.

Findings

1. Rows are marked published before the broker acknowledges them

Location: src/outbox-relay.ts:50-62

markPublished runs after publication starts but before delivery.acknowledgement resolves. If the broker rejects the delivery or the process exits while waiting, the database says the event is complete and no relay will retry it.

Keep the row pending and claimed until acknowledgement succeeds, then mark it published durably. An ambiguous or failed acknowledgement must leave the row eligible for another at-least-once attempt.

2. Every retry generates a different event identity

Location: src/outbox-relay.ts:51-55

The broker message uses ids.next() rather than the outbox row's stable ID. Any retry is therefore indistinguishable from a new business event, so an idempotent consumer cannot collapse an allowed duplicate delivery.

Use row.id as the broker eventId on every attempt. Attempt identifiers may be separate telemetry, but they must not replace the stable event identity.

3. Claims are released while publication is still in progress

Location: src/outbox-relay.ts:57-62

The relay releases ownership before either the published transition or broker acknowledgement. A second relay can claim the still-pending row in that window and publish it concurrently, even though the first relay remains responsible for the delivery.

Retain and, when necessary, renew the owner-scoped claim through acknowledgement and the final database transition. Release it only after completion or after recording a retryable failure.

4. One poison event aborts the batch and starves later rows

Location: src/outbox-relay.ts:25-37,50-55

If startPublish rejects for a malformed or repeatedly rejected event, the catch increments one counter and rethrows. It never records the row failure or releases it for a delayed retry, and the loop never reaches later rows that were already claimed in the same batch.

Handle failures per row: increment its durable failure state, make it eligible after backoff, and continue with the rest of the batch. Repeatedly failing rows must not prevent later events from eventual delivery.

5. Shutdown abandons in-flight and not-yet-started claims

Location: src/outbox-relay.ts:14-38,46-48

shutdown only flips a flag. A claimBatch that resolves during shutdown, or a batch claimed just before it, reaches the loop's stop check and leaves every not-yet-started row claimed. Shutdown also does not await inFlight, honor the ten-second deadline, or recover the current claim if publication cannot finish.

Coordinate shutdown with pending claim acquisition, release every already-claimed row whose send has not started, and wait up to ten seconds for the current acknowledgement and durable transition. On timeout, preserve the current row as unpublished and release its owner-scoped claim for retry.

6. Telemetry omits pending age and repeated failures

Location: src/outbox-relay-types.ts:1-6,28-32 and src/outbox-relay-metrics.ts:3-32

The telemetry surface exposes only aggregate claimed, published, and failed counters. It never reports the age of the oldest pending row or the stored failureCount, so operators cannot distinguish normal throughput from a stuck or repeatedly failing backlog.

Add an oldest-pending-age gauge and a repeated-failure measure derived from durable rows, while retaining the useful aggregate counters.

7. The test uses one relay and an immediate acknowledgement

Location: src/outbox-relay.test.ts:7-39

The only broker acknowledgement is already resolved, and there is one relay and one successful row. This cannot expose the pre-ack state transition, changing retry identity, competing claims, poison-row fairness, or shutdown during an in-flight send.

Add delayed and rejected acknowledgements, two coordinated relay instances, repeat attempts for one row, a poison row followed by a valid row, and shutdown at each publication phase.

Reasonable non-findings

At-least-once delivery may produce duplicates after ambiguous failures. Stable event identity lets consumers handle those duplicates safely.
A batch size of 50 and sequential publication are acceptable; fairness on individual failure is the required behavior.
Aggregate claimed, published, and failed counters remain useful. They are incomplete rather than inherently wrong.
Atomic claim acquisition is supplied by claimBatch under the contract; the defect is releasing a valid claim too early.