Transactional inbox: when the ack deletes the message

An ack to Google Cloud Pub/Sub is not proof of processing, it is a delete command. If a workflow engine sits between receipt and ack, one that can also reject a message on business grounds, then the position of that single call decides whether a missing completion message becomes a visible operational incident or silent data loss. This post is a full tutorial: it builds the typical receiver, dissects its failure modes and sets five variants against it, from nack with a dead-letter topic to the transactional inbox with a relay. All code runs on Spring Boot with Spring Cloud GCP and Flowable.

Contents

• The scene: a completion message that never arrives
• The receiver that looks harmless
• What an ack actually promises
• Why the correlation rejects
• The first mistake: a retry against a business rejection
• The second mistake: the ack in the catch block
• Why the repair builds the next outage
• Variant 1: nack, retry policy and dead-letter topic
• Variant 2: the transactional inbox
• The relay, and the ordering trap
• Technical or business error?
• Variant 3: the engine consumes directly
• Variant 4: an engine that buffers messages
• Variant 5: claim check
• What the inbox does not solve
• The resumable worker
• Signal or message?
• Which variant when
• FAQ
• Conclusion
• Sources

The scene: a completion message that never arrives

A work order system hands work orders over to an executing service provider. A work order comprises several thousand line items. A BPMN process work-order-execution drives the flow: it hands the work order over to the service provider, then waits at an intermediate message catch event for its completion message “work order completed”, and only after that does the work order count as done and get billed.

┌──────────────────────┐   ┌─────────────────────┐   ┌─────────────────────┐
│ Hand over work order │───│ Wait for completion │───│ Create invoice      │
└──────────────────────┘   └─────────────────────┘   └─────────────────────┘
       Send Task             Message Catch Event       Service Task, async

The completion message arrives via Google Cloud Pub/Sub. The executing service provider publishes to a topic, a Spring Boot service named work-order-service holds the subscription and correlates each message into the running process instance via the correlation key workOrderId.

If the completion message never arrives, there is no error, no red alert, no stack trace. The process simply sits at the catch event and waits, the work order stays open, and nothing gets billed. In practice, such a standstill has only ever surfaced when someone asked where a work order went, and then it is not about one line item but several thousand.

The question of this tutorial is therefore: how do you build the path from the Pub/Sub message into the process instance so that a disruption on that path stays visible and no message is lost for good?

The receiver that looks harmless

This is the receiver that just about everyone has written at some point. It uses PubSubTemplate.subscribe from Spring Cloud GCP (starter com.google.cloud:spring-cloud-gcp-starter-pubsub, version 8.1.0 for Spring Boot 4.0 and 4.1, alongside 7.4.10 for Spring Boot 3.5), reads the message, correlates it into the engine and acks.

@Component
public class WorkOrderCompletedReceiver {

    private static final Logger log =
            LoggerFactory.getLogger(WorkOrderCompletedReceiver.class);

    private final PubSubTemplate pubSubTemplate;
    private final WorkOrderProcessService processService;
    private final ObjectMapper objectMapper;

    public WorkOrderCompletedReceiver(PubSubTemplate pubSubTemplate,
                                      WorkOrderProcessService processService,
                                      ObjectMapper objectMapper) {
        this.pubSubTemplate = pubSubTemplate;
        this.processService = processService;
        this.objectMapper = objectMapper;
    }

    @PostConstruct
    void startSubscription() {
        pubSubTemplate.subscribe("work-order-completed-sub", this::process);
    }

    private void process(BasicAcknowledgeablePubsubMessage message) {
        try {
            WorkOrderCompleted event = objectMapper.readValue(
                    message.getPubsubMessage().getData().toByteArray(),
                    WorkOrderCompleted.class);

            processService.workOrderCompleted(event.workOrderId(), Map.of(
                    "completedItems", event.completedItems(),
                    "completionTimestamp", event.completionTimestamp().toString()));

            message.ack();
        } catch (Exception e) {
            log.error("Completion message could not be processed", e);
            message.ack();   // so the subscription does not clog up
        }
    }
}

public record WorkOrderCompleted(String workOrderId,
                                 int completedItems,
                                 Instant completionTimestamp) {
}

The code compiles, runs, passes every demo test and survives months in production. It still has three quiet weaknesses, and each gets its own chapter in this article:

1. The workflow engine sits in the ack path. Any disruption of the engine, even an ordinary deployment, prevents the ack and turns the receiver into a bottleneck.
2. All errors land in the same catch. A dropped database connection and a business rejection of the correlation are treated identically, although they would need opposite reactions.
3. The ack sits in the catch block. The comment next to it sounds reasonable, in reality that line is a delete command for every message that triggered an error.

All three hang on what an ack actually means in Pub/Sub.

What an ack actually promises

An ack to Pub/Sub promises exactly one thing: this message no longer needs to be delivered. Once one subscriber per subscription has acked, Pub/Sub removes the message from storage. Whether your code successfully processed the message beforehand, Pub/Sub neither knows nor checks.

Three clocks govern the behavior:

The ack deadline controls redelivery. If you do not acknowledge a message within the deadline, Pub/Sub delivers it again. The default is 10 seconds, configurable between 10 and 600 seconds. The client library keeps extending the deadline in the background while the message is in flight on your side, bounded by the client’s maxAckExtensionPeriod.

The subscription’s message retention decides the final expiry date. The default is 7 days, configurable between 10 minutes and 31 days. After it elapses, Pub/Sub is free to drop the message, acknowledged or not. If you could not process a message for 31 days, you lose it even without any ack.

And the third clock does not exist at all: an acknowledged message has no way back. Without the “Retain acknowledged messages” option on the subscription or a configured message retention on the topic, an acked message cannot be brought back via seek. The ack in the catch block of the receiver above is therefore final.

The delivery guarantee to go with it: at-least-once is the default for all subscription types. Any message can arrive more than once, even without anyone making a mistake, for example because an ack got lost on the network path. A receiver that cannot tolerate duplicates is built wrong for Pub/Sub. What exactly-once delivery is about is covered in the FAQ, but one thing up front: it does not solve this article’s problem.

Why the correlation rejects

On the engine side, the process waits at a message catch event. In BPMN it looks like this:

<message id="workOrderCompletedMessage" name="WorkOrderCompleted" />

<process id="work-order-execution" name="Work order execution">

    <sendTask id="handOverWorkOrder" name="Hand over work order"
              flowable:async="true" />
    <sequenceFlow sourceRef="handOverWorkOrder" targetRef="waitForCompletion" />

    <intermediateCatchEvent id="waitForCompletion" name="Wait for completion">
        <messageEventDefinition messageRef="workOrderCompletedMessage" />
    </intermediateCatchEvent>
    <sequenceFlow sourceRef="waitForCompletion" targetRef="createInvoice" />

    <serviceTask id="createInvoice" name="Create invoice"
                 flowable:async="true" />
</process>

Three elements, and the middle one is the vulnerable spot. The send task hands the work order over, the catch event waits, the service task creates the invoice. As long as no token sits at the catch event, no receiver exists for the completion message.

Flowable’s core API offers no one-step call that delivers a message by correlation key on its own. The caller builds the query, evaluates the hits, delivers and handles the errors. Queries of this kind are known as correlation queries, and writing one hardcodes an assumption about the process model: here, at most one instance waits per workOrderId. The correlation logic lives in your receiver, not in the engine.

@Service
public class WorkOrderProcessService {

    private final RuntimeService runtimeService;

    public WorkOrderProcessService(RuntimeService runtimeService) {
        this.runtimeService = runtimeService;
    }

    public void workOrderCompleted(String workOrderId, Map<String, Object> variables) {
        List<Execution> waiting = runtimeService.createExecutionQuery()
                .processDefinitionKey("work-order-execution")
                .messageEventSubscriptionName("WorkOrderCompleted")
                .variableValueEquals("workOrderId", workOrderId)
                .list();

        if (waiting.isEmpty()) {
            throw new NoWaitingInstanceException(workOrderId);
        }
        if (waiting.size() > 1) {
            throw new AmbiguousCorrelationException(workOrderId, waiting.size());
        }

        runtimeService.messageEventReceived(
                "WorkOrderCompleted", waiting.get(0).getId(), variables);
    }
}

Behind this sits a rule that is not printed in bold anywhere in BPMN and still governs everything: for any message there must be exactly one registered waiting point. A message addresses one receiver, and the engine has to deliver it to exactly one execution. If there is none, it has nobody. If there are two, it has no right choice, and then it would rather make none at all.

This call can reject in four ways, and none of them is a bug in the engine:

1. No hit, because the instance has not reached the catch event yet. The event subscription that correlation runs against is a row in ACT_RU_EVENT_SUBSCR, and it comes into existence only when the token reaches the catch event. Flowable has no buffering mechanism for BPMN messages, correlation runs exclusively against existing event subscriptions. If the service provider’s completion message overtakes your own process, for example because the hand-over step is still running, the query finds nothing.
2. No hit, because the instance is already through. The message arrived twice, or someone pushed the process forward by hand. From the outside this case cannot be distinguished from the first one, the query returns an empty list in both.
3. More than one hit. If two executions with the same workOrderId wait for the same message, list() returns both, and the caller has to decide what that means. Take singleResult() instead and the decision comes back as a FlowableException with the wording Query return 2 results instead of max 1, minus the information of how many there were. How two waiting executions come about in the first place is shown in the chapter about the repair.
4. The execution exists but no longer holds the subscription. Between query and delivery lies a time window, and if the instance moves on exactly within it, messageEventReceived throws, according to its Javadoc, a FlowableObjectNotFoundException when the execution is missing, or a FlowableException when it has not subscribed to the message.

All four rejections are statements about the state of the process, not about the infrastructure. Only the first one heals by itself if you give it time. The other three persist no matter how often you repeat the same call. This distinction returns in every variant.

The first mistake: a retry against a business rejection

The first reflex against the rejection is a retry in the receiver: if the correlation fails, wait briefly and try again.

private void process(BasicAcknowledgeablePubsubMessage message) throws Exception {
    WorkOrderCompleted event = read(message);

    for (int attempt = 1; attempt <= 30; attempt++) {
        try {
            processService.workOrderCompleted(event.workOrderId(), variables(event));
            message.ack();
            return;
        } catch (NoWaitingInstanceException e) {
            Thread.sleep(10_000);   // the instance is bound to reach the catch event any moment now
        }
    }
}

For rejection number one, where the instance is simply not there yet, this even works: time heals that case. The place of the waiting is still wrong, for two reasons.

The first is head-of-line blocking. The callback occupies a processing thread, and the client library’s flow control limits how many messages may be outstanding at once. If the thread hangs in the loop for work order 4711 for five minutes, the completion messages of all other work orders wait behind it. Head-of-line blocking here is a property of the consumer configuration, not of the broker. Pub/Sub would have delivered the other messages long ago, your receiver just does not accept them.

The second reason is the lease. The ack deadline caps out at 600 seconds, anything beyond that is held solely by the client library, which extends the deadline in the background until its maxAckExtensionPeriod is reached. A retry that fights an error for a hundred minutes ends up working with a message whose lease may long since have lapsed. Pub/Sub has then already redelivered it, possibly to another instance of the work-order-service, and nobody can say anymore whether your own late ack still has any effect.

Against rejections two to four, the retry achieves nothing anyway. No amount of repetition brings back a process instance that is already through, and none makes a duplicate token vanish either. The retry turns a business rejection into an endless loop with waiting time.

The second mistake: the ack in the catch block

After the first retry storm that clogged the subscription, the second repair reliably follows: on error, ack, so things keep moving. That is exactly how the catch block in the receiver from the beginning came about.

Now the two defects have to be separated. Nack on an exception and you have an availability and coupling problem: the subscription backs up as long as the engine is unreachable. But the message still exists, and when the disruption is over, it comes back. Ack on an exception and you turn the same disruption into data loss: Pub/Sub deletes the message, there is no way back without “Retain acknowledged messages” or topic retention, and the work order’s process waits forever.

The treacherous part: this antipattern needs no hand-written catch block. It also arises purely from configuration. If you go the Spring Integration route via the PubSubInboundChannelAdapter with @ServiceActivator instead of PubSubTemplate.subscribe, you pick an AckMode, and its variants behave in fundamentally different ways on error:

The line that arms the data loss looks completely harmless:

@Bean
public PubSubInboundChannelAdapter workOrderAdapter(PubSubTemplate pubSubTemplate,
                                                    MessageChannel workOrderChannel) {
    var adapter = new PubSubInboundChannelAdapter(pubSubTemplate, "work-order-completed-sub");
    adapter.setOutputChannel(workOrderChannel);
    adapter.setAckMode(AckMode.AUTO_ACK);
    adapter.setErrorChannelName("pubsubErrors");   // from here on, errors get acked
    return adapter;
}

With AUTO_ACK and an error handler, the message counts as done as soon as the handler has seen it, even if it does nothing beyond logging. For full control, take MANUAL and fetch the BasicAcknowledgeablePubsubMessage from the GcpPubSubHeaders.ORIGINAL_MESSAGE header. That settles the recommendation before a single line of inbox code is written: an ack belongs exclusively behind the point at which the message is permanently safe.

Why the repair builds the next outage

That leaves the question of where rejection number three comes from, the duplicate token. It arises not from a bug but from a manual repair, and months can lie between cause and effect.

The backstory: a completion message got lost, say through the ack in the catch block. The work order stands still. Someone from operations steps in and, via a change-state operation, places a token at the hand-over step so the work order gets handed over again.

The hand-over step is modeled asynchronously with flowable:async="true", as befits an external call. And exactly that makes the intervention deceptive: the operation merely creates a job in ACT_RU_JOB, it is executed only once the job executor picks it up. Right after the intervention everything therefore looks unchanged. No new call in the logs, no visible progress in the process diagram. The obvious conclusion is: the intervention did not work, try again. Each of these attempts places another token next to the first.

At the moment of its creation, the extra token causes not a single symptom. Both tokens pass through the hand-over, both reach the catch event, both create an event subscription with the same workOrderId, and in the monitor the process looks the way it always does. Only when the next completion message for this work order arrives does singleResult() fail with Query return 2 results instead of max 1. If the receiver with the ack in the catch is still running at that point, this business rejection is logged, the message is deleted, and the work order stands still again. The repair of the last outage has built the next one.

The lesson is uncomfortable: a correlation error can stem from an intervention that lies months in the past. A system that throws such messages away instead of keeping them robs itself of any chance of diagnosis and repair.

Variant 1: nack, retry policy and dead-letter topic

The first viable variant stays entirely within Pub/Sub and needs no new infrastructure: on errors you nack, a retry policy takes over the repetition, and whatever fails permanently moves to a dead-letter topic.

The receiver gets shorter, not longer:

private void process(BasicAcknowledgeablePubsubMessage message) {
    try {
        WorkOrderCompleted event = read(message);
        processService.workOrderCompleted(event.workOrderId(), variables(event));
        message.ack();
    } catch (Exception e) {
        log.warn("Completion message deferred: {}", e.getMessage());
        message.nack();
    }
}

Two configuration details decide whether this ends well, and both are regularly overlooked.

First, the retry policy. Without a configured policy, Pub/Sub’s default is immediate redelivery, without any backoff. A nack on a business rejection then produces a tight loop of delivery, rejection, delivery. With a policy, exponential backoff applies, minimumBackoff default 10 seconds, maximumBackoff default 600 seconds, both settable between 0 and 600 seconds. The policy applies per message, and it kicks in both on nack and on expiry of the ack deadline.

Second, the dead-letter topic. It is configured on the subscription, with maxDeliveryAttempts between 5 and 100, default 5:

gcloud pubsub topics create work-order-completed-dlt

gcloud pubsub subscriptions update work-order-completed-sub 
    --dead-letter-topic=work-order-completed-dlt 
    --max-delivery-attempts=10 
    --min-retry-delay=10s 
    --max-retry-delay=600s

This comes with two IAM bindings without which no forwarding takes place at all: the Pub/Sub service account service-<project-number>@gcp-sa-pubsub.iam.gserviceaccount.com needs roles/pubsub.publisher on the dead-letter topic and roles/pubsub.subscriber on the source subscription.

gcloud pubsub topics add-iam-policy-binding work-order-completed-dlt 
    --member="serviceAccount:service-123456789@gcp-sa-pubsub.iam.gserviceaccount.com" 
    --role="roles/pubsub.publisher"

gcloud pubsub subscriptions add-iam-policy-binding work-order-completed-sub 
    --member="serviceAccount:service-123456789@gcp-sa-pubsub.iam.gserviceaccount.com" 
    --role="roles/pubsub.subscriber"

The promise comes with limits, and they belong on the table: forwarding to the dead-letter topic is best-effort, fewer or more delivery attempts than configured can occur. The forwarded message is wrapped and carries CloudPubSubDeadLetterSource attributes, among them source subscription and delivery counter. The delivery_attempt field, by contrast, is what subscribers of the source subscription see on every delivery. The counter behind it is only maintained when the dead-letter topic is configured correctly, and it can drop back to 0, a case that shows up above all on pull subscriptions whose subscribers go quiet for a while.

Many inbox articles omit this point: for a consumer without its own database, with moderate volume and no ordering requirements, this variant is in many cases the better solution. No new code path, no table, no relay, and the monitoring comes for free, for example via the oldest_unacked_message_age metric and the fill level of the dead-letter topic. What it lacks only shows up against four concrete requirements.

Variant 2: the transactional inbox

The transactional inbox, the receiving-side counterpart to Chris Richardson’s transactional outbox, separates two things that the receiver from the beginning mixed up: receiving the message and processing it. The receiver now only writes the message into a dedicated table and acks. The correlation into the engine is taken over later by a separate relay.

The table:

CREATE TABLE work_order_inbox (
    id            BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    event_type    VARCHAR(64)  NOT NULL,
    work_order_id VARCHAR(36)  NOT NULL,
    payload       JSONB        NOT NULL,
    status        VARCHAR(16)  NOT NULL DEFAULT 'OPEN',
    attempts      INT          NOT NULL DEFAULT 0,
    next_attempt  TIMESTAMPTZ  NOT NULL DEFAULT now(),
    last_error    TEXT,
    received_at   TIMESTAMPTZ  NOT NULL DEFAULT now(),
    processed_at  TIMESTAMPTZ,
    CONSTRAINT uq_work_order_inbox UNIQUE (event_type, work_order_id)
);

The receiver shrinks to receive, store, acknowledge:

private void process(BasicAcknowledgeablePubsubMessage message) {
    try {
        WorkOrderCompleted event = read(message);
        inbox.store("WorkOrderCompleted", event.workOrderId(), rawData(message));
        message.ack();
    } catch (Exception e) {
        log.warn("Completion message could not be stored: {}", e.getMessage());
        message.nack();
    }
}

@Repository
public class WorkOrderInbox {

    private final JdbcTemplate jdbc;

    public WorkOrderInbox(JdbcTemplate jdbc) {
        this.jdbc = jdbc;
    }

    public boolean store(String eventType, String workOrderId, String payload) {
        int inserted = jdbc.update("""
                INSERT INTO work_order_inbox (event_type, work_order_id, payload)
                VALUES (?, ?, ?::jsonb)
                ON CONFLICT ON CONSTRAINT uq_work_order_inbox DO NOTHING
                """, eventType, workOrderId, payload);
        return inserted == 1;
    }
}

Inside this small piece of code sit three decisions.

Between insert and ack remains a crash window. If the pod dies exactly between the two lines, the message is in the table but not acknowledged, and Pub/Sub delivers it again. The insert has to survive this repetition unharmed, which is why it is idempotent via the unique key: the second insert of the same message hits the constraint, does nothing, and the ack still goes out. That is exactly the idempotent consumer from microservices.io, cast into a table.

The choice of the unique key is not trivial. (event_type, work_order_id) only works as long as the service provider sends at most one completion message per work order. If two messages that differ in business terms carry the same key, for example because a work order is completed in two tranches, the unique key swallows the second one without comment, without an error and without a log entry. Then a business event id from the sender has to go into the key, and if the sender does not provide one, that is a conversation with the sender, not an implementation detail.

And yes: the database now sits in the ack path itself. That does not contradict the criticism of the receiver from the beginning, it resolves it. In front of the ack goes what can fail for purely technical reasons, because a technical failure is healed by Pub/Sub’s redelivery. An INSERT into your own table cannot reject on business grounds, the correlation into the engine can. That is why the insert belongs before the ack and the correlation behind it.

The relay, and the ordering trap

The second building block is the relay: a poller that reads open inbox entries and correlates them into the engine. The obvious form looks like this:

@Component
public class WorkOrderInboxRelay {

    private final JdbcTemplate jdbc;
    private final WorkOrderProcessService processService;

    public WorkOrderInboxRelay(JdbcTemplate jdbc, WorkOrderProcessService processService) {
        this.jdbc = jdbc;
        this.processService = processService;
    }

    @Scheduled(fixedDelay = 2000)
    @Transactional
    public void processOpenEntries() {
        List<WorkOrderInboxEntry> entries = jdbc.query("""
                SELECT * FROM work_order_inbox
                WHERE status = 'OPEN' AND next_attempt <= now()
                ORDER BY id
                LIMIT 10
                FOR UPDATE SKIP LOCKED
                """, entryMapper());

        for (WorkOrderInboxEntry entry : entries) {
            process(entry);
        }
    }
}

FOR UPDATE SKIP LOCKED makes the relay horizontally scalable: several pods of the work-order-service poll at the same time, each locks its rows, none waits for another. That is the competing consumers pattern from Hohpe and Woolf, transferred to a table.

And exactly therein lies the trap. As soon as the inbox transports more than one event type per work order, say “execution started” for the status history and “work order completed” for the correlation, two pods can pull the two messages of the same work order at the same time. Then “completed” may get correlated before “started” is processed, the process is already past the second event, and the correlation of “started” rejects. That is exactly the race from rejection number one, rebuilt one floor down, in your own infrastructure instead of in the broker.

The solution is: serialize per correlation key, stay parallel across keys. In SQL this can be expressed directly: an entry may only be pulled if no older unfinished entry exists for the same work order.

SELECT i.*
FROM work_order_inbox i
WHERE i.status = 'OPEN'
  AND i.next_attempt <= now()
  AND NOT EXISTS (
      SELECT 1
      FROM work_order_inbox older
      WHERE older.work_order_id = i.work_order_id
        AND older.status IN ('OPEN', 'IN_PROGRESS')
        AND older.id < i.id)
ORDER BY i.id
LIMIT 10
FOR UPDATE SKIP LOCKED;

If pod A has locked the older row of a work order, pod B still sees it as unfinished and leaves the younger one alone. In the edge case an entry waits one poll cycle longer than necessary, which errs in the safe direction. If you want to skip the construct, there is a simpler option with a clear price: exactly one relay instance, strict ORDER BY id, and throughput is capped at this one worker. For a handful of work orders per hour that is perfectly fine.

Technical or business error?

In the relay, the decision now falls that the original receiver never made: what does an error mean, and what follows from it? The common classification from distributed systems separates transient errors, which are retried with exponential backoff, from permanent ones, which go straight to the final stop. For the correlation this means concretely:

private void process(WorkOrderInboxEntry entry) {
    try {
        processService.workOrderCompleted(entry.workOrderId(), entry.variables());
        markCompleted(entry);
    } catch (NoWaitingInstanceException e) {
        // may heal with time: backoff, but with a deadline
        if (entry.attempts() >= maxAttempts) {
            park(entry, e);
        } else {
            rescheduleWithBackoff(entry, e);
        }
    } catch (AmbiguousCorrelationException e) {
        // two waiting points: no retry in the world helps
        park(entry, e);
    } catch (FlowableException e) {
        // subscription vanished between query and delivery
        park(entry, e);
    } catch (Exception e) {
        // technical: database, engine unreachable, timeout
        rescheduleWithBackoff(entry, e);
    }
}

The mapping follows from the chapter on the four rejections. No hit can mean the instance is still on its way, so patience pays, but not indefinitely: once a deadline has passed, “not there yet” has turned into “never was there or is already through”, and the entry gets parked. A multiple hit and a vanished subscription are immediately a case for humans, every automatic retry would only produce the same rejection again. And everything technical gets backoff, because there re-execution actually heals.

One fuzzy edge remains, and you should know it: FlowableException is not a purely business exception. The engine also wraps technical errors in it, for example when the database underneath it breaks away. To separate that case cleanly, there is no way around inspecting the cause instead of relying on the exception type. The two exceptions from your own service, NoWaitingInstanceException and AmbiguousCorrelationException, are dedicated types for exactly this reason and not passed-through engine errors.

Parked means: status = 'PARKED', the last error sits in the row, and an alert fires. Operations needs at least two alerts, one on parked entries and one on the age of the oldest open entry. Both are simple SQL queries, and that you have to build and run them yourself is not a detail but a real cost of this architecture. The chapter on the limits of the inbox comes back to this.

Variant 3: the engine consumes directly

If the correlation logic is tightly attached to the engine anyway, an obvious question follows: why not let the engine consume the broker itself? Some engines ship an event integration for this, where a broker subscription is bound declaratively to the process model and the hand-written receiver disappears entirely.

The appeal is real: one layer less, no correlation logic of your own, the mapping from message to correlation key sits in the model instead of in Java code. For simple cases this is the leanest solution.

Three things deserve a sober look here. First, the correlation rejection does not disappear, it only relocates: an engine adapter also runs into instances that are not yet or no longer waiting, and how it then deals with ack, retry and final stop is now determined by the adapter implementation instead of your code. The questions from this article have to be asked of the adapter just the same, only someone else answers them there. Second, the broker consumer moves into the engine’s lifecycle: a restart of the engine is now automatically a restart of the consumer as well, and the coupling we wanted out of the ack path returns in another form. And third, the integration is a matter of what ships with the engine: which broker is supported depends on engine and version, and for Google Cloud Pub/Sub it comes down, depending on the stack, to a self-maintained adapter, at which point the code you supposedly saved is back.

As a rule of thumb: this variant plays to its strength when the adapter for your broker exists, its error handling is documented and acceptable, and you have no requirements of your own for deduplication or repair. Otherwise the explicit route via receiver or inbox remains the more controllable choice.

Variant 4: an engine that buffers messages

The four rejections from the correlation chapter share one root: Flowable correlates exclusively against event subscriptions that exist at the moment of the call. There are engines built differently at this point: published messages are buffered in the engine’s broker, with a time-to-live whose parameter is called timeToLive and is given in milliseconds. If the TTL is 0, buffering is off. While the TTL runs, a buffered message still correlates even if the matching subscription only comes into existence after publishing. In one widespread implementation the client sets one hour by default, plus there is an optional messageId for idempotency.

Transfer this to our scenario and rejection number one disappears completely: the completion message may overtake the process, it simply waits in the engine until the token arrives at the catch event. The race between hand-over and completion message, which the retry in the receiver and the deadline in the relay were built against, simply does not exist in such a model.

You have to stay honest about the remaining rejections. An instance that is already through is not brought back by a buffer either, once the TTL has expired. A duplicate token remains a duplicate token. If you use the separate synchronous correlation call of such engines, you get no buffering on top, it applies only to the publishing route. And for signals it does not apply at all. The buffer clears away one of the four rejections, the other three remain.

For everyone on Flowable the consequence is simpler: the buffering others have in the broker is, here, the inbox table. Same concept, only self-operated and in return accessible via SQL.

Variant 5: claim check

Claim check comes from the Enterprise Integration Patterns by Hohpe and Woolf. The pattern actually solves a different problem: large messages. The payload moves into a data store, only the reference flows through the broker, and the receiver fetches the data when processing.

In our scenario there would even be a real occasion for it: the complete completion report of a work order with several thousand line items has no business being inside a Pub/Sub message. The service provider puts the report into a bucket, the message carries workOrderId and the reference, and the correlation works with the slim message only.

To the correlation problem, though, claim check contributes only indirectly, and that contribution should not be oversold. The indirect contribution: the business data sits permanently in the store, independent of ack, retention and dead-letter storage. Even if the message is lost for good, the store still shows which work orders were completed, and re-running the correlation by hand or by script is possible. That is a safety net under the safety net, but it correlates nothing on its own: the engine’s rejections, the crash window in the receiver and the ordering in the relay remain exactly as they were.

What the inbox does not solve

A chapter of its own for the limits, so they do not vanish into the fine print.

The inbox does not heal the cause. The duplicate token from the repair chapter remains a duplicate token, the correlation keeps rejecting its completion message, whether it comes from the receiver or from the relay. What the inbox changes is the outcome: a deleted piece of evidence becomes a parked row with payload and error text. The failure becomes survivable and repairable, not impossible. The repair itself is then deliberately unspectacular:

UPDATE work_order_inbox
SET status = 'OPEN', attempts = 0, next_attempt = now()
WHERE id = 4711;

First remove the duplicate token in the engine, then this one line, and the relay does the rest. No re-publish tooling, no copying out of the dead-letter topic.

In return you take on operating costs, and the accusation that regularly comes up is justified: you are building a queue in front of the queue. The backlog moves out of a subscription that comes with finished metrics like oldest_unacked_message_age and well-rehearsed alerting, into a table that at first nobody monitors. Add the growth and cleanup of completed rows, a second retry mechanism next to Pub/Sub’s retry policy, and a second final stop next to the dead-letter topic. Each of these parts has to be built, tested and understood by the team.

When is it still worth it? The comparison with variant 1 can be boiled down to four requirements. The retention clock: it keeps ticking in the dead-letter topic too, after 31 days at the latest the message is gone there as well, while inbox rows never expire. Deduplication: the inbox’s unique key makes the consumer idempotent, Pub/Sub alone does not. Repair via SQL instead of re-publish tooling, as shown above. And an existing local transaction into which the insert belongs atomically, for example when the receipt writes business data anyway. If none of the four applies, nack with retry policy and dead-letter topic is the simpler and therefore better choice.

The resumable worker

The story does not end with the correlation, because behind the catch event the process continues, and the same laws apply there. The step “create invoice” is modeled with flowable:async="true", and what that means in a failure is worth walking through once in detail.

flowable:async="true" creates a job in ACT_RU_JOB. When an async executor picks the job up, it writes lock owner and lock expiration into the row. If the pod dies mid-execution, the lock stands until it expires, then the reset thread releases it, and another executor runs the job. And it runs it from the very beginning: what the dead pod already did, nobody knows. That is at-least-once, the same semantics as with Pub/Sub, just one level up, and the consequence is the same: invoicing has to be idempotent, otherwise the same work order gets billed twice in the retry case.

The defaults, all configurable:

Two of these numbers are traps. Three retries are used up quickly when the invoicing system is stuck for ten minutes, after that the job sits in the deadletter table and needs a human again. And a five-minute lock means: after a pod death the step stands still for up to five minutes plus the reset interval before anyone takes it over.

If you want invoicing out of the engine, take the external worker: the process puts a job onto a topic, an external worker fetches it with a time-limited exclusive claim, a lease.

String workerId = "billing-worker-1";

List<AcquiredExternalWorkerJob> jobs = managementService
        .createExternalWorkerJobAcquireBuilder()
        .topic("create-invoice", Duration.ofMinutes(10L))
        .acquireAndLock(20, workerId);

for (AcquiredExternalWorkerJob job : jobs) {
    try {
        createInvoice(job);
        managementService
                .createExternalWorkerCompletionBuilder(job.getId(), workerId)
                .complete();
    } catch (Exception e) {
        managementService
                .createExternalWorkerJobFailureBuilder(job.getId(), workerId)
                .errorMessage(e.getMessage())
                .fail();
    }
}

If processing fails, the worker reports the failure instead of complete(), and the job is handed out again. An extension of a running lease is not documented in the Java external client, so the lock duration in the topic() call should sit generously above the expected processing time: if the lease expires mid-work, a second worker can take over the same job, and the idempotency has to carry that case too.

Signal or message?

When modeling the completion message, the question regularly comes up whether a signal would not be simpler than the message, precisely because signals throw no error when no receiver exists. The answer for this scenario is unambiguous.

In BPMN 2.0 a signal has only a name and no addressee, a message has a sender and a receiver. A signal acts globally: it reaches every running instance that is currently waiting for it, and none of them is named in the signal. signalEventReceived(String signalName) accordingly notifies all executions with an active signal subscription. If not a single one exists, nothing happens: no exception, no buffering, the signal fizzles out. On top of that, a signal throw is synchronous by default: the throwing instance blocks until every catcher has taken delivery.

Exactly the property that makes the signal look convenient disqualifies it for the completion message. “Work order completed” for work order 4711 is a statement to exactly one process instance, and if that instance is not waiting, that is information the sender or operations must learn about. The message delivers that information as a rejection, which the chapters above deal with. The signal does not deliver it: it reports the same state as success without effect, and the loud loss becomes a silent one again. A signal fits where genuinely many instances care about the same news, say a tariff change at the service provider that affects all running work orders. An addressed completion message with a correlation key is a message, in every model shown here. What happens if you go for the broadcast anyway, and how to make the damage visible in a test, is covered in BPMN signal or message.

Which variant when

The decision table, with the receiver from the beginning as the yardstick of what to avoid:

Across all variants lie two rules that always hold. First: ack only once the message is permanently safe, and before the ack only steps that can fail exclusively for technical reasons. Second: business rejections of the correlation need a final stop and an alert, not a retry.

FAQ

Why is my Pub/Sub message gone although the code threw an exception?
Because somewhere an ack happened anyway. The usual two routes: an ack() in the catch block, or AckMode.AUTO_ACK combined with an error handler, because that combination acknowledges the message even on error. Without “Retain acknowledged messages” on the subscription or message retention on the topic, an acked message is not recoverable.

Does Pub/Sub’s exactly-once delivery solve the problem?
No. Exactly-once delivery exists only for pull subscriptions, it holds only within one region, and it guarantees exactly-once delivery, not exactly-once processing. Publisher-side duplicates also remain possible. The business rejection of the correlation, the core problem of this article, it does not touch at all.

Why does Flowable not find my process instance although it is running?
Because correlation happens before the instance has reached the catch event. The event subscription only comes into existence when the token arrives at the catch event, and Flowable has no buffering mechanism for BPMN messages. A completion message that overtakes your own process therefore finds no subscription and has to be retried later.

Do ordering keys help against the ordering trap in the relay?
Only on the broker side, and with a price tag: per ordering key, 1 MBps of throughput is possible, and one redelivered message pulls every later message of its key back with it, already acknowledged ones included. As soon as a parallel relay pulls from a table, the serialization per correlation key has to happen there anyway, and the subscription’s ordering keys no longer help then.

Do I still need the inbox if I already have a dead-letter topic?
Often not. The dead-letter topic already covers the case “permanently failed, human required”. The inbox only wins when one of four requirements counts: retention beyond the subscription’s limit, because the dead-letter topic is also capped at 31 days, deduplication via a unique key, repair via SQL, or an atomic insert into an existing local transaction.

How do I prevent the inbox table from growing without bounds?
With a cleanup job of your own, because nobody else provides one: delete or archive completed rows after a retention period, explicitly exempting parked rows. This comes with two alerts, one on parked entries and one on the age of the oldest open entry, because the finished Pub/Sub metrics do not see the table.

What happens to a message that was not processed anywhere for 31 days?
It is gone. A subscription’s message retention is at most 31 days, and once that has run out Pub/Sub is free to drop the message no matter its acknowledgment state. That also holds for the subscription on a dead-letter topic. If you have to retain longer, you need storage of your own, such as the inbox or a claim check store.

Conclusion

In front of the ack goes only what can fail for purely technical reasons. A technical failure is healed by the broker’s redelivery, a business rejection of the correlation is healed by no retry in the world, and that is why the workflow engine belongs behind the ack, not in front of it.

On the way there, two separate defects have to be kept apart. The engine in the ack path is an availability problem: every deployment backs up the subscription, but the messages survive. The ack on error, whether as a catch block or as AUTO_ACK with an error handler, turns that into data loss, and with Pub/Sub, without precautions, that loss is final.

For the solution, this order applies: check nack with retry policy and dead-letter topic first, because for many consumers that is the better choice, with metrics and alerting for free. The transactional inbox enters the picture when retention, deduplication, SQL repair or a local transaction counts, and it comes with tangible operating costs: your own monitoring, your own cleanup, your own final stop, and the ordering trap in the parallel relay, which has to be serialized per correlation key. And whichever variant you land on: none of them makes the causes of business rejections disappear, from the overtaking race to the duplicate token from the manual repair. Visible, retained and repairable: a delivery architecture cannot achieve more than that.

Sources

All code examples in this article are my own.

• Google Cloud Pub/Sub, Subscription properties: ack deadline, message retention, dead-letter topic, docs.cloud.google.com

• Google Cloud Pub/Sub, Retry policy, docs.cloud.google.com

• Google Cloud Pub/Sub, Dead-letter topics, docs.cloud.google.com

• Google Cloud Pub/Sub, Exactly-once delivery, docs.cloud.google.com

• Google Cloud Pub/Sub, Ordering keys, docs.cloud.google.com

• gcloud pubsub subscriptions update, flag reference, docs.cloud.google.com

• Spring Cloud GCP Reference, Pub/Sub Support, googlecloudplatform.github.io

• Spring Cloud GCP Reference, Spring Integration channel adapters and AckMode, googlecloudplatform.github.io

• Flowable open source documentation, BPMN 2.0 Constructs: Message Events and Signal Events, flowable.com

• Flowable Javadoc, RuntimeService and ManagementService, developer-docs.flowable.com

• Chris Richardson, Pattern: Idempotent Consumer and Transactional Outbox, microservices.io

• Gregor Hohpe, Bobby Woolf: Enterprise Integration Patterns, Claim Check and Competing Consumers, enterpriseintegrationpatterns.com

As of August 2026, checked against Spring Cloud GCP 8.1 and Flowable 8.0.