Quarkus and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Quarkus’s SmallRye Fault Tolerance and Mutiny both make retry easy to configure — @Retry is a one-line annotation and Uni.onFailure().retry().atMost(3) is a single reactive pipeline stage — but where the idempotency key is computed relative to the retry boundary determines whether a transient Stripe error produces a safe retry or a duplicate charge. Three Quarkus-specific failure modes: SmallRye Fault Tolerance’s @Retry interceptor re-invokes the CDI method body in full on each retry attempt — UUID.randomUUID() at method entry fires on every invocation; Mutiny’s Uni.onFailure().retry() re-subscribes the upstream Uni pipeline — any computation inside a Uni supplier or flatMap lambda is deferred and re-evaluated on each subscription; and Quarkus @Scheduled (without Quartz clustering) runs the annotated method on every Kubernetes replica independently — three pods pass a concurrent database check simultaneously, each generate distinct UUID.randomUUID() values per customer, and each create ch_A, ch_B, and ch_C per billing period.

This post covers all three failure modes with Quarkus 3.x and Java code, content-hash idempotency keys stable across CDI interceptor re-invocations, Mutiny re-subscriptions, and multi-pod concurrent billing loops, Quarkus Quartz with quarkus.quartz.clustered=true for native @Scheduled serialization, pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex, pg_try_advisory_lock() as a backstop for cases the Quartz lock does not cover — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For AOP retry interception in the Spring ecosystem, see the Micronaut and Stripe Integration post. For Kotlin coroutine retry failure modes, see the Ktor and Stripe Integration post. For reactive stream retry scoping in Spring, see the Spring WebFlux and Stripe Integration post.

Failure mode 1: SmallRye Fault Tolerance @Retry re-invokes the CDI method body — UUID.randomUUID() at method entry fires on every retry attempt — the initial attempt created ch_A before the StripeException — the first retry creates ch_B

MicroProfile Fault Tolerance’s @Retry annotation, implemented by SmallRye Fault Tolerance in Quarkus, works via a CDI interceptor that wraps the annotated method. When the method throws a retryable exception, the interceptor catches it, applies the configured backoff delay, and calls context.proceed() to re-execute the method body for the next attempt. context.proceed() calls the method from its beginning — not from a suspended point inside it, not from a captured continuation — the full method body executes on every retry, including any UUID.randomUUID() call at method entry:

// BillingService.java
// UNSAFE: UUID computed inside the @Retry-annotated CDI method body
// SmallRye's @Retry interceptor calls context.proceed() to re-invoke this method body
// on each retry — the method executes from the top on every attempt

import io.smallrye.faulttolerance.api.ExponentialBackoff;
import org.eclipse.microprofile.faulttolerance.Retry;
import com.stripe.model.Charge;
import com.stripe.net.RequestOptions;
import com.stripe.param.ChargeCreateParams;
import com.stripe.exception.StripeException;

import jakarta.enterprise.context.ApplicationScoped;
import java.time.temporal.ChronoUnit;
import java.util.UUID;

@ApplicationScoped
public class BillingService {

    @Retry(maxRetries = 3,
           delay = 1000,
           delayUnit = ChronoUnit.MILLIS,
           retryOn = {StripeException.class})
    @ExponentialBackoff(factor = 2, maxDelay = 10_000, maxDelayUnit = ChronoUnit.MILLIS)
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents)
            throws StripeException {
        // UNSAFE: UUID computed at method entry.
        // SmallRye's @Retry interceptor re-invokes this entire method body on each retry.
        // Attempt 0 (initial call): UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Attempt 1 (@Retry re-invocation): UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        // Attempt 2 (@Retry re-invocation): UUID = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"
        String idempotencyKey = UUID.randomUUID().toString();

        RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build();

        ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(amountCents)
                .setCurrency("usd")
                .setCustomer(customerId)
                .setDescription("Billing period " + billingPeriod)
                .build();

        return Charge.create(params, options);
    }
}

The failure scenario: the agent calls billingService.chargeCustomer("cust_123", "2026-08", 9900L) on the CDI proxy. SmallRye’s interceptor intercepts the call and invokes context.proceed() to run the method body for attempt 0. The method body executes from the top: UUID.randomUUID() returns "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e". The RequestOptions object is built with this UUID as the idempotency key. Charge.create(params, options) sends POST /v1/charges with Idempotency-Key: 3f7a9b2c... to Stripe.

Stripe receives the request. The card is authorized and the charge object ch_A is created and committed to Stripe’s ledger. Before the HTTP response is assembled and returned, Stripe’s infrastructure encounters a transient overload and returns an ApiException with HTTP 503. The Stripe Java SDK converts this to a StripeException. SmallRye’s interceptor catches the StripeException, checks that it matches retryOn = {StripeException.class}, applies the 1-second backoff delay, and calls context.proceed() again for attempt 1.

context.proceed() runs the method body from the top. UUID.randomUUID() evaluates again — this is a fresh call to UUID.randomUUID(), independent of any prior call. It returns "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d". The retry request sends POST /v1/charges with Idempotency-Key: b8d2e4f6.... Stripe has never seen this key. ch_A was committed against "3f7a9b2c...", not "b8d2e4f6...". Stripe processes the retry as a fresh charge request. ch_B is created. Customer 123 is charged $99 twice for August 2026.

The root of the failure is the mismatch between how @Retry works (re-invocation of the method body, not continuation from a saved state) and where UUID.randomUUID() appears (inside the method body, not outside it). The fix requires the idempotency key to be computed before @Retry can re-execute the code that uses it.

The subtler variant: @Retry combined with @Timeout — @Timeout fires before Stripe responds but ch_A is already committed — @Retry re-invokes the method body with a new UUID — ch_B created

A common addition to @Retry is @Timeout — another SmallRye Fault Tolerance annotation that interrupts the method after N milliseconds and throws a TimeoutException. Developers add @Timeout to prevent Stripe slow responses from blocking the calling thread for too long. The interaction between @Timeout and @Retry is the source of a subtler duplicate-charge failure:

// UNSAFE: @Timeout interrupts the Stripe call after 5 seconds
// @Retry is configured to retry on TimeoutException
// If Stripe has already committed ch_A before the timeout fires,
// the retry re-invokes the method body with a new UUID → ch_B

import org.eclipse.microprofile.faulttolerance.Retry;
import org.eclipse.microprofile.faulttolerance.Timeout;

@ApplicationScoped
public class BillingService {

    // Annotation order: SmallRye applies @Timeout inside @Retry.
    // @Retry wraps @Timeout: if @Timeout fires a TimeoutException,
    // @Retry catches it (TimeoutException extends FaultToleranceException,
    // which @Retry includes by default) and re-invokes the method body.
    @Retry(maxRetries = 2, delay = 2000, delayUnit = ChronoUnit.MILLIS)
    @Timeout(5000)  // 5-second timeout per attempt
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents)
            throws StripeException {
        // UNSAFE: UUID at method entry — same re-invocation failure as above,
        // but now triggered by @Timeout rather than a Stripe 503.
        // Timeline:
        // t=0:   Attempt 0 starts. UUID = "3f7a9b2c..."
        //        POST /v1/charges with Idempotency-Key: 3f7a9b2c...
        // t=3s:  Stripe completes charge processing: ch_A committed to ledger.
        //        Stripe is preparing the HTTP response.
        // t=5s:  @Timeout fires. TimeoutException thrown.
        //        ch_A exists in Stripe's ledger, but the client never received confirmation.
        // t=7s:  After 2s delay, @Retry re-invokes method body (attempt 1).
        //        UUID.randomUUID() = "b8d2e4f6..." (different UUID)
        //        POST /v1/charges with Idempotency-Key: b8d2e4f6...
        //        Stripe sees new key, creates ch_B. Customer charged twice.
        String idempotencyKey = UUID.randomUUID().toString();  // re-evaluated on every @Retry attempt

        RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build();

        ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(amountCents)
                .setCurrency("usd")
                .setCustomer(customerId)
                .setDescription("Billing period " + billingPeriod)
                .build();

        return Charge.create(params, options);
    }
}

The @Timeout annotation is not aware of Stripe’s processing state. When the 5-second timeout fires, SmallRye interrupts the thread or cancels the async operation. From the client’s perspective, the call failed. From Stripe’s perspective, the charge was committed before the response was flushed to the client. ch_A exists in Stripe’s ledger regardless of whether the client received the HTTP 200. The @Retry interceptor sees a TimeoutException, applies its backoff, and re-invokes the method body. UUID.randomUUID() evaluates fresh. ch_B created. The @Timeout annotation did exactly what it was designed to do — it cut off a slow Stripe call — and that correct behavior, combined with a new UUID on retry, is what produces the duplicate charge.

The fix for failure mode 1

The idempotency key must be computed before the @Retry interceptor can re-invoke any code that depends on it. This means computing the key in the caller, outside the @Retry-annotated method, and passing it as an explicit parameter into the method. The @Retry-annotated method receives the stable key as a parameter; on every retry invocation, the parameter value is the same object reference from the caller’s stack frame, not a fresh computation:

// Safe: stable key computed by the CALLER before invoking the @Retry method
// The @Retry method receives idempotencyKey as a parameter — it does not re-compute it

@ApplicationScoped
public class BillingOrchestrator {

    @Inject
    BillingService billingService;

    @Inject
    BillingRepository billingRepository;

    public Charge executeMonthlyBilling(String customerId, String billingPeriod, long amountCents)
            throws StripeException {
        // Safe: computed once by the caller, before any @Retry invocation.
        // The stable hash is the same value on every @Retry attempt within this call.
        String idempotencyKey = stableKey(customerId, billingPeriod);

        // Pre-flight: claim the billing slot before any Stripe call.
        // ON CONFLICT DO NOTHING means only the first caller reaches Stripe.
        boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
        if (!claimed) {
            // Another thread already claimed this slot — return existing result or wait
            return billingRepository.findChargeIdByCustomerAndPeriod(customerId, billingPeriod)
                    .map(chargeId -> Charge.retrieve(chargeId))
                    .orElseThrow(() -> new IllegalStateException("Billing in progress by another thread"));
        }

        return billingService.chargeCustomer(customerId, billingPeriod, amountCents, idempotencyKey);
    }

    private static String stableKey(String customerId, String billingPeriod) {
        try {
            var digest = java.security.MessageDigest.getInstance("SHA-256");
            var hash = digest.digest(
                    (customerId + ":" + billingPeriod + ":quarkus-billing")
                            .getBytes(java.nio.charset.StandardCharsets.UTF_8));
            var sb = new StringBuilder(32);
            for (int i = 0; i < 16; i++) sb.append(String.format("%02x", hash[i]));
            return sb.toString();
        } catch (java.security.NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
    }
}

@ApplicationScoped
public class BillingService {

    @Retry(maxRetries = 3, delay = 1000, delayUnit = ChronoUnit.MILLIS,
           retryOn = {StripeException.class})
    @Timeout(8000)
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents,
                                  String idempotencyKey)  // ← key as parameter, not re-computed
            throws StripeException {
        // Safe: idempotencyKey is a parameter received from the caller.
        // The @Retry interceptor re-invokes this method body on each retry,
        // but the parameter value is the stable hash computed by BillingOrchestrator —
        // the same 32-character hex string on every invocation within the same billing call.
        RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build();

        ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(amountCents)
                .setCurrency("usd")
                .setCustomer(customerId)
                .setDescription("Billing period " + billingPeriod)
                .build();

        Charge charge = Charge.create(params, options);

        // Mark the billing slot as completed with the returned charge ID.
        // If @Retry fires and Stripe returns the cached ch_A, this is called again
        // with ch_A's ID — the UPDATE is idempotent (same charge_id, same row).
        billingRepository.markComplete(customerId, billingPeriod, charge.getId());
        return charge;
    }
}

// What to EXCLUDE from stableKey — any value that re-evaluates differently
// between @Retry invocations (i.e., on each call to context.proceed()):
//   UUID.randomUUID()              ← random, different on every invocation
//   System.currentTimeMillis()     ← different on each retry attempt
//   System.nanoTime()              ← monotonic but different per JVM start and per call
//   attempt counter                ← intentionally increases per retry; never put in key
//   Thread.currentThread().getId() ← SmallRye may use different thread per retry attempt
//   System.identityHashCode(this)  ← same CDI proxy instance, but not useful as a key part

// What IS safe in stableKey:
//   customerId                     ← stable business identifier
//   billingPeriod                  ← stable business identifier (e.g., "2026-08")
//   a vendor namespace string      ← literal constant ("quarkus-billing")
// → sha256("cust_123:2026-08:quarkus-billing")[:32] = same 32-char hex on every @Retry attempt

With the stable key computed in the caller and passed as a parameter, every @Retry invocation of chargeCustomer() sends the same Idempotency-Key header to Stripe. After the initial attempt creates ch_A and a transient StripeException is thrown, the retry sends the identical key. Stripe’s idempotency cache looks up the key, finds the completed charge, and returns the cached ch_A response without creating a new charge. The pre-flight billing_records table provides a database-level guard that survives Stripe’s 24-hour idempotency window.

Failure mode 2: Mutiny Uni.onFailure().retry() re-subscribes the upstream Uni pipeline — UUID.randomUUID() inside a Uni supplier lambda is deferred — each re-subscription computes a new UUID — ch_B on the first retry

Mutiny’s retry mechanism works differently from @Retry’s CDI interception. When Uni.onFailure().retry().atMost(N) is applied to a Uni pipeline, Mutiny re-subscribes the upstream Uni on each failure. Subscription — not construction — is when a Uni’s work actually executes. A Uni built from a supplier (Uni.createFrom().item(supplier)), a callable (Uni.createFrom().callable(callable)), or a pipeline of .flatMap() / .chain() lambdas is lazy: the lambdas execute at subscription time, not when the Uni object is constructed. Each re-subscription by retry() is a fresh subscription, and every lambda in the upstream pipeline executes again:

// BillingService.java
// UNSAFE: UUID.randomUUID() inside Uni.createFrom().item(supplier)
// The supplier executes at subscription time — re-evaluated on each re-subscription.
// Mutiny's .onFailure().retry().atMost(3) re-subscribes the upstream Uni on each failure,
// so the supplier runs again on every retry attempt, producing a new UUID each time.

import io.smallrye.mutiny.Uni;
import io.smallrye.mutiny.infrastructure.Infrastructure;
import com.stripe.model.Charge;
import com.stripe.net.RequestOptions;
import com.stripe.param.ChargeCreateParams;
import com.stripe.exception.StripeException;

import jakarta.enterprise.context.ApplicationScoped;
import java.time.Duration;
import java.util.UUID;

@ApplicationScoped
public class BillingService {

    public Uni<Charge> chargeCustomerReactive(
            String customerId, String billingPeriod, long amountCents) {

        // Developer wraps the synchronous Stripe call inside a Uni supplier
        // to avoid blocking the Vert.x event loop thread.
        // runSubscriptionOn() moves execution to a worker thread.
        // The supplier is a lambda — deferred, not eager.
        // It executes on EACH subscription, including every re-subscription by retry().
        return Uni.createFrom().item(() -> {
                    // UNSAFE: UUID computed inside the supplier lambda.
                    // This lambda executes on every subscription.
                    // Subscription 0 (initial): UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
                    // Subscription 1 (retry 1): UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
                    // Subscription 2 (retry 2): UUID = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"
                    String idempotencyKey = UUID.randomUUID().toString();

                    try {
                        RequestOptions options = RequestOptions.builder()
                                .setIdempotencyKey(idempotencyKey)
                                .build();
                        ChargeCreateParams params = ChargeCreateParams.builder()
                                .setAmount(amountCents)
                                .setCurrency("usd")
                                .setCustomer(customerId)
                                .setDescription("Billing period " + billingPeriod)
                                .build();
                        return Charge.create(params, options);
                    } catch (StripeException e) {
                        throw new RuntimeException(e);
                    }
                })
                .runSubscriptionOn(Infrastructure.getDefaultWorkerPool())
                .onFailure().retry()
                    .withBackOff(Duration.ofSeconds(1), Duration.ofSeconds(10))
                    .atMost(3);
    }
}

The failure scenario: the Quarkus REST endpoint calls billingService.chargeCustomerReactive("cust_123", "2026-08", 9900L). The method returns a Uni<Charge> immediately — no work has executed yet. The response is subscribed (by the Quarkus reactive HTTP layer, or by an explicit .subscribe().with(...) call). Mutiny subscribes to the upstream Uni. The runSubscriptionOn() stage schedules the supplier for execution on a worker thread from Infrastructure.getDefaultWorkerPool(). The worker thread executes the supplier lambda: UUID.randomUUID() returns "3f7a9b2c...". Charge.create(params, options) sends POST /v1/charges with Idempotency-Key: 3f7a9b2c....

Stripe commits ch_A and encounters a transient 503 before flushing the response. The catch (StripeException e) block wraps it in a RuntimeException and re-throws. The supplier completes with a failure. The .onFailure().retry() stage catches this failure, applies the 1-second backoff, and re-subscribes the upstream Uni. Re-subscribing means Mutiny calls the supplier lambda again on a new worker thread. The supplier lambda executes from its first statement: UUID.randomUUID() returns "b8d2e4f6..." — a new UUID, independent of the first call. The retry request sends Idempotency-Key: b8d2e4f6.... Stripe has ch_A cached against "3f7a9b2c...", not this new key. Stripe processes the retry as a fresh request. ch_B created.

The developer may reason: “I computed the UUID inside the Uni, so it’s encapsulated — surely it’s associated with this Uni instance and reused on retry.” But a Uni is a subscription factory, not a subscription. The Uni object returned by Uni.createFrom().item(supplier) holds a reference to the supplier lambda but does not hold the result of invoking it. Each subscription invokes the supplier. Each re-subscription by retry() invokes the supplier again. There is no result caching inside the Uni object itself.

The subtler variant: UUID computed in a .chain() or .flatMap() lambda appears to be “downstream” but is still deferred and re-evaluated on re-subscription

A variant of this failure occurs when the developer believes they have separated the UUID generation from the Stripe call by computing it in an upstream .map() or .chain() stage. The intent is to make the UUID visible in the pipeline without putting it inside the downstream Stripe call lambda. But all stages in a Uni pipeline are deferred — they execute from left to right on each subscription — so a UUID in any upstream stage is re-computed on every retry’s re-subscription:

// UNSAFE: UUID generated in a .map() stage — still deferred, re-evaluated on re-subscription.
// Developer thinks: "The UUID is computed in the mapping stage, not in the Stripe call stage.
// Surely the mapping stage's output is cached between retries."
// It is NOT cached. The entire Uni pipeline re-executes on each re-subscription.

public Uni<Charge> chargeCustomerReactive(String customerId, String billingPeriod, long amountCents) {
    return Uni.createFrom().item(() -> {
                // This supplier is a trigger — it produces an input for the downstream stages.
                // Executed on each subscription. Returns a placeholder to start the pipeline.
                return customerId;  // the idempotency key is not here...
            })
            .map(cid -> {
                // UNSAFE: UUID in a .map() lambda — also deferred, also re-evaluated per subscription.
                // Developer believes this stage "runs once" because it's upstream of the retry,
                // but .onFailure().retry() re-subscribes the entire Uni from the beginning.
                // This .map() lambda runs again on every retry.
                return UUID.randomUUID().toString();  // ← new UUID per retry
            })
            .flatMap(key -> Uni.createFrom().item(() -> {
                try {
                    RequestOptions options = RequestOptions.builder().setIdempotencyKey(key).build();
                    ChargeCreateParams params = ChargeCreateParams.builder()
                            .setAmount(amountCents).setCurrency("usd").setCustomer(customerId)
                            .setDescription("Billing period " + billingPeriod).build();
                    return Charge.create(params, options);
                } catch (StripeException e) {
                    throw new RuntimeException(e);
                }
            }).runSubscriptionOn(Infrastructure.getDefaultWorkerPool()))
            .onFailure().retry().withBackOff(Duration.ofSeconds(1)).atMost(3);
}

The position of UUID.randomUUID() in the pipeline — whether in the initial supplier, a .map() stage, or a .flatMap() stage — does not matter. Every stage in the pipeline re-executes on each re-subscription. The .onFailure().retry() stage is at the outermost level; when it re-subscribes the upstream Uni, “upstream” means the entire chain before it, from the initial Uni.createFrom().item() through every .map() and .flatMap() stage. All of them run again. All deferred computations — including any UUID.randomUUID() call at any stage — produce new values on each re-subscription.

The fix for failure mode 2

The idempotency key must be computed eagerly — before the Uni pipeline is constructed — and captured in the lambda closure as a final value. A computation outside any lambda, before the first Uni.createFrom() call, executes at the time the method is called, not at subscription time. The captured final variable is a closure over the computed value, not over a computation that re-evaluates:

// Safe: UUID computed eagerly before the Uni chain is constructed.
// Any computation outside a lambda executes when the method is called,
// not at subscription time. The captured final variable is the same
// on every re-subscription by .onFailure().retry().atMost(3).

@ApplicationScoped
public class SafeBillingService {

    @Inject
    BillingRepository billingRepository;

    public Uni<Charge> chargeCustomerReactive(
            String customerId, String billingPeriod, long amountCents) {

        // Safe: computed before any lambda. Not inside Uni.createFrom(), .map(), or .flatMap().
        // The same value is captured by closure into every downstream lambda.
        // Every re-subscription by .onFailure().retry() uses this same value.
        final String idempotencyKey = stableKey(customerId, billingPeriod);

        // Pre-flight: claim the billing slot before any Stripe call.
        // Returns a Uni that emits true if claimed, false if the slot was already taken.
        return billingRepository.claimSlotReactive(customerId, billingPeriod, idempotencyKey)
                .flatMap(claimed -> {
                    if (!claimed) {
                        // Another subscriber already claimed — return existing result
                        return billingRepository.findChargeReactive(customerId, billingPeriod);
                    }
                    // This subscriber won the claim — proceed to Stripe
                    return Uni.createFrom().item(() -> {
                                try {
                                    // Safe: idempotencyKey captured from outer scope.
                                    // final, computed before the Uni chain.
                                    // Same value on every re-subscription by retry().
                                    RequestOptions options = RequestOptions.builder()
                                            .setIdempotencyKey(idempotencyKey)
                                            .build();
                                    ChargeCreateParams params = ChargeCreateParams.builder()
                                            .setAmount(amountCents)
                                            .setCurrency("usd")
                                            .setCustomer(customerId)
                                            .setDescription("Billing period " + billingPeriod)
                                            .build();
                                    Charge charge = Charge.create(params, options);
                                    billingRepository.markCompleteBlocking(
                                            customerId, billingPeriod, charge.getId());
                                    return charge;
                                } catch (StripeException e) {
                                    throw new RuntimeException(e);
                                }
                            })
                            .runSubscriptionOn(Infrastructure.getDefaultWorkerPool())
                            .onFailure().retry()
                                .withBackOff(Duration.ofSeconds(1), Duration.ofSeconds(10))
                                .atMost(3);
                });
    }

    private static String stableKey(String customerId, String billingPeriod) {
        try {
            var digest = java.security.MessageDigest.getInstance("SHA-256");
            var hash = digest.digest(
                    (customerId + ":" + billingPeriod + ":quarkus-billing")
                            .getBytes(java.nio.charset.StandardCharsets.UTF_8));
            var sb = new StringBuilder(32);
            for (int i = 0; i < 16; i++) sb.append(String.format("%02x", hash[i]));
            return sb.toString();
        } catch (java.security.NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
    }
}

-- Pre-flight table: billing_records as the cluster-wide billing mutex
CREATE TABLE billing_records (
    customer_id      TEXT        NOT NULL,
    billing_period   TEXT        NOT NULL,
    idempotency_key  TEXT        NOT NULL,
    status           TEXT        NOT NULL DEFAULT 'pending',
    charge_id        TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT billing_records_pk  PRIMARY KEY (idempotency_key),
    CONSTRAINT billing_records_uq  UNIQUE (customer_id, billing_period)
);

-- claimSlotReactive: returns 1 row if claimed, 0 rows if conflict
INSERT INTO billing_records (customer_id, billing_period, idempotency_key)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id, billing_period) DO NOTHING
RETURNING idempotency_key;

-- markComplete: record the Stripe charge_id after success
UPDATE billing_records
   SET status = 'completed', charge_id = $3
 WHERE customer_id = $1 AND billing_period = $2;

With the stable key computed before the Uni chain and captured as a final closure variable, every re-subscription by .onFailure().retry() supplies the same Idempotency-Key to Stripe. The pre-flight billing_records table provides a second layer of protection: if two concurrent subscribers race to claim the same billing slot — a case that can occur when the Quarkus reactive router dispatches two requests for the same customer in quick succession — the ON CONFLICT DO NOTHING ensures only one of them reaches Stripe. The other finds the existing row and returns the already-committed charge ID.

Failure mode 3: Quarkus @Scheduled billing job per Kubernetes replica — no built-in cluster coordination without Quartz — three pods execute the billing loop simultaneously — distinct UUID.randomUUID() per pod per customer — ch_A, ch_B, and ch_C per customer per billing period

Quarkus’s @Scheduled annotation from quarkus-scheduler runs the annotated method on every JVM where the application is deployed. In a Kubernetes Deployment with replicas: 3, all three pods execute the @Scheduled method at the same cron expression time. Quarkus does not provide cross-pod scheduling coordination out of the box for the default in-memory scheduler — coordination requires switching to the Quartz-backed scheduler (quarkus-quartz) and enabling the clustered mode. Without this, three billing loops run at exactly the same second:

// BillingScheduler.java
// UNSAFE: @Scheduled runs on every Kubernetes replica independently.
// With replicas: 3, all three pods fire runMonthlyBilling() at the same cron second.
// No cross-pod coordination — all three pods query and charge customers concurrently.

import io.quarkus.scheduler.Scheduled;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import java.time.YearMonth;
import java.util.UUID;

@ApplicationScoped
public class BillingScheduler {

    @Inject
    BillingService billingService;

    @Inject
    CustomerRepository customerRepository;

    @Inject
    BillingRepository billingRepository;

    // Fires at 01:00:00 on the first day of every month.
    // @Scheduled uses the Quarkus in-memory scheduler by default — no clustering.
    // With replicas: 3, all three pods execute this method at 01:00:00.
    @Scheduled(cron = "0 0 1 1 * ?")
    void runMonthlyBilling() {
        String billingPeriod = YearMonth.now().toString();  // "2026-08"

        // DATABASE CHECK — all three pods query this row simultaneously.
        // Before any pod has committed a 'billing_started' record,
        // all three reads return false: TOCTOU (time-of-check time-of-use) race.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) {
            return;
        }

        // All three pods reach this point concurrently:
        var customers = customerRepository.findAllActive();  // 500 customers on every pod
        customers.forEach(customer -> {
            // Pod 1 at 01:00:00.003: UUID = "3f7a9b2c..." → ch_1A for cust_1
            // Pod 2 at 01:00:00.005: UUID = "b8d2e4f6..." → ch_1B for cust_1
            // Pod 3 at 01:00:00.006: UUID = "1c9d3e5a..." → ch_1C for cust_1
            // Across 500 customers: 1,500 Stripe charges created, 500 intended.
            String idempotencyKey = UUID.randomUUID().toString();
            try {
                billingService.chargeCustomer(
                        customer.getId(), billingPeriod, customer.getAmountCents(), idempotencyKey);
            } catch (StripeException e) {
                log.errorf(e, "Billing failed for customer %s in period %s",
                        customer.getId(), billingPeriod);
            }
        });
    }
}

The failure scenario: it is 01:00:00 on September 1, 2026. All three Kubernetes pods have the BillingScheduler bean active. The Quarkus in-memory scheduler on each pod fires the runMonthlyBilling() method at the same configured cron second — within milliseconds of each other, not at exactly the same system clock tick, but well within the TOCTOU race window. All three pods call billingRepository.hasCompletedForPeriod("2026-09"). The monthly billing has not run yet. The billing_completed row for September 2026 does not exist. All three database reads return false. All three pods pass the check and proceed to billing.

Each pod calls customerRepository.findAllActive() and receives all 500 active customers. The billing loops start concurrently on all three pods. For customer cust_1: Pod 1 generates UUID.randomUUID() = "3f7a9b2c..." and sends POST /v1/charges with Idempotency-Key: 3f7a9b2c.... Pod 2 generates "b8d2e4f6..." and sends a concurrent request with that key. Pod 3 generates "1c9d3e5a..." and sends a third concurrent request. Stripe sees three distinct idempotency keys for cust_1 in the same billing period. Three distinct keys mean three distinct billing intents as far as Stripe is concerned. Stripe creates ch_1A, ch_1B, and ch_1C. Across all 500 customers, 1,500 Stripe charges are created where 500 were intended. Each customer is charged three times for September 2026.

The hasCompletedForPeriod check is a standard TOCTOU race. Even if the check and the write are both database operations, the check-then-act gap between SELECT EXISTS(... WHERE billing_period = '2026-09') and any subsequent INSERT or Stripe call is long enough for all three pods to complete the read before any pod commits a write. The check provides no mutual exclusion guarantee without an atomic claim — it is an optimistic read that three concurrent pods can all pass simultaneously.

The fix for failure mode 3

The fix requires two layers. The first layer eliminates the three-pod billing race entirely by ensuring only one pod runs the billing job at all. The second layer provides content-hash keys and a pre-flight database claim as an authoritative backstop that handles edge cases the first layer cannot prevent.

The Quarkus-native first layer is the Quartz-backed scheduler with clustering enabled. quarkus-quartz uses a database table (QRTZ_*) to coordinate schedule execution across all instances of the application. When the cron expression fires, all three pods race to acquire a Quartz trigger lock in the database. Only one pod wins the lock and executes the job. The other two pods see the lock held and skip the trigger. This is a built-in, database-backed distributed lock for @Scheduled jobs, requiring no additional coordination code:

# application.properties — switch from in-memory to Quartz-backed clustered scheduler
quarkus.quartz.clustered=true
quarkus.quartz.store-type=jdbc-cmt  # Use JDBC JobStore with container-managed transactions
quarkus.quartz.datasource=<default>  # Use the application's default datasource

# The Quartz schema tables (QRTZ_JOB_DETAILS, QRTZ_TRIGGERS, QRTZ_LOCKS, etc.)
# are created automatically by quarkus-quartz on startup if they do not exist.
# With clustered=true, Quartz uses SELECT ... FOR UPDATE on QRTZ_LOCKS to ensure
# only one cluster node fires each trigger at its scheduled time.

// BillingScheduler.java — Safe with Quartz clustering
// @Scheduled fires on all pods, but Quartz's database lock ensures only one pod
// executes the job body. The other two pods see the lock held and skip the cycle.

import io.quarkus.scheduler.Scheduled;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import java.time.YearMonth;

@ApplicationScoped
public class BillingScheduler {

    @Inject
    SafeBillingOrchestrator orchestrator;

    // With quarkus.quartz.clustered=true, this method executes on exactly one pod per firing.
    // Quartz acquires a database row lock before executing the trigger, serializing the job
    // across all pods. The other two pods' schedulers see the trigger as already fired
    // and do not execute the method body.
    @Scheduled(cron = "0 0 1 1 * ?")
    void runMonthlyBilling() {
        String billingPeriod = YearMonth.now().toString();
        orchestrator.runBillingForPeriod(billingPeriod);
    }
}

@ApplicationScoped
public class SafeBillingOrchestrator {

    @Inject
    CustomerRepository customerRepository;

    @Inject
    BillingRepository billingRepository;

    @Inject
    SafeBillingService billingService;

    @jakarta.transaction.Transactional
    public void runBillingForPeriod(String billingPeriod) {
        // Even with Quartz clustering, include the advisory lock as a backstop.
        // Edge cases that bypass Quartz: operator-triggered billing via admin endpoint,
        // rolling deploy overlap where two pod versions both start Quartz schedulers,
        // or a future code path that calls runBillingForPeriod() outside the scheduler.
        long lockKey = Math.abs(("monthly-billing:" + billingPeriod).hashCode());

        // pg_try_advisory_xact_lock: acquired for the duration of the transaction.
        // Auto-released on COMMIT or ROLLBACK. Only one database connection holds it.
        var acquired = (Boolean) entityManager
                .createNativeQuery("SELECT pg_try_advisory_xact_lock(" + lockKey + ")")
                .getSingleResult();

        if (!acquired) {
            log.infof("Advisory lock for period %s not acquired — another node running billing",
                    billingPeriod);
            return;
        }

        var customers = customerRepository.findAllActive();
        customers.forEach(customer -> {
            // Safe: stableKey computed per customer, per period.
            // Same value on all pods — Stripe's idempotency cache deduplicates
            // the rare case where the advisory lock fails (e.g., network partition
            // causing two pods to simultaneously believe they hold the lock).
            String idempotencyKey = stableKey(customer.getId(), billingPeriod);

            // Pre-flight: claim billing slot before Stripe call.
            // ON CONFLICT DO NOTHING is the authoritative cluster-wide mutex.
            // If the advisory lock is the first layer (prevents concurrent runs),
            // ON CONFLICT DO NOTHING is the second layer (prevents double-billing
            // even if two runs overlap in the narrow edge cases above).
            boolean claimed = billingRepository.claimSlot(
                    customer.getId(), billingPeriod, idempotencyKey);
            if (claimed) {
                try {
                    billingService.chargeCustomer(
                            customer.getId(), billingPeriod, customer.getAmountCents(), idempotencyKey);
                } catch (StripeException e) {
                    log.errorf(e, "Billing failed for customer %s in period %s",
                            customer.getId(), billingPeriod);
                }
            }
        });
    }
}

With quarkus.quartz.clustered=true, the Quartz database lock ensures only one pod runs the billing job per trigger firing. With the content-hash idempotency key, every pod that might run the billing job (due to edge cases the Quartz lock cannot cover) sends the same key per customer per billing period. With the ON CONFLICT DO NOTHING pre-flight claim, even if two pods somehow both reach Stripe for the same customer in the same billing period, only one of them commits a charge — the second finds the existing billing_records row and returns the already-committed charge ID without sending a second Stripe request.

The pg_try_advisory_xact_lock() variant used here (transaction-level advisory lock, rather than session-level pg_try_advisory_lock()) releases automatically when the transaction commits or rolls back, returning the database connection to the pool immediately. This avoids holding a dedicated connection open for the entire billing run, which can last minutes for a large customer base. The trade-off is that the lock is only held within the transaction — if the billing run requires multiple transactions (for example, if each customer’s billing is in its own @Transactional method that commits independently), the advisory lock must be acquired in a dedicated outer transaction that wraps the entire billing run. For a single-transaction billing job this is not a concern.

-- Alternative: session-level advisory lock for multi-transaction billing runs
-- Hold the connection open for the duration of the billing job
-- Release explicitly via pg_advisory_unlock() in a finally block

DataSource ds = ...;
try (Connection conn = ds.getConnection()) {
    long lockKey = Math.abs(("monthly-billing:" + billingPeriod).hashCode());
    boolean acquired;
    try (PreparedStatement stmt = conn.prepareStatement("SELECT pg_try_advisory_lock(?)")) {
        stmt.setLong(1, lockKey);
        try (ResultSet rs = stmt.executeQuery()) {
            acquired = rs.next() && rs.getBoolean(1);
        }
    }
    if (!acquired) return;  // another node holds the lock

    try {
        runBillingCustomers(billingPeriod, conn);  // multiple transactions inside
    } finally {
        try (PreparedStatement stmt = conn.prepareStatement("SELECT pg_advisory_unlock(?)")) {
            stmt.setLong(1, lockKey);
            stmt.executeQuery();
        }
    }
}

Gap analysis: other Quarkus billing patterns not covered above

The three failure modes above cover the most common Quarkus retry surfaces. Several adjacent patterns introduce the same class of failure through different mechanisms:

Quarkus REST client with @Retry on the interface method — a MicroProfile REST client interface method annotated with @Retry is re-invoked by the SmallRye interceptor on each retry attempt. If the request body includes a UUID generated at the call site of the interface method (e.g., inside a JAX-RS @POST handler that builds the request object including UUID.randomUUID() and passes it to the REST client), the UUID is computed before the @Retry boundary and is safe. But if the @Retry annotation is on the interface method itself and the interface method’s implementation includes UUID generation (e.g., via a CDI interceptor or a default method that appends a header), the UUID generation re-runs on each retry. The fix: compute the stable key in the caller and inject it as a header via a ClientRequestFilter that reads from a thread-local or request-scoped context set before the REST client call.

Quarkus @Incoming / @Outgoing (SmallRye Reactive Messaging) billing consumer — a billing consumer annotated with @Incoming("billing-requests") that uses Uni.createFrom().item(() -> ...) with a UUID inside the supplier and .onFailure().retry() for transient Stripe errors falls into the same Mutiny re-subscription failure as failure mode 2. The message delivery semantics (at-least-once from Kafka or AMQP) mean that if the consumer crashes before acknowledging the message, the broker redelivers it and the consumer processes it again — but this is a message-level redelivery, not a Mutiny retry; the key fix is the same content-hash key stable across both Mutiny retries and message redeliveries. See the Kafka and Stripe Integration post for the message redelivery failure modes specific to Kafka at-least-once semantics.

Quarkus @Retry on a @Transactional method where the transaction has already committed before the exception is thrown — a Quarkus JPA method annotated with both @Transactional and @Retry: SmallRye’s @Retry interceptor fires after the CDI transaction interceptor, meaning it sees exceptions thrown after the transaction has already committed. If the transaction commits the billing record but the code after the transaction boundary (e.g., sending a confirmation email or posting to a webhook) throws, @Retry re-invokes the method body including the billing code inside the transaction, which inserts a duplicate billing record — but the UNIQUE (customer_id, billing_period) constraint throws a ConstraintViolationException, which @Retry also retries. The fix: ensure the @Transactional boundary wraps only the pre-flight claim and the Stripe call; post-transaction side effects (emails, webhooks) are in a separate @ApplicationScoped method without @Retry, called after the transaction commits.

Quarkus Mutiny Uni.createFrom().deferred(() -> ...) — Uni.createFrom().deferred(supplier) is explicitly named for deferred execution: the supplied Uni factory is called on each subscription. This is more obvious than Uni.createFrom().item(supplier), but developers sometimes use deferred to delay Stripe client initialization and accidentally include UUID.randomUUID() inside the deferred factory: Uni.createFrom().deferred(() -> { String key = UUID.randomUUID().toString(); return buildStripeCall(key); }). Every re-subscription by retry() calls the deferred factory, re-computes the UUID, and builds a new Stripe call with a different key. The fix is the same: compute the stable key before the deferred() call and capture it in the factory lambda’s closure.

Quarkus @Scheduled with concurrent = false but without clustering — @Scheduled(cron = "...", concurrentExecution = Scheduled.ConcurrentExecution.SKIP) prevents concurrent executions of the same scheduled method within a single JVM — it does not prevent concurrent executions across multiple JVM replicas. The concurrentExecution attribute is a per-JVM guard, not a cross-pod guard. With replicas: 3, SKIP ensures each pod fires at most once per trigger period within its own JVM, but all three pods can still fire simultaneously. The fix requires quarkus.quartz.clustered=true or pg_try_advisory_lock() for cross-pod serialization; concurrentExecution = SKIP is not a substitute.

Summary

The pattern connecting all three: the idempotency key must be derived from the business intent of the billing operation — customer ID, billing period, vendor namespace — not from any ephemeral runtime value that re-evaluates between @Retry invocations, Mutiny re-subscriptions, or concurrent pod executions. UUID.randomUUID(), System.currentTimeMillis() at method entry, System.nanoTime(), and System.identityHashCode() of any object all produce different values on each invocation, subscription, or pod. A key derived from sha256(customerId + ":" + billingPeriod + ":quarkus-billing")[:32] produces the same 32-character hex string on every @Retry attempt, on every Mutiny re-subscription, on every pod, and on every billing path within the same period — stable across all three failure modes. Backing it with a database-level UNIQUE (customer_id, billing_period) constraint and either a Quartz database trigger lock or a pg_try_advisory_xact_lock() distributed mutex moves the deduplication guarantee out of ephemeral in-memory state and into durable storage that survives timeouts, reconnects, rolling deploys, and multi-pod races.

The vault key spend cap adds a hard financial boundary as a last line of defence: a per-billing-period vault key issued with a max_amount set to expected_total × 1.10 caps the maximum spend that any combination of bugs, retries, and multi-pod races can produce. Once the cap is reached, Stripe rejects further charges with a 402 Payment Required on the vault key, containing the blast radius to a known maximum regardless of how many @Retry invocations, Mutiny re-subscriptions, or concurrent billing pods are in flight.