Helidon and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Helidon MP’s MicroProfile Fault Tolerance @Retry annotation and Helidon SE 4.x’s virtual-thread-friendly retry loops both make retry feel natural to add, but where the idempotency key is computed relative to the retry boundary determines whether a transient Stripe error produces a safe retry or a duplicate charge. Three Helidon-specific failure modes: Helidon MP’s MicroProfile Fault Tolerance interceptor re-invokes the CDI method body in full on each retry attempt — UUID.randomUUID() at method entry fires on every InvocationContext.proceed() call; Helidon SE 4.x’s virtual-thread retry loop produces a new UUID on each iteration if the key is generated inside the loop body, and a common refactoring from Helidon SE 3.x reactive code accidentally introduces this pattern; and Helidon MP’s built-in scheduler fires @Scheduled methods on every Kubernetes replica independently — three pods pass a concurrent database check simultaneously, each generates distinct UUID.randomUUID() values per customer, and each creates ch_A, ch_B, and ch_C per billing period.

This post covers all three failure modes with Helidon MP 4.x and Helidon SE 4.x Java code, content-hash idempotency keys stable across MicroProfile FT re-invocations, virtual-thread retry iterations, and multi-pod concurrent billing loops, pg_try_advisory_lock() for cross-pod scheduler serialization, pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For the CDI interceptor retry pattern in the Quarkus ecosystem (SmallRye Fault Tolerance), see the Quarkus and Stripe Integration post. For MicroProfile Fault Tolerance retry in the Micronaut ecosystem, see the Micronaut and Stripe Integration post. For Kotlin coroutine retry failure modes in Ktor, see the Ktor and Stripe Integration post.

Failure mode 1: Helidon MP MicroProfile Fault Tolerance @Retry interceptor re-invokes the CDI method body — UUID.randomUUID() at method entry fires on every InvocationContext.proceed() call — the initial attempt created ch_A before the StripeException — the first retry creates ch_B

Helidon MP implements the MicroProfile Fault Tolerance specification using CDI interceptors. When a method annotated with @Retry throws a retryable exception, Helidon’s fault tolerance interceptor catches the exception, waits for the configured backoff delay, and calls InvocationContext.proceed() to re-execute the method body for the next attempt. InvocationContext.proceed() does not resume from a suspended point — it re-invokes the entire method body from its first statement, including any UUID.randomUUID() call placed at method entry:

// BillingService.java
// UNSAFE: UUID computed inside the @Retry-annotated CDI method body
// Helidon MP's MicroProfile FT interceptor calls InvocationContext.proceed()
// to re-invoke this method body on each retry — the method executes from the top
// on every attempt including retries.

import org.eclipse.microprofile.faulttolerance.Retry;
import org.eclipse.microprofile.faulttolerance.Timeout;
import com.stripe.model.Charge;
import com.stripe.net.RequestOptions;
import com.stripe.param.ChargeCreateParams;
import com.stripe.exception.StripeException;

import jakarta.enterprise.context.ApplicationScoped;
import java.time.temporal.ChronoUnit;
import java.util.UUID;

@ApplicationScoped
public class BillingService {

    @Retry(maxRetries = 3,
           delay = 1000,
           delayUnit = ChronoUnit.MILLIS,
           retryOn = { StripeException.class })
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents)
            throws StripeException {
        // UNSAFE: UUID computed at method entry.
        // Helidon MP's @Retry interceptor re-invokes the entire method body on each retry.
        // Attempt 0 (initial call): UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Attempt 1 (@Retry re-invocation): UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        // Attempt 2 (@Retry re-invocation): UUID = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"
        String idempotencyKey = UUID.randomUUID().toString();

        RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build();

        ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(amountCents)
                .setCurrency("usd")
                .setCustomer(customerId)
                .setDescription("Billing period " + billingPeriod)
                .build();

        return Charge.create(params, options);
    }
}

The failure scenario: the agent calls billingService.chargeCustomer("cust_123", "2026-08", 9900L) on the CDI proxy. Helidon’s MicroProfile FT interceptor intercepts the call and invokes InvocationContext.proceed() to run the method body for attempt 0. The method body executes from the top: UUID.randomUUID() returns "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e". Charge.create(params, options) sends POST /v1/charges with Idempotency-Key: 3f7a9b2c... to Stripe.

Stripe receives the request. The card is authorized and the charge object ch_A is committed to Stripe’s ledger. Before the HTTP response reaches the client, Stripe’s infrastructure encounters a transient overload and returns HTTP 503. The Stripe Java SDK throws a StripeException. Helidon’s fault tolerance interceptor catches the exception, verifies it matches retryOn = { StripeException.class }, applies the 1-second backoff, and calls InvocationContext.proceed() again for attempt 1.

InvocationContext.proceed() runs the method body from the top. UUID.randomUUID() evaluates again — a completely independent call that has no knowledge of the UUID generated in attempt 0. It returns "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d". The retry request sends POST /v1/charges with Idempotency-Key: b8d2e4f6.... Stripe has ch_A cached against "3f7a9b2c...", not this new key. Stripe processes the retry as a fresh charge request. ch_B is created. Customer 123 is charged $99 twice for August 2026.

This failure is structurally identical to the same bug in Quarkus SmallRye Fault Tolerance and Micronaut @Retryable — all three implement the MicroProfile Fault Tolerance specification using the same CDI interception model. The mechanism is specification-mandated: @Retry must re-invoke the method on each attempt, and any code at method entry executes fresh on every attempt. The fix is not framework-specific; it applies to every MicroProfile FT implementation.

The subtler variant: Helidon MP REST Client with @ClientHeaderParam using a default interface method that calls UUID.randomUUID() — the method is invoked on each @Retry re-invocation of the REST client proxy — ch_B

Helidon MP integrates with MicroProfile REST Client. Developers can use @ClientHeaderParam to automatically inject request headers from a method reference. A common pattern is to factor the idempotency key generation into a default interface method and reference it from @ClientHeaderParam. The intent is clean separation: the interface declares the header, the implementation computes the value. But if the method referenced by @ClientHeaderParam calls UUID.randomUUID(), it is invoked on every call to the interface method — including each @Retry re-invocation of the REST client proxy:

// StripeRestClient.java
// UNSAFE: @ClientHeaderParam references a default method that calls UUID.randomUUID().
// The default method is invoked on each call to stripeClient.createCharge(),
// which happens on every @Retry re-invocation of the REST client interface proxy.

import org.eclipse.microprofile.rest.client.annotation.ClientHeaderParam;
import org.eclipse.microprofile.rest.client.inject.RegisterRestClient;
import org.eclipse.microprofile.faulttolerance.Retry;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import java.util.UUID;

@RegisterRestClient(baseUri = "https://api.stripe.com")
public interface StripeRestClient {

    // @ClientHeaderParam with a method reference — the method is called once per
    // REST client interface method invocation, including every @Retry retry attempt.
    @POST
    @Path("/v1/charges")
    @ClientHeaderParam(name = "Idempotency-Key", value = "{generateIdempotencyKey}")
    @Retry(maxRetries = 3, delay = 1000, delayUnit = java.time.temporal.ChronoUnit.MILLIS)
    ChargeResponse createCharge(ChargeRequest body);

    default String generateIdempotencyKey() {
        // UNSAFE: UUID.randomUUID() called here is re-evaluated on every invocation
        // of createCharge(), including each @Retry re-invocation.
        // Attempt 0: "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Attempt 1: "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        // The developer intended to encapsulate key generation in one place,
        // but the @ClientHeaderParam framework calls this method per-invocation.
        return UUID.randomUUID().toString();
    }
}

The @ClientHeaderParam annotation’s method reference is evaluated by the REST client framework each time the annotated interface method is called. The MicroProfile REST Client specification is explicit: the method reference is called to get the header value for the current request. When @Retry causes createCharge() to be called again for the next attempt, the REST client framework calls generateIdempotencyKey() again to get the header value for that attempt’s request. UUID.randomUUID() inside the default method produces a new value. The result is identical to putting UUID.randomUUID() directly inside the method body: each retry attempt sends a different Idempotency-Key header.

This variant is especially hard to spot in code review because the key generation appears to be cleanly factored and reusable. The name generateIdempotencyKey sounds correct. The separation of header computation from request logic looks like good practice. The problem is invisible until a transient Stripe error causes a retry and the duplicate charge appears on the customer’s statement.

The fix for failure mode 1

The idempotency key must be computed before the @Retry boundary can re-invoke any code that computes it. This means computing the key in the caller, outside the @Retry-annotated method, and passing it as an explicit parameter into the method. For the REST client case, the key must be injected from the calling context — not generated inside a @ClientHeaderParam method reference:

// Safe: stable key computed by the CALLER before invoking the @Retry method.
// The @Retry-annotated method receives the key as a parameter — it does not re-compute it.

@ApplicationScoped
public class BillingOrchestrator {

    @Inject
    BillingService billingService;

    @Inject
    BillingRepository billingRepository;

    public Charge executeBilling(String customerId, String billingPeriod, long amountCents)
            throws StripeException {
        // Computed once in the caller, before any @Retry invocation.
        // The stable hash is the same value on every InvocationContext.proceed() call.
        String idempotencyKey = stableKey(customerId, billingPeriod);

        // Pre-flight: claim the billing slot before any Stripe call.
        // ON CONFLICT DO NOTHING means only the first caller reaches Stripe.
        boolean claimed = billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey);
        if (!claimed) {
            return billingRepository.findCharge(customerId, billingPeriod);
        }

        return billingService.chargeCustomer(customerId, billingPeriod, amountCents, idempotencyKey);
    }

    static String stableKey(String customerId, String billingPeriod) {
        try {
            var digest = java.security.MessageDigest.getInstance("SHA-256");
            var hash = digest.digest(
                    (customerId + ":" + billingPeriod + ":helidon-billing")
                            .getBytes(java.nio.charset.StandardCharsets.UTF_8));
            var sb = new StringBuilder(32);
            for (int i = 0; i < 16; i++) sb.append(String.format("%02x", hash[i]));
            return sb.toString();
        } catch (java.security.NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
    }
}

@ApplicationScoped
public class BillingService {

    @Retry(maxRetries = 3, delay = 1000, delayUnit = ChronoUnit.MILLIS,
           retryOn = { StripeException.class })
    public Charge chargeCustomer(String customerId, String billingPeriod, long amountCents,
                                  String idempotencyKey)  // ← key as parameter, not re-computed
            throws StripeException {
        // Safe: idempotencyKey is a parameter from the caller.
        // @Retry re-invokes this method body on each retry, but the parameter value
        // is the stable hash computed once by BillingOrchestrator — the same
        // 32-character hex string on every InvocationContext.proceed() call.
        RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build();

        ChargeCreateParams params = ChargeCreateParams.builder()
                .setAmount(amountCents)
                .setCurrency("usd")
                .setCustomer(customerId)
                .setDescription("Billing period " + billingPeriod)
                .build();

        Charge charge = Charge.create(params, options);
        billingRepository.markComplete(customerId, billingPeriod, charge.getId());
        return charge;
    }
}

// For the REST client case: inject the stable key via a ClientRequestFilter
// registered on the Helidon MP REST client, reading from a request-scoped context
// set by the caller before making the REST client call — not from @ClientHeaderParam.

// 1. Set the key in a request-scoped context before calling the REST client:
@RequestScoped
public class BillingRequestContext {
    private String idempotencyKey;
    public String getIdempotencyKey() { return idempotencyKey; }
    public void setIdempotencyKey(String key) { this.idempotencyKey = key; }
}

// 2. In the caller:
//    context.setIdempotencyKey(stableKey(customerId, billingPeriod));
//    stripeClient.createCharge(body);

// 3. In a ClientRequestFilter registered on the REST client:
public class IdempotencyKeyFilter implements ClientRequestFilter {
    @Inject BillingRequestContext ctx;

    @Override
    public void filter(ClientRequestContext requestContext) {
        String key = ctx.getIdempotencyKey();
        if (key != null) {
            requestContext.getHeaders().putSingle("Idempotency-Key", key);
        }
    }
}

// What to EXCLUDE from stableKey — any value that re-evaluates differently
// between @Retry invocations (i.e., on each InvocationContext.proceed() call):
//   UUID.randomUUID()              ← random, different on every invocation
//   System.currentTimeMillis()     ← different on each retry attempt
//   System.nanoTime()              ← monotonic but different per JVM start and per call
//   attempt counter                ← intentionally changes per retry; never put in key
//   Thread.currentThread().getId() ← Helidon may use different thread per retry
//   @ClientHeaderParam method ref  ← called fresh on every interface method invocation

// What IS safe in stableKey:
//   customerId                     ← stable business identifier
//   billingPeriod                  ← stable business identifier (e.g., "2026-08")
//   a vendor namespace string      ← literal constant ("helidon-billing")
// → sha256("cust_123:2026-08:helidon-billing")[:32] = same 32-char hex on every retry

With the stable key computed in the caller and passed as a parameter, every @Retry invocation of chargeCustomer() sends the same Idempotency-Key header to Stripe. After the initial attempt creates ch_A and a transient StripeException is thrown, the retry sends the identical key. Stripe’s idempotency cache finds the completed charge and returns the cached ch_A response without creating a new charge. For the REST client case, the ClientRequestFilter reads the stable key from the request-scoped context on every invocation and always finds the same value set by the caller before the first attempt.

Failure mode 2: Helidon SE 4.x virtual-thread retry loop — UUID.randomUUID() inside the loop body produces a different idempotency key on each iteration — ch_B on the first retry

Helidon SE 4.x moved away from the reactive Single/Multi API of Helidon SE 3.x in favour of blocking code running on virtual threads. The migration is a common refactoring step for teams upgrading from SE 3.x. A side effect of this migration is that retry patterns that were correct in the reactive model can become incorrect in the blocking model if the UUID generation is not carefully repositioned.

In Helidon SE 3.x, the reactive pattern for retry often looked like this:

// Helidon SE 3.x — UNSAFE reactive retry
// The supplier in Single.just() or the chain in .map() is re-evaluated on each
// re-subscription triggered by .retry(). UUID inside the chain produces ch_B.

import io.helidon.reactive.webclient.WebClient;
import io.helidon.reactive.webclient.WebClientResponse;

WebClient webClient = WebClient.builder()
        .baseUri("https://api.stripe.com")
        .build();

// UNSAFE in SE 3.x: UUID.randomUUID() inside the flatMap lambda is deferred —
// re-evaluated on each .retry() re-subscription.
Single<ChargeResponse> chargeSingle = webClient
        .post()
        .path("/v1/charges")
        .flatMap(req -> {
            String idempotencyKey = UUID.randomUUID().toString();  // deferred, re-evaluated on retry
            req.headers().add("Idempotency-Key", idempotencyKey);
            return req.submit(buildBody(customerId, billingPeriod, amountCents));
        })
        .flatMap(response -> response.as(ChargeResponse.class))
        .retry(3);

The SE 3.x safe fix was to compute the UUID before the reactive chain and capture it as final in the lambda closure. When migrating to Helidon SE 4.x blocking style, a developer converting this code might correctly identify that the UUID needs to be stable across retries. But the refactoring to a blocking for loop introduces a new placement decision: where exactly does the UUID go? If the developer co-locates the UUID with the Stripe call — inside the loop body, because they arrived together from the reactive lambda — the result is a new UUID on each iteration:

// Helidon SE 4.x — UNSAFE blocking retry loop
// The migration from reactive SE 3.x moved the Stripe call and UUID generation
// out of the flatMap lambda into a blocking for loop — but UUID stayed inside the loop.

import io.helidon.webclient.api.HttpClientResponse;
import io.helidon.webclient.http1.Http1Client;

Http1Client client = Http1Client.builder()
        .baseUri("https://api.stripe.com")
        .build();

Charge chargeCustomer(String customerId, String billingPeriod, long amountCents)
        throws StripeException {
    int maxRetries = 3;
    long backoffMs = 1_000;

    for (int attempt = 0; attempt < maxRetries; attempt++) {
        // UNSAFE: UUID.randomUUID() computed inside the loop body.
        // Migrated from the flatMap lambda in SE 3.x, where it was also deferred.
        // Attempt 0: UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        //   → POST /v1/charges with Idempotency-Key: 3f7a9b2c...
        //   → Stripe processes the request, creates ch_A, returns 503 before flushing response.
        // Attempt 1: UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"  ← fresh UUID
        //   → POST /v1/charges with Idempotency-Key: b8d2e4f6...
        //   → Stripe has never seen this key; creates ch_B. Customer charged twice.
        String idempotencyKey = UUID.randomUUID().toString();

        try {
            RequestOptions options = RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .build();
            ChargeCreateParams params = ChargeCreateParams.builder()
                    .setAmount(amountCents)
                    .setCurrency("usd")
                    .setCustomer(customerId)
                    .setDescription("Billing period " + billingPeriod)
                    .build();
            return Charge.create(params, options);

        } catch (StripeException e) {
            if (attempt == maxRetries - 1) throw e;
            try {
                Thread.sleep(backoffMs * (1L << attempt));  // virtual thread — suspends without blocking carrier
            } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
                throw new RuntimeException(ie);
            }
        }
    }
    throw new IllegalStateException("unreachable");
}

The error is in the placement of UUID.randomUUID() inside the for loop body. In the SE 3.x reactive version, the UUID was inside the flatMap lambda, where it was deferred and re-evaluated on each .retry() re-subscription. In the SE 4.x blocking version, the UUID is inside the loop body, where it evaluates on each loop iteration. Both patterns produce the same result: a fresh UUID on each retry attempt, and a fresh Stripe charge on each retry. The fact that Helidon SE 4.x uses virtual threads via Thread.sleep() (which suspends the virtual thread, not the carrier thread) does not change this — virtual threads are a concurrency mechanism, not an idempotency mechanism.

The subtler variant: Helidon SE 4.x StructuredTaskScope-based retry submits a new virtual thread per attempt — each thread’s Callable calls UUID.randomUUID() at its entry — ch_B

Helidon SE 4.x embraces Project Loom’s structured concurrency via StructuredTaskScope. A developer implementing a “retry with timeout” pattern might submit each billing attempt as a new Callable to a StructuredTaskScope.ShutdownOnSuccess scope, hoping the first successful attempt wins. If the billing callable is defined as a lambda or method reference that calls UUID.randomUUID() at its start, each submitted callable has its own UUID — and if two callables run concurrently (the scope does not prevent concurrent execution), both may reach Stripe simultaneously with different keys:

// UNSAFE: StructuredTaskScope with concurrent callable submission
// Each callable calls UUID.randomUUID() at its start — different key per callable.
// If both callables reach Stripe before either returns (network latency + scope concurrency),
// Stripe creates ch_A and ch_B simultaneously.

try (var scope = new StructuredTaskScope.ShutdownOnSuccess<Charge>()) {
    // Callable 1 submitted immediately
    scope.fork(() -> attemptCharge(customerId, billingPeriod, amountCents));
    // Callable 2 submitted after 2 seconds (retry delay)
    Thread.sleep(2_000);
    scope.fork(() -> attemptCharge(customerId, billingPeriod, amountCents));
    scope.join();
    return scope.result();
}

Charge attemptCharge(String customerId, String billingPeriod, long amountCents)
        throws StripeException {
    // UNSAFE: UUID.randomUUID() evaluated at the start of each callable invocation.
    // Each forked virtual thread runs this from the top — different UUID per fork.
    String idempotencyKey = UUID.randomUUID().toString();
    // ...
}

The StructuredTaskScope.ShutdownOnSuccess scope shuts down when the first callable succeeds and cancels the remaining forks via interruption. But interruption is cooperative: if callable 2 has already passed the UUID.randomUUID() point and sent the POST /v1/charges request to Stripe before the cancellation signal arrives, Stripe processes the second request. If callable 1 also successfully charged before the scope shut down, ch_A and ch_B are both committed.

The fix for failure mode 2

The idempotency key must be computed once, before the retry loop or before the StructuredTaskScope submits any callable, and shared across all iterations or forks:

// Safe: UUID computed ONCE before the retry loop — shared across all iterations.

Charge chargeCustomerSafe(String customerId, String billingPeriod, long amountCents)
        throws StripeException {
    // Computed once, outside the loop. The same value is used on every iteration.
    // sha256(customerId:billingPeriod:helidon-billing) — stable across JVM restarts,
    // rolling deploys, and any number of retry iterations.
    String idempotencyKey = stableKey(customerId, billingPeriod);

    int maxRetries = 3;
    for (int attempt = 0; attempt < maxRetries; attempt++) {
        try {
            RequestOptions options = RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)  // ← same key on every iteration
                    .build();
            ChargeCreateParams params = ChargeCreateParams.builder()
                    .setAmount(amountCents)
                    .setCurrency("usd")
                    .setCustomer(customerId)
                    .setDescription("Billing period " + billingPeriod)
                    .build();
            Charge charge = Charge.create(params, options);
            billingRepository.markComplete(customerId, billingPeriod, charge.getId());
            return charge;

        } catch (StripeException e) {
            if (attempt == maxRetries - 1) throw e;
            try {
                Thread.sleep(1_000L * (1L << attempt));
            } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
                throw new RuntimeException(ie);
            }
        }
    }
    throw new IllegalStateException("unreachable");
}

// For StructuredTaskScope: compute the stable key before forking any callable.
// Pass it as a closed-over final variable — every forked callable uses the same key.
// With a stable key, if two callables reach Stripe concurrently, the second request
// returns the cached ch_A response rather than creating ch_B.

String stableKey = stableKey(customerId, billingPeriod);  // ← computed before forking

try (var scope = new StructuredTaskScope.ShutdownOnSuccess<Charge>()) {
    scope.fork(() -> {
        RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(stableKey)  // ← same key in both forks
                .build();
        // ...
    });
    Thread.sleep(2_000);
    scope.fork(() -> {
        RequestOptions options = RequestOptions.builder()
                .setIdempotencyKey(stableKey)  // ← same key in both forks
                .build();
        // ...
    });
    scope.join();
    return scope.result();
}

With the stable key computed outside the loop or scope, every retry iteration or forked callable sends the identical Idempotency-Key header. Stripe’s idempotency cache deduplicates concurrent or sequential requests with the same key. When the initial attempt creates ch_A and a transient error fires, the retry sends the same key and Stripe returns the cached ch_A without creating a new charge. Even if two StructuredTaskScope forks both reach Stripe simultaneously, Stripe serializes them: the first to arrive creates ch_A, the second finds the key in the idempotency cache and returns ch_A again.

Failure mode 3: Helidon MP @Scheduled (or Helidon SE ScheduledExecutorService) runs on every Kubernetes replica — TOCTOU race on hasCompletedForPeriod() — three pods fire simultaneously — UUID.randomUUID() per customer per pod — ch_A, ch_B, ch_C per customer per billing period

Helidon MP includes a scheduling extension that exposes the @Scheduled annotation (from the helidon-microprofile-scheduling dependency). The scheduler is backed by a per-JVM ScheduledExecutorService and has no built-in cross-cluster coordination. With a Kubernetes Deployment scaled to replicas: 3, all three pods start their scheduler and fire the billing method at the same cron second:

// BillingScheduler.java — Helidon MP scheduling extension
// UNSAFE: @Scheduled runs on every JVM. With replicas: 3, all three pods fire.

import io.helidon.microprofile.scheduling.FixedRate;
import io.helidon.microprofile.scheduling.Scheduled;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import java.util.UUID;
import java.util.concurrent.TimeUnit;

@ApplicationScoped
public class BillingScheduler {

    @Inject
    private CustomerRepository customerRepository;

    @Inject
    private BillingRepository billingRepository;

    @Inject
    private BillingService billingService;

    // This method fires on all three Kubernetes pods at the configured cron second.
    // Helidon's scheduling extension has no cross-pod synchronization mechanism.
    @Scheduled("0 0 1 1 * ?")  // 01:00 on the 1st of each month
    void runMonthlyBilling() {
        String billingPeriod = java.time.YearMonth.now().toString();  // e.g., "2026-08"

        // TOCTOU: all three pods call hasCompletedForPeriod() concurrently.
        // All three read false before any pod has committed the billing-started record.
        // All three pass the check and proceed to the billing loop.
        if (billingRepository.hasCompletedForPeriod(billingPeriod)) {
            return;
        }

        // All three pods reach here. Each streams all 500 customers.
        customerRepository.findAllActive().forEach(customer -> {
            // Each pod calls UUID.randomUUID() per customer independently.
            // Pod 1: UUID = "3f7a9b2c..." → ch_A for customer 1
            // Pod 2: UUID = "b8d2e4f6..." → ch_B for customer 1
            // Pod 3: UUID = "1c9d3e5a..." → ch_C for customer 1
            // 500 customers × 3 pods = 1,500 Stripe charges created.
            String idempotencyKey = UUID.randomUUID().toString();
            try {
                billingService.chargeCustomer(customer.getId(), billingPeriod,
                        customer.getAmountCents(), idempotencyKey);
            } catch (Exception e) {
                log.error("Billing failed for " + customer.getId(), e);
            }
        });
    }
}

The Helidon scheduling extension’s @Scheduled annotation uses a ScheduledExecutorService per JVM. The Helidon documentation does not describe any cluster-level coordination mechanism. Each pod in a Kubernetes Deployment runs an independent JVM with its own scheduler instance. When the cron expression fires, all three pods invoke runMonthlyBilling() within milliseconds of each other. The hasCompletedForPeriod() check reads from the database, but all three pods issue the read concurrently — before any pod has written the billing-started record. The SELECT-then-INSERT pattern is not atomic across pods. All three read false, all three enter the billing loop, and all three charge every customer with a different UUID-derived idempotency key.

For Helidon SE applications that use ScheduledExecutorService directly, the same failure occurs through a slightly different mechanism:

// Helidon SE application — UNSAFE: ScheduledExecutorService per JVM
// With replicas: 3, three independent schedulers fire simultaneously.

import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

// Helidon SE 4.x — virtual thread executor for non-blocking Task execution.
// Thread.ofVirtual().factory() creates virtual-thread-backed schedulers.
// BUT: virtual threads do not provide cross-JVM coordination — they are still per-JVM.
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(
        1, Thread.ofVirtual().factory());

// Each pod's JVM starts this timer independently.
// With replicas: 3, three independent timers fire monthly.
scheduler.scheduleAtFixedRate(
        () -> runMonthlyBilling(billingPeriod),
        computeInitialDelay(),  // all pods compute roughly the same delay — fire together
        30, TimeUnit.DAYS);

// runMonthlyBilling calls UUID.randomUUID() per customer.
// Three concurrent calls = three distinct UUIDs per customer = three charges.

Helidon SE 4.x’s use of virtual threads via Thread.ofVirtual().factory() provides efficient concurrency within a single JVM but has no effect on cross-JVM coordination. Virtual threads are a JVM concurrency primitive — a single JVM can multiplex many virtual threads onto fewer platform threads — but each pod runs its own JVM with its own virtual thread pool. The three pods are three separate JVM processes with no shared memory. The scheduler fires on each JVM independently.

The subtler variant: Helidon MP @Initialized(ApplicationScoped.class) observer fires on every pod start — rolling deploy creates a two-pod overlap window — both old and new pod execute the billing loop concurrently

Beyond the cron-based scheduling race, Helidon MP CDI applications can trigger billing from lifecycle observers. A common pattern is to check for pending billing on application startup and process it immediately rather than waiting for the next scheduled firing. This is implemented via a CDI @Observes @Initialized(ApplicationScoped.class) method on an @ApplicationScoped bean:

// UNSAFE: CDI lifecycle observer fires on every pod start.
// Rolling deploy: new pod starts, observer fires on new pod, billing runs.
// OLD pod is still running its billing loop from the cron trigger that fired 2 minutes ago.
// Both pods overlap in the billing window — TOCTOU race — same ch_A/ch_B outcome.

@ApplicationScoped
public class BillingStartupCheck {

    void onStart(@Observes @Initialized(ApplicationScoped.class) Object event,
                 BillingRepository billingRepository,
                 BillingOrchestrator billingOrchestrator) {
        String currentPeriod = java.time.YearMonth.now().toString();

        // Fires on EVERY pod start — including rolling deploy restarts.
        // Rolling deploy timeline:
        //   t=0: old pod 1 runs monthly billing (cron trigger, 01:00 UTC).
        //   t=2m: new pod 2 starts (rolling deploy, Kubernetes replaced old pod 3).
        //   t=2m: new pod 2 observer fires. billingRepository.hasPendingForPeriod() reads false
        //         because old pod 1 hasn't committed billing-complete yet (still processing).
        //   t=2m: new pod 2 starts its own billing run. TOCTOU race begins.
        //   500 customers × 2 pods = 1,000 Stripe charges (with UUID-based keys).
        if (!billingRepository.hasCompletedForPeriod(currentPeriod)) {
            billingOrchestrator.runBillingForPeriod(currentPeriod);
        }
    }
}

The Kubernetes rolling deploy creates a window where both the old pod (still processing its billing run from the cron trigger) and the new pod (just started, observer fired) are concurrently executing the billing loop for the same period. The observer checks hasCompletedForPeriod(), finds false (the old pod hasn’t finished yet), and starts another billing run. The TOCTOU race is identical to the cron-based failure, but the trigger is the pod lifecycle event rather than the cron expression.

The fix for failure mode 3

Cross-pod billing serialization requires a mechanism that operates across JVM process boundaries. The two reliable options are a PostgreSQL advisory lock (no additional library) and ShedLock (a Java library). Content-hash idempotency keys and a pre-flight ON CONFLICT DO NOTHING constraint are required as backstops regardless of which locking mechanism is used:

// Safe: pg_try_advisory_lock() as cross-pod distributed mutex.
// Only the pod that acquires the PostgreSQL advisory lock executes the billing loop.
// Other pods call pg_try_advisory_lock(), receive false, and skip the period.
// Released via pg_advisory_unlock() in a finally block (session-level lock).

@ApplicationScoped
public class SafeBillingScheduler {

    @Inject
    private DataSource dataSource;

    @Inject
    private SafeBillingOrchestrator orchestrator;

    @Scheduled("0 0 1 1 * ?")
    void runMonthlyBilling() {
        String billingPeriod = java.time.YearMonth.now().toString();
        long lockKey = Math.abs(("monthly-billing:" + billingPeriod).hashCode());

        // Session-level advisory lock: held for the duration of the billing run.
        // pg_try_advisory_lock returns true if the lock is acquired, false if held by another session.
        // Non-blocking: returns immediately if the lock cannot be acquired.
        try (Connection conn = dataSource.getConnection()) {
            boolean acquired;
            try (PreparedStatement stmt = conn.prepareStatement("SELECT pg_try_advisory_lock(?)")) {
                stmt.setLong(1, lockKey);
                try (ResultSet rs = stmt.executeQuery()) {
                    acquired = rs.next() && rs.getBoolean(1);
                }
            }

            if (!acquired) {
                // Another pod holds the lock for this billing period.
                log.info("pg_try_advisory_lock for period {} not acquired — another pod is billing",
                        billingPeriod);
                return;
            }

            try {
                orchestrator.runBillingForPeriod(billingPeriod, conn);
            } finally {
                // Release the advisory lock explicitly — do not rely on connection close timing.
                try (PreparedStatement stmt = conn.prepareStatement("SELECT pg_advisory_unlock(?)")) {
                    stmt.setLong(1, lockKey);
                    stmt.executeQuery();
                }
            }
        } catch (Exception e) {
            log.error("Monthly billing failed for period {}", billingPeriod, e);
        }
    }
}

// Safe billing orchestrator: content-hash key + pre-flight ON CONFLICT DO NOTHING.
// Called only after the advisory lock is acquired — but ON CONFLICT remains as backstop
// for rolling deploy overlaps, startup observer races, and operator-triggered billing.

@ApplicationScoped
public class SafeBillingOrchestrator {

    public void runBillingForPeriod(String billingPeriod, Connection lockConn) {
        customerRepository.findAllActive().forEach(customer -> {
            // Safe: content-hash key stable across all pods, all runs, all paths.
            // sha256(customerId:billingPeriod:helidon-billing)[:32] = same 32-char hex
            // on pod 1, pod 2, and pod 3 — Stripe's cache deduplicates concurrent requests.
            String idempotencyKey = stableKey(customer.getId(), billingPeriod);

            // Pre-flight: claim billing slot in the shared database before calling Stripe.
            // ON CONFLICT DO NOTHING is the authoritative cluster-wide billing mutex.
            // If advisory lock fails in an edge case and two pods both reach this point,
            // the UNIQUE (customer_id, billing_period) constraint ensures only one pod
            // successfully inserts the billing_records row — only that pod calls Stripe.
            boolean claimed = billingRepository.claimSlot(
                    customer.getId(), billingPeriod, idempotencyKey);

            if (claimed) {
                try {
                    Charge charge = billingService.chargeCustomer(
                            customer.getId(), billingPeriod,
                            customer.getAmountCents(), idempotencyKey);
                    billingRepository.markComplete(
                            customer.getId(), billingPeriod, charge.getId());
                } catch (StripeException e) {
                    log.error("Billing failed for customer {} in period {}",
                            customer.getId(), billingPeriod, e);
                }
            }
        });
    }

    static String stableKey(String customerId, String billingPeriod) {
        try {
            var digest = java.security.MessageDigest.getInstance("SHA-256");
            var hash = digest.digest(
                    (customerId + ":" + billingPeriod + ":helidon-billing")
                            .getBytes(java.nio.charset.StandardCharsets.UTF_8));
            var sb = new StringBuilder(32);
            for (int i = 0; i < 16; i++) sb.append(String.format("%02x", hash[i]));
            return sb.toString();
        } catch (java.security.NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        }
    }
}

-- Required PostgreSQL schema: UNIQUE constraint on (customer_id, billing_period)
-- enables ON CONFLICT DO NOTHING — the two-layer deduplication mechanism.
-- The advisory lock is layer 1: serializes billing runs across pods.
-- ON CONFLICT DO NOTHING is layer 2: handles advisory lock edge cases.

CREATE TABLE billing_records (
    id              BIGSERIAL PRIMARY KEY,
    customer_id     TEXT NOT NULL,
    billing_period  TEXT NOT NULL,         -- e.g., '2026-08'
    idempotency_key TEXT NOT NULL,
    charge_id       TEXT,                  -- populated after Stripe confirms the charge
    status          TEXT NOT NULL DEFAULT 'in_progress',  -- 'in_progress' | 'completed' | 'failed'
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    CONSTRAINT billing_records_unique UNIQUE (customer_id, billing_period)
);

-- claimSlot implementation (called before Stripe):
INSERT INTO billing_records (customer_id, billing_period, idempotency_key, status)
VALUES ($1, $2, $3, 'in_progress')
ON CONFLICT (customer_id, billing_period) DO NOTHING;
-- Returns true if 1 row inserted (this pod claimed the slot).
-- Returns false if 0 rows inserted (another pod already claimed the slot).

-- markComplete implementation (called after Stripe succeeds):
UPDATE billing_records
SET charge_id = $3, status = 'completed', updated_at = NOW()
WHERE customer_id = $1 AND billing_period = $2;

The pg_try_advisory_lock() call holds a session-level lock for the duration of the billing run. If the pod crashes mid-run, PostgreSQL automatically releases session-level locks when the connection closes. The finally block releases the lock explicitly when the run completes normally. The claimSlot ON CONFLICT DO NOTHING pattern handles the edge case where the advisory lock is released mid-run (network partition causing the connection to close while the billing loop is still executing) — in that case, a second pod acquires the lock and restarts the billing loop, but ON CONFLICT DO NOTHING skips every customer that the first pod already claimed, and the content-hash idempotency key ensures the second pod’s Stripe request for those customers returns the cached charge rather than creating a new one.

For the @Initialized(ApplicationScoped.class) observer race, the same advisory lock pattern closes the window: the startup observer calls pg_try_advisory_lock() before checking hasCompletedForPeriod(). If the cron-triggered billing pod holds the lock, the observer on the new pod returns immediately without starting a second billing run.

Gap analysis: other Helidon billing patterns not covered above

The three failure modes above cover the most common Helidon retry and scheduling surfaces. Several adjacent patterns introduce the same class of failure through different mechanisms:

Helidon MP MicroProfile REST Client @Retry with a request body that includes a UUID field — when @Retry is on a MicroProfile REST client interface method and the request body object is constructed before the interface method call (outside the @Retry boundary), any UUID inside the body object is safely computed once. But if the request body is constructed inside the interface method’s CDI proxy via an @Interceptor or a Helidon JAX-RS WriterInterceptor that adds a UUID field to the serialized JSON, that UUID generation re-runs on each @Retry attempt. The fix: construct the request body — including any UUID fields — in the caller before invoking the REST client interface method.

Helidon MP @Retry on a @Transactional method — CDI interceptors have a defined application order in Jakarta EE. Helidon’s MicroProfile FT interceptors and the JTA transaction interceptor both operate on the CDI proxy. Depending on the @Priority values, @Retry may wrap the transaction boundary (retry fires after transaction commit) or be inside it (retry fires before transaction commit). If @Retry wraps @Transactional, a transaction that commits a billing record but then throws from post-commit code causes @Retry to re-invoke the method body — the transaction re-runs, hits the UNIQUE (customer_id, billing_period) constraint, and throws a ConstraintViolationException that @Retry may also retry. The fix: ensure @Retry is scoped to the Stripe call only, not to the entire transaction; or ensure that the content-hash idempotency key and ON CONFLICT DO NOTHING backstop are present to handle the constraint collision gracefully.

Helidon SE 4.x WebClient with a retry policy and a header supplier — Helidon SE 4.x’s Http1Client supports WebClientServiceRequest interceptors that can add or modify headers before each request. If an interceptor adds an Idempotency-Key header by calling a supplier or factory that generates a UUID, and the client is configured with a retry policy, the interceptor runs on each retry attempt and generates a new UUID each time. The fix: compute the stable key before the first request and store it in the Http1ClientRequest’s headers directly, without delegating to a header supplier that re-evaluates on each attempt.

Helidon MP @Bulkhead combined with @Retry — @Bulkhead limits concurrent calls to the annotated method. @Retry re-invokes the method on exception. If a billing thread exhausts the bulkhead queue (max-wait-time exceeded), a BulkheadException is thrown. If retryOn includes BulkheadException, the @Retry interceptor re-invokes the method body when bulkhead capacity is restored. Any UUID inside the method body generates a new value. Because the BulkheadException fires before the Stripe call is made (while the thread is waiting for a bulkhead slot), no charge exists in Stripe for the initial attempt — the retry is safe from a duplicate-charge perspective. But if BulkheadException is not in retryOn and the exception propagates to the caller, and the caller independently retries by calling the billing service again, the caller’s retry creates a new UUID and potential duplicate charge. The fix is the same: stable content-hash key computed in the caller.

Summary

The pattern connecting all three: the idempotency key must be derived from the business intent of the billing operation — customer ID, billing period, vendor namespace — not from any ephemeral runtime value that re-evaluates between @Retry invocations, loop iterations, forked callables, or concurrent pod executions. UUID.randomUUID(), System.currentTimeMillis() at method entry, a @ClientHeaderParam method reference that calls UUID generation, and any per-virtual-thread-startup value all produce different values on each invocation, iteration, fork, or pod. A key derived from sha256(customerId + ":" + billingPeriod + ":helidon-billing")[:32] produces the same 32-character hex string on every InvocationContext.proceed() call, on every loop iteration, on every forked callable, and on every pod — stable across all three failure modes. Backing it with a PostgreSQL-level UNIQUE (customer_id, billing_period) constraint and a pg_try_advisory_lock() distributed mutex moves the deduplication guarantee out of ephemeral per-JVM state and into durable shared storage that survives reboots, rolling deploys, connection drops, and multi-pod races.

The vault key spend cap adds a hard financial boundary as a last line of defence: a per-billing-period vault key issued with a max_amount set to expected_total × 1.10 caps the maximum spend that any combination of bugs, retries, and multi-pod races can produce. Once the cap is reached, Stripe rejects further charges with a 402 Payment Required on the vault key, containing the blast radius to a known maximum regardless of how many @Retry invocations, loop iterations, or concurrent billing pods are in flight.