Micronaut and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Micronaut’s AOP-first compile-time design means @Retryable re-invokes the entire intercepted method body on each retry attempt. Three Micronaut-specific Stripe billing failure modes all follow from this fact: a service method annotated with @Retryable that calls UUID.randomUUID() inside the method body generates a fresh idempotency key on each retry and charges the customer twice; a reactive variant returning Mono<T> causes Micronaut’s ReactiveRetryInterceptor to re-subscribe the Publisher and re-evaluate any defer factory that contains the UUID call; and a @Scheduled billing job with no cluster coordination fires concurrently on all Kubernetes replicas, each generating its own UUID and sending a separate Stripe charge.

This post covers all three failure modes with Kotlin code, content-hash idempotency keys stable across @Retryable re-invocations, Publisher re-subscriptions, and multi-pod scheduler firings, pre-flight PostgreSQL ON CONFLICT DO NOTHING checks — and per-billing-period vault keys via a spend-cap proxy as a hard backstop. For AOP retry failure modes in a different JVM ecosystem, see the Akka HTTP and Stripe Integration post. For retry interceptor failure modes at the RPC transport layer, see the gRPC and Stripe Integration post.

Failure mode 1: @Retryable AOP interceptor re-invokes the annotated method body — UUID.randomUUID() inside the method generates a new idempotency key per retry attempt — the first attempt created ch_A — the retry creates ch_B

Micronaut’s @Retryable annotation (from io.micronaut.retry.annotation.Retryable) is implemented via compile-time AOP. At application startup, Micronaut generates a proxy class for the bean. The proxy’s chargeBilling method delegates to DefaultRetryInterceptor.intercept(context), which calls context.proceed() to execute the original method. On any exception that matches the includes list, the interceptor waits for the configured delay and calls context.proceed() again. The method body re-executes in full. Any UUID.randomUUID() inside the method body produces a different value on each invocation:

// Micronaut + Kotlin — billing service with @Retryable
// UNSAFE: UUID.randomUUID() computed inside the annotated method body
import io.micronaut.http.HttpRequest
import io.micronaut.http.client.HttpClient
import io.micronaut.retry.annotation.Retryable
import jakarta.inject.Inject
import jakarta.inject.Singleton
import java.net.SocketTimeoutException
import java.util.UUID

@Singleton
class BillingService @Inject constructor(
    @param:io.micronaut.http.client.annotation.Client("https://api.stripe.com")
    private val stripeClient: HttpClient,
    private val stripeKey: String
) {

    @Retryable(
        attempts = "3",
        delay = "1s",
        multiplier = "2.0",
        includes = [SocketTimeoutException::class, io.micronaut.http.client.exceptions.HttpClientResponseException::class]
    )
    fun chargeBilling(customerId: String, billingPeriod: String, amountCents: Long): ChargeResponse {

        // UNSAFE: UUID.randomUUID() called inside the @Retryable method body.
        // Micronaut's DefaultRetryInterceptor calls context.proceed() on each retry.
        // context.proceed() re-executes this method body from the top.
        // Attempt 1: key = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Attempt 2: key = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"  ← different
        // Attempt 3: key = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"  ← different again
        val idempotencyKey = UUID.randomUUID().toString()

        val requestBody = mapOf(
            "amount"      to amountCents,
            "currency"    to "usd",
            "customer"    to customerId,
            "description" to "Billing period $billingPeriod"
        )

        val request = HttpRequest.POST("/v1/charges", requestBody)
            .header("Authorization", "Bearer $stripeKey")
            .header("Idempotency-Key", idempotencyKey)

        return stripeClient.toBlocking()
            .retrieve(request, ChargeResponse::class.java)
    }
}

The failure scenario: the agent calls billingService.chargeBilling("cust_123", "2026-08", 9900). Micronaut’s generated proxy intercepts the call. DefaultRetryInterceptor.intercept(context) calls context.proceed(). The method body executes. UUID.randomUUID() returns "3f7a9b2c...". The HttpClient sends a POST /v1/charges request to Stripe with Idempotency-Key: 3f7a9b2c.... Stripe begins processing: the charge object ch_A is created and the card is authorized. Before Stripe returns the 201 response, the network socket times out — perhaps because the agent’s Kubernetes pod briefly lost connectivity to Stripe’s API, or because the Micronaut HTTP client’s read timeout of 20 seconds fired while Stripe was performing fraud scoring. Micronaut’s HttpClient throws a SocketTimeoutException. DefaultRetryInterceptor catches it, sees that it matches the includes list, waits 1 second, and calls context.proceed() again. The method body re-executes from the top. UUID.randomUUID() returns "b8d2e4f6...". A second POST /v1/charges is sent with Idempotency-Key: b8d2e4f6.... Stripe has never seen this key. Stripe creates ch_B. Customer 123 is charged $99 twice for August 2026.

The failure is also triggered by any HttpClientResponseException with a status that Micronaut treats as retryable — such as a 429 Too Many Requests (rate limit) or a 500 Internal Server Error from Stripe’s gateway — and by any transient ConnectException on the HTTP client side. In all cases, the method body re-executes and a new UUID is generated. If ch_A was created before the error, the retry creates ch_B.

A subtler variant occurs when the UUID is passed as a parameter from the caller into the @Retryable method, but the caller also has @Retryable on its own method. If the outer method generates the UUID and passes it to the inner method, and the outer method retries, a new UUID is generated at the outer level and a different one is passed to the inner method on each retry. The fix must be at the level where the UUID is first computed — not just inside the @Retryable-annotated leaf method.

The fix for failure mode 1

The idempotency key must be derived deterministically from the business inputs of the billing operation — the customer ID and the billing period — rather than computed randomly inside the retryable method. A content-hash of these fields is stable across context.proceed() re-invocations because the same inputs always produce the same output:

// Safe: content-hash key derived from business fields only
import java.security.MessageDigest

fun stableKey(customerId: String, billingPeriod: String): String {
    val input = "$customerId:$billingPeriod:micronaut-billing"
    val digest = MessageDigest.getInstance("SHA-256")
        .digest(input.toByteArray(Charsets.UTF_8))
    return digest.take(16).joinToString("") { "%02x".format(it) }  // 32-char hex
}

// What to EXCLUDE from the key — any value that changes between method invocations:
//   UUID.randomUUID()                  ← random, different each time
//   System.currentTimeMillis()         ← different if retry happens 1s later
//   System.nanoTime()                  ← different per JVM, per invocation
//   AtomicInteger.incrementAndGet()    ← different per invocation
//   Thread.currentThread().id          ← different if retry runs on a different thread
//   this.hashCode()                    ← stable per instance but useless for deduplication

// Safe @Retryable billing method — content-hash key is identical across all retries
@Singleton
class SafeBillingService @Inject constructor(
    @param:io.micronaut.http.client.annotation.Client("https://api.stripe.com")
    private val stripeClient: HttpClient,
    private val stripeKey: String,
    private val billingRepository: BillingRepository
) {

    @Retryable(
        attempts = "3",
        delay = "1s",
        multiplier = "2.0",
        includes = [SocketTimeoutException::class,
                    io.micronaut.http.client.exceptions.HttpClientResponseException::class]
    )
    fun chargeBilling(customerId: String, billingPeriod: String, amountCents: Long): ChargeResponse {

        // Safe: same inputs → same key on every context.proceed() invocation.
        val idempotencyKey = stableKey(customerId, billingPeriod)

        // Pre-flight: claim the billing slot before touching Stripe.
        // If the row already exists (prior attempt created it), return the cached result.
        val existing = billingRepository.findByCustomerAndPeriod(customerId, billingPeriod)
        if (existing?.chargeId != null) {
            return ChargeResponse(id = existing.chargeId, status = "succeeded")
        }
        billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)
            ?: return billingRepository.findByCustomerAndPeriod(customerId, billingPeriod)!!
                .let { ChargeResponse(id = it.chargeId!!, status = "succeeded") }

        val requestBody = mapOf(
            "amount"      to amountCents,
            "currency"    to "usd",
            "customer"    to customerId,
            "description" to "Billing period $billingPeriod"
        )

        val request = HttpRequest.POST("/v1/charges", requestBody)
            .header("Authorization", "Bearer $stripeKey")
            .header("Idempotency-Key", idempotencyKey)

        val response = stripeClient.toBlocking()
            .retrieve(request, ChargeResponse::class.java)

        billingRepository.markComplete(customerId, billingPeriod, response.id)
        return response
    }
}

-- Pre-flight table: INSERT ... ON CONFLICT DO NOTHING as durable billing mutex
CREATE TABLE billing_records (
    customer_id      TEXT        NOT NULL,
    billing_period   TEXT        NOT NULL,
    idempotency_key  TEXT        NOT NULL,
    status           TEXT        NOT NULL DEFAULT 'pending',
    charge_id        TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT billing_records_pk PRIMARY KEY (idempotency_key),
    CONSTRAINT billing_records_uq UNIQUE (customer_id, billing_period)
);

-- claimSlot(): returns the inserted row (first attempt wins) or null (row already exists)
INSERT INTO billing_records (customer_id, billing_period, idempotency_key)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id, billing_period) DO NOTHING
RETURNING idempotency_key;

-- markComplete(): write the charge_id after Stripe responds
UPDATE billing_records
   SET status    = 'completed',
       charge_id = $3
 WHERE customer_id    = $1
   AND billing_period = $2;

With the content-hash key, every call to context.proceed() — whether it is the original attempt or the third retry — computes stableKey("cust_123", "2026-08") and arrives at the same "a1b2c3d4..." string. The Stripe request always carries the same Idempotency-Key. If ch_A was created by the first attempt before the socket timeout fired, Stripe’s idempotency cache returns ch_A on the retry without charging again. If the billing_records pre-flight row exists, the service returns the cached result immediately and does not call Stripe at all.

Failure mode 2: reactive @Retryable on a method returning Mono<T> — ReactiveRetryInterceptor re-invokes context.proceed() on each subscription — UUID.randomUUID() inside Mono.defer or at the top of the method body evaluates fresh per retry

Micronaut supports reactive programming via Project Reactor (Mono, Flux) and RxJava (Single, Observable, Flowable). When a method annotated with @Retryable returns a Publisher<T> such as Mono<ChargeResponse>, Micronaut’s ReactiveRetryInterceptor — rather than calling context.proceed() synchronously in a loop — wraps the returned publisher in a retry operator. The key detail is how re-invocation works: on each error-triggered retry, the interceptor calls context.proceed() again to get a fresh Publisher from the method, then subscribes to it. This means the method body re-runs on each subscription, including any code that precedes the return Mono.defer { ... } call and any code inside the defer lambda itself:

// Micronaut + Kotlin — reactive billing service returning Mono
// UNSAFE: UUID.randomUUID() inside Mono.defer{} re-evaluates on each subscription
import io.micronaut.http.HttpRequest
import io.micronaut.http.client.HttpClient
import io.micronaut.retry.annotation.Retryable
import jakarta.inject.Singleton
import reactor.core.publisher.Mono
import java.util.UUID

@Singleton
class ReactiveBillingService(
    @io.micronaut.http.client.annotation.Client("https://api.stripe.com")
    private val stripeClient: HttpClient,
    private val stripeKey: String
) {

    @Retryable(
        attempts = "3",
        delay = "500ms",
        multiplier = "2.0"
    )
    fun chargeBilling(customerId: String, billingPeriod: String, amountCents: Long): Mono<ChargeResponse> {

        // Pattern A — UUID computed before the Mono, but inside the @Retryable method body.
        // UNSAFE: context.proceed() re-runs the method body on each retry.
        // Attempt 1: idempotencyKey = "3f7a9b2c..."
        // Attempt 2: idempotencyKey = "b8d2e4f6..."  ← method body re-executed
        val idempotencyKey = UUID.randomUUID().toString()

        return Mono.fromCallable {
            val request = HttpRequest.POST("/v1/charges", mapOf(
                "amount"   to amountCents,
                "currency" to "usd",
                "customer" to customerId
            ))
                .header("Authorization", "Bearer $stripeKey")
                .header("Idempotency-Key", idempotencyKey)
            stripeClient.toBlocking().retrieve(request, ChargeResponse::class.java)
        }
    }

    @Retryable(attempts = "3", delay = "500ms", multiplier = "2.0")
    fun chargeBillingDeferred(customerId: String, billingPeriod: String, amountCents: Long): Mono<ChargeResponse> {

        // Pattern B — UUID inside Mono.defer{}.
        // UNSAFE: Mono.defer{} factory re-evaluates when ReactiveRetryInterceptor
        // re-subscribes. Same effect: new UUID per retry.
        return Mono.defer {
            val idempotencyKey = UUID.randomUUID().toString()  // new value per subscription

            val request = HttpRequest.POST("/v1/charges", mapOf(
                "amount"   to amountCents,
                "currency" to "usd",
                "customer" to customerId
            ))
                .header("Authorization", "Bearer $stripeKey")
                .header("Idempotency-Key", idempotencyKey)

            Mono.fromCallable {
                stripeClient.toBlocking().retrieve(request, ChargeResponse::class.java)
            }
        }
    }
}

The failure scenario for Pattern A: a subscriber calls reactiveBillingService.chargeBilling("cust_123", "2026-08", 9900).block(). Micronaut’s ReactiveRetryInterceptor intercepts the call. It calls context.proceed() to get the Mono<ChargeResponse> from the method. The method body executes. UUID.randomUUID() computes "3f7a9b2c...". The Mono.fromCallable captures this key in its closure. The interceptor subscribes to the Mono. The callable executes: HttpClient sends the POST to Stripe with Idempotency-Key: 3f7a9b2c.... Stripe begins processing ch_A. The read socket times out after 20 seconds. The Mono terminates with a SocketTimeoutException. The interceptor catches the error. It determines that the error is retryable. It waits 500ms. It calls context.proceed() again to get a fresh publisher from the method. The method body re-executes. UUID.randomUUID() now computes "b8d2e4f6..." — a completely different value, because the closure from the first Mono.fromCallable is discarded and a new one is assembled. The interceptor subscribes to the new Mono. Stripe receives a second POST with Idempotency-Key: b8d2e4f6.... Stripe has never seen this key. ch_B is created.

Pattern B is identical in effect but the UUID is inside Mono.defer{}. The defer factory runs on each subscription. The interceptor re-subscribes on each retry by calling context.proceed() and then subscribing to the returned publisher. If the outer method returns a Mono.defer{}, the defer factory re-evaluates on that subscription, producing a new UUID. The failure is the same: ch_A on the first subscription, ch_B on the retry subscription.

A more subtle variant: the method returns a cold Mono assembled from a non-defer chain, but the UUID is captured as a method-local variable at the top of the method. Pattern A already shows this. The important point is that “method-local variable at the top of the method” does not survive a context.proceed() re-invocation — the old closure is gone and a new one is created. The UUID must be either derived deterministically from the arguments or computed outside the method in a caller that is not itself retryable.

The fix for failure mode 2

The same content-hash approach applies to reactive methods. The key insight is that the inputs to stableKey() are the method parameters, which do not change between context.proceed() re-invocations — only the UUID does. Replace UUID.randomUUID() with stableKey(customerId, billingPeriod) and the key is identical whether the method is invoked once or three times:

// Safe reactive billing service — content-hash key stable across all re-subscriptions
@Singleton
class SafeReactiveBillingService(
    @io.micronaut.http.client.annotation.Client("https://api.stripe.com")
    private val stripeClient: HttpClient,
    private val stripeKey: String,
    private val billingRepository: BillingRepository
) {

    @Retryable(
        attempts = "3",
        delay = "500ms",
        multiplier = "2.0"
    )
    fun chargeBilling(customerId: String, billingPeriod: String, amountCents: Long): Mono<ChargeResponse> {

        // Safe: stableKey() is a pure function of the method parameters.
        // context.proceed() re-invocation produces the same key every time.
        val idempotencyKey = stableKey(customerId, billingPeriod)

        return Mono.fromCallable {
            // Pre-flight: skip Stripe if billing was already completed.
            billingRepository.findByCustomerAndPeriod(customerId, billingPeriod)
                ?.chargeId
                ?.let { return@fromCallable ChargeResponse(id = it, status = "succeeded") }

            billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)

            val request = HttpRequest.POST("/v1/charges", mapOf(
                "amount"   to amountCents,
                "currency" to "usd",
                "customer" to customerId
            ))
                .header("Authorization", "Bearer $stripeKey")
                .header("Idempotency-Key", idempotencyKey)

            val response = stripeClient.toBlocking()
                .retrieve(request, ChargeResponse::class.java)

            billingRepository.markComplete(customerId, billingPeriod, response.id)
            response
        }
    }
}

An additional safeguard for reactive pipelines: if the method returns a Mono or Flux, do not include any per-subscription state in the idempotency key. Common anti-patterns include using Mono.deferContextual { ctx -> ... UUID based on ctx.get("requestId") ... } where the requestId is set to a new UUID in the reactive context per subscription, or computing the key from Schedulers.single().now() captured inside the pipeline. The pre-flight INSERT ... ON CONFLICT DO NOTHING is the authoritative guard regardless of what happens in the reactive chain — even if a new key somehow reaches the server, the database constraint prevents a second billing row from being inserted for the same (customer_id, billing_period) pair.

Failure mode 3: @Scheduled(fixedDelay = “30d”) billing job runs in a Micronaut application with replicas: 3 — no cross-pod coordination — three concurrent billing runs generate different UUIDs per pod — ch_A, ch_B, and ch_C created for every customer

Micronaut’s @Scheduled annotation creates a scheduled task backed by a ScheduledExecutorService per JVM. There is no built-in cluster-level coordination between pods. Unlike Quartz with a JDBC JobStore (which uses database row locking to guarantee that exactly one node acquires a trigger) or ShedLock (which inserts a lock row before task execution), Micronaut’s default scheduler is entirely in-process. In a Kubernetes Deployment with replicas: 3, all three pods have their own independent scheduler. If all three pods start at approximately the same time — which is typical for a rolling deploy that replaces all pods within a few minutes — their fixedDelay timers fire at approximately the same wall-clock time. Each pod executes the scheduled billing method independently, generating its own UUID.randomUUID() for each customer’s charge:

// Micronaut + Kotlin — @Scheduled billing job
// UNSAFE: fires independently on every pod; UUID generated per pod per customer
import io.micronaut.scheduling.annotation.Scheduled
import jakarta.inject.Inject
import jakarta.inject.Singleton
import java.util.UUID

@Singleton
class BillingScheduler @Inject constructor(
    private val customerRepository: CustomerRepository,
    private val billingService: BillingService
) {

    // UNSAFE: fixedDelay timer is per-JVM.
    // In a Deployment with replicas:3 all three schedulers fire within seconds.
    // No coordination: pg_try_advisory_lock, Kubernetes Lease, or ShedLock.
    @Scheduled(fixedDelay = "30d", initialDelay = "60s")
    fun runMonthlyBilling() {
        val billingPeriod = currentBillingPeriod()
        val customers = customerRepository.findAllDueForBilling(billingPeriod)

        for (customer in customers) {
            // UUID.randomUUID() called inside the scheduled method per customer.
            // Pod A charges customer 1 with idempotencyKey = "3f7a9b2c..."
            // Pod B charges customer 1 with idempotencyKey = "b8d2e4f6..."  ← different JVM
            // Pod C charges customer 1 with idempotencyKey = "1c9d3e5a..."  ← different JVM
            // All three reach Stripe before any of them have committed a billing_records row.
            // ch_A, ch_B, ch_C created. Customer charged three times.
            val idempotencyKey = UUID.randomUUID().toString()

            billingService.chargeCustomer(
                customerId     = customer.id,
                billingPeriod  = billingPeriod,
                amountCents    = customer.planAmountCents,
                idempotencyKey = idempotencyKey
            )
        }
    }

    private fun currentBillingPeriod(): String {
        val now = java.time.YearMonth.now()
        return "${now.year}-${now.monthValue.toString().padStart(2, '0')}"
    }
}

The failure scenario in detail: the agent application is deployed as a Kubernetes Deployment with replicas: 3 and a RollingUpdate strategy. All three pods start within a 90-second window. Each pod’s Micronaut application context initializes its own ScheduledExecutorService and registers the BillingScheduler.runMonthlyBilling() task with an initialDelay of 60 seconds and a fixedDelay of 30 days. 60 seconds after the last pod starts, pod A fires the task. Pod B fires it approximately 8 seconds later. Pod C approximately 12 seconds after that — within the window of a typical Stripe API call duration of 2–4 seconds.

Pod A iterates over all due customers. For customer cust_123, pod A generates UUID = "3f7a9b2c..." and sends POST /v1/charges to Stripe with Idempotency-Key: 3f7a9b2c.... Stripe begins creating ch_A. Before Stripe’s response returns to pod A, pod B reaches customer cust_123 in its own loop. Pod B generates UUID = "b8d2e4f6..." and sends its own POST /v1/charges with a different key. Stripe has not yet committed ch_A (still processing). Stripe creates ch_B. Pod C follows with UUID = "1c9d3e5a..." — ch_C is created. All three requests return 201 Created. Three charge objects exist in Stripe for customer 123 for the same billing period. Customer 123 is charged $99 three times.

The failure is more insidious than the retry failure modes above because there is no retry — each pod believes it is the only one performing billing. The duplicate charges are only discovered when a customer calls support, a credit card chargeback is filed, or a periodic reconciliation query detects three billing_records rows (if they existed) for the same customer and period. In practice, without pre-flight DB coordination, the rows do not exist at all: each pod charges Stripe directly and the duplicates are invisible at the application layer until the Stripe dashboard is checked.

The fix for failure mode 3

There are two layers to the fix: cluster-level execution serialization (so only one pod runs the billing loop at a time) and content-hash idempotency keys (so even if two pods somehow overlap, Stripe’s cache and the pre-flight DB guard deduplicate the charge).

For cluster-level serialization, PostgreSQL advisory locks provide a lightweight, connection-scoped mutex that does not require a separate coordination service:

// Safe @Scheduled billing: pg_try_advisory_lock for cluster-level serialization
@Singleton
class SafeBillingScheduler @Inject constructor(
    private val customerRepository: CustomerRepository,
    private val billingService: SafeBillingService,
    private val dataSource: javax.sql.DataSource
) {

    @Scheduled(fixedDelay = "30d", initialDelay = "60s")
    fun runMonthlyBilling() {
        val billingPeriod = currentBillingPeriod()

        // pg_try_advisory_lock returns true on the pod that acquires the lock,
        // false on all others. Lock is released when the connection is closed
        // (end of this method's @Transactional scope, or explicit unlock).
        val lockKey = "billing:$billingPeriod".hashCode().toLong()

        dataSource.connection.use { conn ->
            val acquired = conn.prepareStatement(
                "SELECT pg_try_advisory_lock(?)"
            ).use { stmt ->
                stmt.setLong(1, lockKey)
                stmt.executeQuery().use { rs ->
                    rs.next() && rs.getBoolean(1)
                }
            }

            if (!acquired) {
                // Another pod is already running billing for this period.
                // Exit immediately — do not charge any customers.
                return
            }

            try {
                val customers = customerRepository.findAllDueForBilling(billingPeriod)
                for (customer in customers) {
                    // Content-hash key — stable and pod-independent.
                    billingService.chargeBilling(
                        customerId    = customer.id,
                        billingPeriod = billingPeriod,
                        amountCents   = customer.planAmountCents
                    )
                }
            } finally {
                conn.prepareStatement("SELECT pg_advisory_unlock(?)").use { stmt ->
                    stmt.setLong(1, lockKey)
                    stmt.execute()
                }
            }
        }
    }

    private fun currentBillingPeriod(): String {
        val now = java.time.YearMonth.now()
        return "${now.year}-${now.monthValue.toString().padStart(2, '0')}"
    }
}

-- Alternatively: use ShedLock with Micronaut's @SchedulerLock annotation
-- ShedLock inserts a row in shedlock table and holds it for the task duration.
-- Only one node can acquire the lock for a given task name + lock-at-most-for duration.

-- Add to build.gradle.kts:
-- implementation("net.javacrumbs.shedlock:shedlock-micronaut:5.x.x")
-- implementation("net.javacrumbs.shedlock:shedlock-provider-jdbc-template:5.x.x")

-- Create the shedlock table:
CREATE TABLE shedlock (
    name       VARCHAR(64)  NOT NULL,
    lock_until TIMESTAMP    NOT NULL,
    locked_at  TIMESTAMP    NOT NULL,
    locked_by  VARCHAR(255) NOT NULL,
    PRIMARY KEY (name)
);

// With ShedLock + Micronaut (using shedlock-micronaut integration):
import net.javacrumbs.shedlock.core.SchedulerLock

@Scheduled(fixedDelay = "30d", initialDelay = "60s")
@SchedulerLock(
    name = "monthly-billing",
    lockAtMostFor = "PT4H",     // release lock after 4h even if the task hangs
    lockAtLeastFor = "PT30M"    // hold lock for at least 30m to prevent rapid re-fire
)
fun runMonthlyBilling() {
    val billingPeriod = currentBillingPeriod()
    val customers = customerRepository.findAllDueForBilling(billingPeriod)
    for (customer in customers) {
        billingService.chargeBilling(customer.id, billingPeriod, customer.planAmountCents)
    }
}

The content-hash idempotency key remains essential even with advisory locks or ShedLock. The lock prevents concurrent runs under normal conditions, but does not protect against a pod that acquires the lock, charges half the customers, and then crashes — the next pod to acquire the lock (after lockAtMostFor expires) will attempt to bill all customers again. Without a content-hash key and a pre-flight UNIQUE (customer_id, billing_period) constraint, customers who were already charged in the first run will be charged again in the recovery run. With the content-hash key and the pre-flight guard, the recovery run is a safe no-op for already-charged customers.

Kubernetes leader election (via the coordination.k8s.io/v1 Lease resource) is an alternative to advisory locks that does not require a database connection. A Micronaut application can use the Fabric8 Kubernetes client or the official Java client to acquire a Lease before running the scheduled task. The tradeoff is that Kubernetes RBAC must allow the pod’s service account to create and update Lease objects in the target namespace — an additional permission many security-conscious teams prefer to avoid.

What all three failure modes share

All three Micronaut-specific failure modes above share a root cause: the idempotency key is computed at execution time, inside the scope being retried or parallelized, rather than derived statically from the billing intent. The framework’s retry mechanism — whether DefaultRetryInterceptor calling context.proceed(), ReactiveRetryInterceptor re-subscribing the publisher, or the scheduler firing on multiple pods — creates a second execution context that independently computes a second UUID. The solution is always to push the key derivation upstream of the re-entrant scope:

• For @Retryable: derive the key from the method parameters using a deterministic hash, not from UUID.randomUUID() inside the method body.
• For reactive @Retryable: same rule — the key must be derived from the parameters visible to the method signature before any publisher is assembled or deferred.
• For @Scheduled: derive the key from the billing period (a shared, deterministic value) so that if two pods do overlap, they produce the same key. Add cluster-level serialization so overlapping runs do not happen in the first place.

Gap analysis: other Micronaut patterns that produce the same failure

Micronaut Retry with @CircuitBreaker and @Fallback. A @CircuitBreaker-annotated billing method that opens after consecutive failures routes calls to a @Fallback implementation. If the fallback method generates its own UUID.randomUUID() to attempt a secondary Stripe call (for example, charging against a backup payment method on file), and the original call to the primary Stripe key had already created ch_A before the circuit opened, the fallback creates ch_B on the backup. The fallback must either share the same content-hash key from the original call or have its own distinct pre-flight check keyed on (customer_id, billing_period, payment_method).

Micronaut @EventListener on a billing event. A service calls applicationEventPublisher.publishEventAsync(BillingRequestedEvent(customerId, billingPeriod, UUID.randomUUID())). The UUID is captured in the event at publish time. If the event listener throws an exception and the event publisher is configured to retry delivery (via an external queue like RabbitMQ or Kafka with DLQ re-enqueue), the event is re-delivered with the same UUID — which is safe. But if the retry mechanism re-publishes a new event with a new UUID.randomUUID() at re-publish time (for example, a retry scheduled task that reads unacknowledged billing requests and re-emits them), the new event carries a different UUID and creates ch_B. The UUID must be stored in the original billing request record in the database, not re-generated at re-publish time.

Micronaut HTTP Client declarative client with automatic retry on connection failure. Micronaut’s @Client interface supports configuration-based retry via micronaut.http.services.stripe.connection-pool-idle-timeout and connection-level retry in the underlying Netty pool. If the Stripe HTTP client connection pool recycles a stale connection mid-request and retries automatically at the transport layer (before the application-level @Retryable interceptor sees an exception), the request is re-sent on a new connection. If the Idempotency-Key header was set with a stable content-hash before the request was issued, the transport-level retry is safe. If it was set with a UUID.randomUUID() in the HttpRequest.headers() builder, the retry carries the same in-memory value — because the HttpRequest object is reused by the transport retry, not rebuilt. This is actually safe for transport-level retry, unlike the AOP retry which re-runs the method. But the distinction is fragile: if the @Retryable interceptor above the transport retry also fires, it rebuilds the HttpRequest with a new UUID. The two layers interact in ways that are hard to audit without instrumentation.

Micronaut @Scheduled with fixedRate instead of fixedDelay. fixedDelay measures the delay from the end of the previous execution. fixedRate measures from the start. If a billing job takes longer than the fixedRate interval (for example, billing 10,000 customers takes 45 minutes and fixedRate = "30m"), Micronaut will fire a second concurrent execution of the same method on the same pod while the first is still running. This is equivalent to the multi-replica problem, but within a single JVM. The pre-flight ON CONFLICT DO NOTHING guard and the content-hash key prevent double-charges in this case, but the advisory lock approach above must acquire the lock at the start of each execution and check for a pre-existing run from the same pod as well as from other pods.

Micronaut data transactions with @Transactional retry on serialization failure. Some Micronaut Data configurations automatically retry transactions that fail with a PostgreSQL ERROR: could not serialize access due to concurrent update. If the billing method is annotated with both @Retryable and @Transactional, a serialization failure triggers the retry interceptor, which calls context.proceed() and re-opens a new transaction. Any UUID.randomUUID() inside the transaction body generates a new key. The fix is the same: content-hash key, and verify that the Stripe call happens inside the transaction so the pre-flight INSERT and the Stripe call are atomic with respect to the serialization domain (noting that Stripe is an external system and true atomicity is not achievable — the pre-flight guard is the closest approximation).

The spend-cap vault key as a hard backstop

Content-hash keys and pre-flight database guards are defense-in-depth for production Micronaut services. But an autonomous agent running a @Scheduled billing loop has an additional failure mode: an accounting bug in the billing logic causes the agent to schedule the same customer for billing twice in a single period, with two different billingPeriod strings (for example, "2026-08" and "2026-08-01") that produce two different content-hash keys. The database constraint on UNIQUE (customer_id, billing_period) does not catch this because the two strings are different. Stripe’s idempotency cache does not catch this because the keys are different. Two charges are created from the correct billing code, not from a retry bug.

A spend-cap vault key closes this gap. The agent uses a vault_key_xxx issued by a proxy rather than the bare Stripe restricted key. The vault key has a policy: daily_usd_cap: 10000 (the maximum revenue the agent should process on any given day). If the accounting bug causes the agent to charge a customer twice for the same billing period — even with different period strings and different idempotency keys — the total spend reported by the proxy’s audit log will exceed the cap. The proxy rejects the second charge with a 402 Payment Required response carrying a X-Keybrake-Reason: daily_cap_exceeded header before the request reaches Stripe. The HttpClientResponseException bubbles up through @Retryable as a non-retryable error (it is excluded from the includes list) and the agent logs the cap violation for manual review.

The vault key also handles the case where the advisory lock implementation has a bug — a missed pg_advisory_unlock() in an error path, a Kubernetes pod that restarts while holding a Lease, or a ShedLock row that was not cleaned up after a deployment. In all of these scenarios, if two pods do manage to run the billing loop concurrently, the spend cap is the last line of defense before money leaves the customer’s account.

Summary

The pattern across all three: push key derivation outside the re-entrant scope, make it a deterministic function of the business intent, and use a database constraint as the authoritative deduplication layer. The vault key spend cap adds a hard boundary around the total financial exposure regardless of how many retries, subscriptions, or pods execute concurrently.