Ktor Client and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Ktor’s HttpRequestRetry plugin and Kotlin coroutine-based retry loops both make retry feel natural in Ktor applications, but whether a retry produces a safe Stripe response or a duplicate charge depends on exactly where the idempotency key is computed relative to each retry attempt. Three Ktor-specific failure modes: the modifyRequest callback runs only before retry attempts, not the initial request — a key generated inside modifyRequest means the initial attempt has no idempotency key at all, and each retry carries a different one; a manual retry loop that re-calls a billing suspend fun executes UUID.randomUUID() fresh at the top of every invocation; and Application.launch {} starts one independent billing coroutine per Kubernetes replica with no cross-pod coordination, so three pods each generate distinct UUIDs per customer and charge the same customer three times.

This post covers all three failure modes with Kotlin and Ktor 2.3.x code, content-hash idempotency keys stable across modifyRequest overwrites, function re-invocations, and multi-pod concurrent billing loops, pre-flight PostgreSQL ON CONFLICT DO NOTHING as a cluster-wide billing mutex, pg_try_advisory_lock() for Application.launch coroutine scheduler serialization — and per-billing-period vault keys via a spend-cap proxy as a hard financial backstop. For reactive retry failure modes in coroutine-adjacent frameworks, see the Spring WebFlux and Stripe Integration post. For AOP-level retry that re-invokes entire method bodies, see the Micronaut and Stripe Integration post.

Failure mode 1: modifyRequest in HttpRequestRetry is the sole source of the idempotency key — the callback runs only on retry attempts — the initial request carries no key — each retry carries a different UUID.randomUUID()

Ktor’s HttpRequestRetry plugin is configured at the HttpClient level and intercepts the HttpSend pipeline phase. Its modifyRequest callback is documented as “modif[ying] a request before retrying” — that is, before each retry attempt, not before the initial attempt. The initial request is sent exactly as the call-site post { } builder assembled it; modifyRequest is never called for the first transmission. This distinction is the root of the first failure mode.

A common pattern for centralising Stripe idempotency key generation is to configure it once on the shared HttpClient rather than threading it through every call site. The modifyRequest callback looks like the right place for this — it runs on every request, right? It does not:

// Ktor 2.3.x — BillingClient.kt
// UNSAFE: modifyRequest is the only place Idempotency-Key is set
// It runs only on RETRY attempts — the initial request has no Idempotency-Key header

import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.plugins.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import io.ktor.http.*
import java.util.UUID

val stripeClient = HttpClient(CIO) {
    install(HttpRequestRetry) {
        retryOnServerErrors(maxRetries = 3)
        exponentialDelay()
        modifyRequest {
            // WRONG: developer intends this to run on every request including initial.
            // ACTUAL behaviour: runs only before each RETRY attempt.
            //
            // Initial request: modifyRequest never called → no Idempotency-Key header
            // Retry attempt 1: modifyRequest called → UUID_1 = "b8d2e4f6..."
            // Retry attempt 2: modifyRequest called → UUID_2 = "1c9d3e5a..."
            headers {
                set("Idempotency-Key", UUID.randomUUID().toString())
            }
        }
    }
}

suspend fun chargeBilling(
    customerId: String,
    billingPeriod: String,
    amountCents: Long
): ChargeResponse {
    return stripeClient.post("https://api.stripe.com/v1/charges") {
        header("Authorization", "Bearer $stripeApiKey")
        // No Idempotency-Key set here — developer relies on modifyRequest above
        setBody(FormDataContent(Parameters.build {
            append("amount", amountCents.toString())
            append("currency", "usd")
            append("customer", customerId)
            append("description", "Billing period $billingPeriod")
        }))
    }.body()
}

The failure scenario: the agent calls chargeBilling("cust_123", "2026-08", 9900L). Ktor sends the initial POST /v1/charges request. The modifyRequest block does not run for the initial attempt, so the request has no Idempotency-Key header. Stripe’s API receives a charge request without a client-supplied idempotency key. Stripe assigns its own internal deduplication identifier for the request’s lifetime, but this identifier is opaque to the client and not reusable across connections. Stripe processes the charge: the card is authorized, the charge object ch_A is created and committed to Stripe’s ledger. Before the HTTP response is flushed to the client, Stripe’s API gateway encounters a brief internal overload and returns a transient 503 Service Unavailable.

Ktor’s HttpRequestRetry plugin catches the 503. retryOnServerErrors matches this response code. The plugin is about to send retry attempt 1. Before it does, it calls modifyRequest. The block executes: UUID.randomUUID() returns "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d". The retry request carries Idempotency-Key: b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d. Stripe has never seen this key. ch_A was already created with no client idempotency key and cannot be found by a lookup on this new key. Stripe processes the retry as a fresh charge request. ch_B is created. Customer 123 is charged $99 twice for August 2026.

If retry attempt 1 also encounters a transient error before Stripe responds, modifyRequest runs again before retry attempt 2: UUID.randomUUID() returns "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f". ch_C is created. Three charges for one customer and one billing period, from a three-attempt retry configuration that was intended to be safe.

The subtler variant: modifyRequest overwrites the stable key set on the initial request

A more subtle form of this failure occurs when the developer correctly sets a deterministic idempotency key in the initial post { } builder block, but the modifyRequest callback also manipulates the Idempotency-Key header — perhaps to add a per-attempt suffix for tracing, or because a copy-paste error put the UUID.randomUUID() call in the wrong block:

// The subtle variant — stable key set on initial request, but modifyRequest overwrites it

val stripeClient = HttpClient(CIO) {
    install(HttpRequestRetry) {
        retryOnServerErrors(maxRetries = 3)
        exponentialDelay()
        modifyRequest {
            // Developer intends to add a retry counter for tracing.
            // But set() REPLACES the existing Idempotency-Key header value
            // rather than adding a separate header.
            // The stable key set in post{} is overwritten with a new UUID on every retry.
            headers {
                set("Idempotency-Key", UUID.randomUUID().toString())  // ← overwrites stable key
                set("X-Retry-Count", retryCount.toString())
            }
        }
    }
}

suspend fun chargeBilling(customerId: String, billingPeriod: String, amountCents: Long): ChargeResponse {
    // Stable key correctly computed from business fields — safe for the initial request
    val idempotencyKey = stableKey(customerId, billingPeriod)

    return stripeClient.post("https://api.stripe.com/v1/charges") {
        header("Authorization", "Bearer $stripeApiKey")
        header("Idempotency-Key", idempotencyKey)  // ← correct on initial request
        // On retry 1: modifyRequest replaces this with UUID.randomUUID() → different key → ch_B
        setBody(FormDataContent(Parameters.build {
            append("amount", amountCents.toString())
            append("currency", "usd")
            append("customer", customerId)
            append("description", "Billing period $billingPeriod")
        }))
    }.body()
}

// stableKey: sha256(customerId + ":" + billingPeriod + ":ktor-billing") truncated to 32 hex chars
// Identical on every call with the same inputs — does NOT re-evaluate to a new value
fun stableKey(customerId: String, billingPeriod: String): String {
    val digest = java.security.MessageDigest.getInstance("SHA-256")
    val hash = digest.digest(
        "$customerId:$billingPeriod:ktor-billing".toByteArray(Charsets.UTF_8)
    )
    return hash.take(16).joinToString("") { "%02x".format(it) }
}

The scenario: the initial request sends Idempotency-Key: <stable-hash>. Stripe creates ch_A with that key before returning a 503. For retry attempt 1, modifyRequest runs and calls headers { set("Idempotency-Key", UUID.randomUUID().toString()) }. In Ktor, headers { set(name, value) } replaces any existing value for the named header. The retry’s Idempotency-Key is now the new UUID. Stripe has cached ch_A against the stable hash key, not against this UUID. It processes the retry as a new request. ch_B created. The X-Retry-Count: 1 header the developer wanted for tracing is correctly added, but the side-effect of calling set on the idempotency key header made the retry non-idempotent.

The fix requires separating tracing metadata from idempotency key management. The idempotency key should never be set or modified inside modifyRequest; it should be computed once from stable business fields before the request builder and captured in the closure:

The fix for failure mode 1

// Safe: stable key computed before the request builder, never touched by modifyRequest
// modifyRequest adds only the retry counter header, not the idempotency key

val stripeClient = HttpClient(CIO) {
    install(HttpRequestRetry) {
        retryOnServerErrors(maxRetries = 3)
        exponentialDelay()
        modifyRequest {
            headers {
                // Safe: only adds the retry counter — does not touch Idempotency-Key
                set("X-Retry-Count", retryCount.toString())
            }
        }
    }
    // Alternative: set the idempotency key via a custom HttpRequestInterceptor
    // that reads from a per-request attribute set by the call site.
    // This separates the concerns cleanly without touching modifyRequest.
}

suspend fun chargeBilling(
    customerId: String,
    billingPeriod: String,
    amountCents: Long
): ChargeResponse {
    // Safe: stableKey computed once, before the post{} builder.
    // modifyRequest never touches Idempotency-Key, so this value survives to the retry.
    // Every retry carries the same key — Stripe returns the cached ch_A response.
    val idempotencyKey = stableKey(customerId, billingPeriod)

    return stripeClient.post("https://api.stripe.com/v1/charges") {
        header("Authorization", "Bearer $stripeApiKey")
        header("Idempotency-Key", idempotencyKey)
        setBody(FormDataContent(Parameters.build {
            append("amount", amountCents.toString())
            append("currency", "usd")
            append("customer", customerId)
            append("description", "Billing period $billingPeriod")
        }))
    }.body()
}

// What to EXCLUDE from stableKey — any value that re-evaluates to a different result
// between the initial request and each retry attempt:
//   UUID.randomUUID()              ← random, differs per invocation
//   System.currentTimeMillis()     ← different if retry fires 1s later
//   System.nanoTime()              ← different per JVM, per invocation
//   retryCount                     ← the retry plugin's own counter (0, 1, 2, ...)
//   System.identityHashCode(obj)   ← differs per object instance, per JVM
//   Thread.currentThread().name    ← coroutine dispatcher thread name changes per retry

// What IS safe in stableKey:
//   customerId                     ← stable business identifier
//   billingPeriod                  ← stable business identifier (e.g., "2026-08")
//   a vendor namespace string      ← literal constant ("ktor-billing")
// → sha256("cust_123:2026-08:ktor-billing")[:32] = same value on every call

With the stable key computed before the post { } builder and never touched by modifyRequest, every retry attempt carries the same Idempotency-Key header value. Stripe’s idempotency cache matches the retry against the original request by key and returns the cached ch_A response without creating a new charge. The X-Retry-Count header from modifyRequest gives the desired tracing signal without interfering with idempotency. The pre-flight database guard below (from the fix for failure mode 2) provides additional protection for cases where Stripe’s 24-hour idempotency window has expired.

Failure mode 2: manual retry loop re-invokes the billing suspend fun — UUID.randomUUID() at function entry produces a new idempotency key on every call — the first call created ch_A before the timeout — the second call creates ch_B

The HttpRequestRetry plugin is useful for engine-level retries of a single HttpClient call, but many Ktor applications handle retry at a higher level: a retry helper that catches exceptions and calls the billing function again. This is a natural pattern in coroutine code, where a suspend function can be called from within a repeat loop or a while loop exactly like any other function. The failure is that each call to the billing suspend fun executes its body from the top, including any UUID.randomUUID() call at function entry:

// BillingService.kt
// UNSAFE: retry loop calls chargeBilling() multiple times
// UUID.randomUUID() at function entry fires on every invocation

import io.ktor.client.*
import io.ktor.client.call.*
import io.ktor.client.plugins.*
import io.ktor.client.request.*
import io.ktor.http.*
import kotlinx.coroutines.delay
import java.util.UUID

class BillingService(
    private val client: HttpClient,
    private val stripeApiKey: String
) {

    suspend fun chargeBillingWithRetry(
        customerId: String,
        billingPeriod: String,
        amountCents: Long,
        maxAttempts: Int = 3
    ): ChargeResponse {
        var lastError: Throwable? = null
        repeat(maxAttempts) { attempt ->
            try {
                // Each call to chargeBilling() is a fresh function invocation.
                // The function body executes from the top.
                // UUID.randomUUID() inside chargeBilling() produces a new value each time.
                return chargeBilling(customerId, billingPeriod, amountCents)
            } catch (e: ServerResponseException) {
                if (e.response.status.value in 500..599) {
                    lastError = e
                    delay(1_000L * (attempt + 1))
                } else throw e
            } catch (e: HttpRequestTimeoutException) {
                // Timeout: Stripe may have already processed the charge on its side
                // and ch_A already exists in Stripe's ledger.
                // The next call to chargeBilling() produces UUID_next → ch_B.
                lastError = e
                delay(1_000L * (attempt + 1))
            }
        }
        throw lastError ?: IllegalStateException("No billing attempts were made")
    }

    private suspend fun chargeBilling(
        customerId: String,
        billingPeriod: String,
        amountCents: Long
    ): ChargeResponse {
        // UNSAFE: UUID computed at function entry on every call to chargeBilling()
        // Attempt 0 (from repeat() index 0): UUID = "3f7a9b2c-1d4e-4a5f-8b6c-0e2d1a3c7f9e"
        // Attempt 1 (from repeat() index 1): UUID = "b8d2e4f6-3a7c-4e9f-1b5d-0c8a7e2f4b6d"
        // Attempt 2 (from repeat() index 2): UUID = "1c9d3e5a-7f2b-4d8c-6e0f-4a3b9c1d5e7f"
        val idempotencyKey = UUID.randomUUID().toString()

        return client.post("https://api.stripe.com/v1/charges") {
            header("Authorization", "Bearer $stripeApiKey")
            header("Idempotency-Key", idempotencyKey)
            setBody(FormDataContent(Parameters.build {
                append("amount", amountCents.toString())
                append("currency", "usd")
                append("customer", customerId)
                append("description", "Billing period $billingPeriod")
            }))
        }.body()
    }
}

The failure scenario in detail: the agent calls billingService.chargeBillingWithRetry("cust_123", "2026-08", 9900L). The repeat(3) loop begins. Attempt 0 calls chargeBilling("cust_123", "2026-08", 9900L). The function executes from the top: UUID.randomUUID() returns "3f7a9b2c...". The HttpClient sends POST /v1/charges with Idempotency-Key: 3f7a9b2c.... Stripe’s servers receive the request. The charge object ch_A is created and the card is authorized. Stripe is preparing to flush the HTTP response. At this moment, the Ktor client’s request timeout fires (the timeout was configured at 15 seconds, and Stripe’s processing plus response flush crossed that threshold). Ktor throws an HttpRequestTimeoutException.

The catch (e: HttpRequestTimeoutException) block handles this. lastError = e. After a 1-second delay, the repeat loop proceeds to index 1. Attempt 1 calls chargeBilling("cust_123", "2026-08", 9900L) again. This is a fresh call to the function. Kotlin’s coroutine machinery suspends and resumes, but the function body executes from the top of the function regardless — it is not a coroutine continuation resuming mid-function from where it previously suspended. UUID.randomUUID() evaluates again: "b8d2e4f6...", a completely different UUID. The second POST /v1/charges carries Idempotency-Key: b8d2e4f6.... Stripe has never seen this key. ch_A was already committed in Stripe’s ledger with key "3f7a9b2c...". Stripe processes attempt 1 as an entirely new charge request. ch_B is created. Customer 123 now has two $99 charges for August 2026.

This failure is mechanically identical to the Micronaut @Retryable failure mode where the AOP interceptor calls context.proceed() to re-execute the method body on each retry. The mechanism differs — Micronaut uses reflection-based AOP interception; Ktor uses an explicit repeat loop — but the result is the same: a function that computes UUID.randomUUID() at its entry is called multiple times, producing a different UUID on each call, and creating a new Stripe charge on each call that follows a timeout or error after ch_A was committed.

The fix for failure mode 2

The idempotency key must be derived from stable business fields and computed once, in the retry wrapper, before the retry loop begins. The key is then passed as a parameter into the billing function so that every invocation of the function within the same retry scope uses the same key:

// Safe: stable key computed OUTSIDE the repeat() loop, passed into chargeBilling()
// Every call to chargeBilling() within the retry scope uses the same idempotencyKey
class SafeBillingService(
    private val client: HttpClient,
    private val stripeApiKey: String,
    private val billingRepository: BillingRepository
) {

    suspend fun chargeBillingWithRetry(
        customerId: String,
        billingPeriod: String,
        amountCents: Long,
        maxAttempts: Int = 3
    ): ChargeResponse {
        // Safe: computed once from stable inputs.
        // The repeat() loop calls chargeBilling() multiple times, but idempotencyKey
        // is a val captured in the outer scope — it does not re-evaluate on each call.
        val idempotencyKey = stableKey(customerId, billingPeriod)

        // Pre-flight: claim the billing slot before any Stripe call.
        // ON CONFLICT DO NOTHING means only the first claimer reaches Stripe.
        // A second call (retry, race with another coroutine) finds the row exists
        // and short-circuits — returns the existing charge_id if already completed.
        val existingCharge = billingRepository.findByCustomerAndPeriod(customerId, billingPeriod)
        if (existingCharge?.chargeId != null) {
            return ChargeResponse(existingCharge.chargeId, "succeeded")
        }

        billingRepository.claimSlot(customerId, billingPeriod, idempotencyKey)
        // If claimSlot returns false (conflict), another coroutine already claimed it —
        // wait for it to complete and return the result

        var lastError: Throwable? = null
        repeat(maxAttempts) { attempt ->
            try {
                return chargeBilling(customerId, billingPeriod, amountCents, idempotencyKey)
            } catch (e: ServerResponseException) {
                if (e.response.status.value in 500..599) {
                    lastError = e
                    delay(1_000L * (attempt + 1))
                } else throw e
            } catch (e: HttpRequestTimeoutException) {
                lastError = e
                delay(1_000L * (attempt + 1))
            }
        }
        throw lastError ?: IllegalStateException("All retry attempts exhausted")
    }

    private suspend fun chargeBilling(
        customerId: String,
        billingPeriod: String,
        amountCents: Long,
        idempotencyKey: String   // ← key passed in, not re-computed inside the function
    ): ChargeResponse {
        // Safe: idempotencyKey is a parameter, not computed here.
        // Every retry attempt from chargeBillingWithRetry() passes the same value.
        // Stripe's idempotency cache matches every retry to the same original request
        // and returns the cached ch_A response without creating a new charge.
        val response = client.post("https://api.stripe.com/v1/charges") {
            header("Authorization", "Bearer $stripeApiKey")
            header("Idempotency-Key", idempotencyKey)
            setBody(FormDataContent(Parameters.build {
                append("amount", amountCents.toString())
                append("currency", "usd")
                append("customer", customerId)
                append("description", "Billing period $billingPeriod")
            }))
        }.body<ChargeResponse>()

        billingRepository.markComplete(customerId, billingPeriod, response.id)
        return response
    }
}

-- Pre-flight table: billing_records as the cluster-wide billing mutex
CREATE TABLE billing_records (
    customer_id      TEXT        NOT NULL,
    billing_period   TEXT        NOT NULL,
    idempotency_key  TEXT        NOT NULL,
    status           TEXT        NOT NULL DEFAULT 'pending',
    charge_id        TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT billing_records_pk  PRIMARY KEY (idempotency_key),
    CONSTRAINT billing_records_uq  UNIQUE (customer_id, billing_period)
);

-- claimSlot(): atomic INSERT — returns 1 row if claimed, 0 rows if conflict
INSERT INTO billing_records (customer_id, billing_period, idempotency_key)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id, billing_period) DO NOTHING;

-- markComplete(): record the Stripe charge_id after success
UPDATE billing_records
   SET status = 'completed', charge_id = $3
 WHERE customer_id = $1 AND billing_period = $2;

With the stable key computed before the retry loop and passed as a parameter, every invocation of chargeBilling() within the retry scope sends the same Idempotency-Key to Stripe. After the first attempt creates ch_A and the timeout fires, the retry sends the identical key. Stripe’s idempotency cache looks up the key, finds the completed charge, and returns the cached response without creating a new charge. The pre-flight billing_records table provides a database-level guard that survives Stripe’s 24-hour idempotency window: even if the retry happens 25 hours after the original request, the ON CONFLICT DO NOTHING on (customer_id, billing_period) prevents the billing function from reaching Stripe at all for a period that already has a committed row.

Failure mode 3: Application.launch {} billing scheduler per Ktor replica — no cross-pod coordination in Kubernetes — three pods generate distinct UUID.randomUUID() values per customer — ch_A, ch_B, and ch_C per customer per billing period

Ktor applications often run periodic billing jobs as coroutines started within the application’s coroutine scope. Ktor’s Application object implements CoroutineScope, which means launch { } called inside a module function starts a coroutine that lives for the lifetime of the application and is cancelled on shutdown. This is idiomatic Ktor — and it is per-JVM. In a Kubernetes Deployment with replicas: 3, three Ktor pods start three independent JVMs, and launch { } is called in each one:

// Application.kt
// UNSAFE: Application.launch {} runs on every pod — no cross-pod coordination
import io.ktor.server.application.*
import kotlinx.coroutines.delay
import kotlinx.coroutines.isActive
import java.time.YearMonth
import java.util.UUID
import kotlin.time.Duration.Companion.hours
import kotlin.time.Duration.Companion.seconds

fun Application.configureBillingScheduler() {
    val billingService = BillingService(
        client = httpClient,
        stripeApiKey = environment.config.property("stripe.apiKey").getString()
    )

    // Application.launch {} starts one coroutine per JVM.
    // With replicas: 3 in Kubernetes, this coroutine runs on all three pods
    // simultaneously — there is no built-in cross-pod coordination.
    launch {
        delay(30.seconds)  // Wait for application startup before first billing check

        while (isActive) {
            val currentPeriod = YearMonth.now().toString()  // "2026-08"

            // DATABASE CHECK: all three pods query this simultaneously.
            // Before any pod has committed a 'billing_started' record,
            // all three see alreadyRan = false and proceed to bill.
            val alreadyRan = billingRepository.hasCompletedForPeriod(currentPeriod)
            if (!alreadyRan) {
                runMonthlyBilling(currentPeriod, billingService)
            }

            delay(1.hours)
        }
    }
}

private suspend fun Application.runMonthlyBilling(
    billingPeriod: String,
    billingService: BillingService
) {
    val customers = customerRepository.findAllActive()  // 500 customers, read on all three pods
    customers.forEach { customer ->
        // UNSAFE: UUID.randomUUID() called per customer per pod
        // Pod 1: "3f7a9b2c..." → ch_1A, ch_2A, ch_3A, ..., ch_500A
        // Pod 2: "b8d2e4f6..." → ch_1B, ch_2B, ch_3B, ..., ch_500B
        // Pod 3: "1c9d3e5a..." → ch_1C, ch_2C, ch_3C, ..., ch_500C
        val idempotencyKey = UUID.randomUUID().toString()
        billingService.chargeCustomer(customer, billingPeriod, idempotencyKey)
    }
}

The failure scenario: a Kubernetes rolling deployment starts three Ktor pods. All three execute configureBillingScheduler() in their application module. Each pod’s launch { } coroutine starts. After the 30-second startup delay, all three pods enter their while (isActive) loop at approximately the same wall-clock time. All three call billingRepository.hasCompletedForPeriod("2026-08"). The monthly billing has not run yet: all three database reads return false. All three pods pass the check and call runMonthlyBilling("2026-08", billingService).

All three pods query customerRepository.findAllActive() and receive all 500 customers. All three pods begin billing every customer. For customer cust_1: Pod 1 generates UUID.randomUUID() = "3f7a9b2c..." and sends POST /v1/charges with Idempotency-Key: 3f7a9b2c.... Pod 2 generates "b8d2e4f6..." and sends a concurrent POST /v1/charges with that key. Pod 3 generates "1c9d3e5a..." and sends a third concurrent request. Stripe sees three distinct idempotency keys for cust_1 in the same billing period — it has no way to know these are semantically the same billing intent, because the keys are different. Stripe creates ch_1A, ch_1B, and ch_1C. Across 500 customers, 1,500 Stripe charges are created where 500 were intended.

Even if the hasCompletedForPeriod check is accurate — a database row per period, written atomically — the check-then-act pattern is a time-of-check time-of-use (TOCTOU) race condition at distributed-system scale. Three pods executing the check concurrently can all read false from the database before any of them has written the “billing started” record. The gap between the SELECT and the subsequent INSERT (or the first Stripe call) is long enough for all three pods to pass the check simultaneously and proceed to billing. The check provides no mutual exclusion guarantee across pods without an atomic lock.

The fix for failure mode 3

The fix requires two layers. The first layer is a distributed advisory lock that ensures only one pod actually runs the billing job at all, eliminating the triple database and Stripe workload. The second layer is the content-hash idempotency key and pre-flight ON CONFLICT DO NOTHING from failure mode 2 as an authoritative backstop that handles edge cases the lock cannot prevent (operator-triggered billing runs, rolling deploy overlaps, or future code paths that bypass the lock).

PostgreSQL advisory locks provide an efficient distributed mutex without requiring a separate coordination service like Redis or ZooKeeper. A session-level advisory lock (pg_try_advisory_lock()) is acquired by one connection and held until the connection is closed or the lock is explicitly released. Only one PostgreSQL session can hold a given advisory lock key at a time — the second pod’s pg_try_advisory_lock() call returns false immediately if another session holds the lock:

// Safe: pg_try_advisory_lock() as a distributed mutex for the billing coroutine
// Only the pod that acquires the lock runs the billing job — others skip silently

import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.time.YearMonth
import kotlin.math.abs

fun Application.configureBillingScheduler() {
    val billingService = SafeBillingService(httpClient, stripeApiKey, billingRepository)

    launch {
        delay(30.seconds)

        while (isActive) {
            val currentPeriod = YearMonth.now().toString()
            tryRunMonthlyBillingWithLock(currentPeriod, billingService)
            delay(1.hours)
        }
    }
}

private suspend fun Application.tryRunMonthlyBillingWithLock(
    billingPeriod: String,
    billingService: SafeBillingService
) {
    // pg_try_advisory_lock is a session-level lock — must hold the connection open.
    // withContext(Dispatchers.IO) borrows a thread from the IO pool for the blocking JDBC call.
    // The connection is held for the duration of the billing run, then returned to the pool.
    val lockKey = abs("monthly-billing:$billingPeriod".hashCode().toLong())

    withContext(Dispatchers.IO) {
        val conn = dataSource.connection  // borrow a dedicated connection for the lock
        try {
            val acquired = conn.prepareStatement("SELECT pg_try_advisory_lock(?)").use { stmt ->
                stmt.setLong(1, lockKey)
                stmt.executeQuery().use { rs -> rs.next() && rs.getBoolean(1) }
            }

            if (!acquired) {
                // Another pod holds the lock — this pod skips this billing cycle
                log.info("Monthly billing lock for $billingPeriod not acquired — skipping (another pod running)")
                return@withContext
            }

            // Only this pod reaches here — run billing for this period
            log.info("Monthly billing lock for $billingPeriod acquired — running billing job")
            runMonthlyBillingSafe(billingPeriod, billingService, conn)

        } finally {
            // Release the session-level advisory lock before returning the connection
            try {
                conn.prepareStatement("SELECT pg_advisory_unlock(?)").use { stmt ->
                    stmt.setLong(1, lockKey)
                    stmt.executeQuery()
                }
            } finally {
                conn.close()
            }
        }
    }
}

private fun runMonthlyBillingSafe(
    billingPeriod: String,
    billingService: SafeBillingService,
    conn: java.sql.Connection
) {
    val customers = customerRepository.findAllActiveBlocking(conn)
    customers.forEach { customer ->
        // Safe: stableKey derived from business fields — identical on all pods
        // Pre-flight ON CONFLICT DO NOTHING prevents double-billing even if lock fails
        val idempotencyKey = stableKey(customer.id, billingPeriod)
        val claimed = billingRepository.claimSlotBlocking(conn, customer.id, billingPeriod, idempotencyKey)
        if (claimed) {
            billingService.chargeCustomerBlocking(conn, customer, billingPeriod, idempotencyKey)
        }
    }
}

The pg_try_advisory_lock(lockKey) call returns true for the first pod that executes it and false for the other two. The two pods that do not acquire the lock log a skip message and return. The one pod that acquires the lock runs the billing job end-to-end. When the billing job completes (or throws), the finally block releases the lock with pg_advisory_unlock() and closes the connection, returning it to the HikariCP pool.

The lock key is derived from a string hash of the billing period to make locks for different billing months independent: the lock for "monthly-billing:2026-08" does not block concurrent execution of "monthly-billing:2026-09". The abs() call converts the hash to a positive value because PostgreSQL’s advisory lock key space is 64-bit signed integers and a negative hash would cause a type mismatch error.

A note on connection pool impact: holding a connection open for the duration of the billing job (potentially minutes for 500 customers with Stripe round-trips) removes that connection from HikariCP’s pool for the duration. Configure the pool with at least one extra connection for the billing job (maximumPoolSize = N + 1 where N is your normal query concurrency), or use the transaction-level advisory lock variant (pg_try_advisory_xact_lock() inside an explicit transaction) which releases the lock on transaction commit and allows the connection to be returned sooner — at the cost of the billing job needing to handle the case where the lock is released mid-run if the transaction commits partway through. For most billing jobs where the entire batch runs atomically, the session-level lock with a held connection is simpler and more predictable.

-- Alternative: transaction-level advisory lock (auto-released on COMMIT/ROLLBACK)
-- Use when you want the lock to be released at a transactional boundary
-- rather than holding the connection for the full billing run duration

BEGIN;
SELECT pg_try_advisory_xact_lock($lockKey);
-- If returns false: ROLLBACK, skip
-- If returns true: run billing inside the transaction
-- Lock auto-released on COMMIT or ROLLBACK — connection returned to pool immediately

Both the session-level and transaction-level advisory lock approaches require the content-hash key and ON CONFLICT DO NOTHING pre-flight as a second layer. The advisory lock serializes the billing job under normal operation and eliminates the triple database load. The pre-flight guard handles the residual cases: a deployment with two overlapping pod versions where both run the billing coroutine within milliseconds of each other before the rolling update removes the old pod, a manual kubectl exec that triggers billing outside the scheduler, or a future code change that adds a second billing path without updating the distributed lock acquisition. The combination of both layers means no single point of failure can produce a duplicate charge.

Gap analysis: other Ktor billing patterns not covered above

The three failure modes above cover the most common Ktor retry surfaces. Several adjacent patterns introduce the same class of failure through different mechanisms:

Ktor’s retry { } DSL from kotlinx-coroutines or custom extension functions that call the billing function inside the retry block — some teams build a retry(times, delay) { block } higher-order function that calls block() on each attempt. If block is a lambda that includes UUID.randomUUID() at its top (or closes over a billing function that includes it), each retry invocation of block() evaluates the UUID fresh. The fix is identical to failure mode 2: compute stableKey() outside the retry lambda and capture it in the lambda’s closure, or pass it as an explicit parameter to the billing function.

Ktor HttpClient with a custom HttpSendPlugin interceptor that generates the idempotency key per-send — a plugin(HttpSend) { intercept { request, next -> ... } } interceptor runs on every HTTP send event, including retry sends triggered by HttpRequestRetry. If the interceptor computes UUID.randomUUID() inside the intercept lambda and sets it as the Idempotency-Key header, every retry attempt generates a new UUID at the intercept level. The fix: the interceptor should read the idempotency key from a request attribute set by the call site (request.attributes.getOrNull(IdempotencyKeyKey) ?: stableKey(request)) rather than generating one internally.

Ktor WebSocket billing handler with reconnect — a billing handler implemented as a WebSocket session that reconnects on disconnect and replays unacknowledged billing commands: if the per-command idempotency key is derived from the WebSocket session identifier (call.request.headers["Sec-WebSocket-Key"], the HTTP/1.1 upgrade request timestamp, or System.identityHashCode() of the DefaultWebSocketSession object), each reconnect produces a new session identifier and a different key for the same billing command. The fix: the key must be derived from the billing command’s business payload (customer ID, billing period) rather than from any session-level attribute.

Ktor scheduleAtFixedRate-style coroutine via delay in a loop with a YearMonth.now() key component but without a distributed lock — replacing the advisory lock with a database-row check-then-act pattern (SELECT EXISTS(... WHERE billing_period = ?) followed by INSERT) is a TOCTOU race unless the check and the insert are atomic. The correct pattern is either the advisory lock from failure mode 3, or an INSERT ... ON CONFLICT DO NOTHING RETURNING * that atomically claims the billing slot and returns whether the claim succeeded in a single statement. If the claim returns zero rows (conflict), the billing is already running and this pod should skip. If it returns one row, this pod won the claim and should proceed to Stripe.

Ktor application with a by lazy { } idempotency key initializer — a pattern that computes val idempotencyKey by lazy { UUID.randomUUID().toString() } on a class property appears safe at first glance because by lazy initialises only once per instance. The failure occurs when the class instance is recreated per billing run — for example, if BillingJob is instantiated inside the retry loop or inside a coroutine that is cancelled and restarted. Each new instance computes a new lazy UUID on first access. The fix is the same: stableKey(customerId, billingPeriod) derived from business fields, not from a lazy initializer whose uniqueness depends on instance lifetime.

Summary

The thread connecting all three: the idempotency key must be derived from the business intent of the billing operation — customer ID, billing period, vendor namespace — not from any ephemeral runtime value that changes between attempts, requests, or JVMs. UUID.randomUUID(), System.currentTimeMillis() at request-build time, retryCount, and System.identityHashCode() of any object all produce different values on each retry attempt or each pod. A key derived from sha256(customerId + ":" + billingPeriod + ":ktor-billing")[:32] produces the same 32-character hex string on every retry, on every pod, and on every function invocation within the same billing period — stable across all three failure modes above. Backing it with a database-level UNIQUE (customer_id, billing_period) constraint and a pg_try_advisory_lock() distributed mutex moves the deduplication guarantee out of ephemeral in-memory state and into durable storage that survives timeouts, reconnects, and rolling deploys.

The vault key spend cap adds a hard financial boundary as a last line of defence: a per-billing-period vault key issued with a max_amount set to expected_total × 1.10 caps the maximum spend that any combination of bugs, retries, and multi-pod races can produce, regardless of how many idempotency key failures occur. Once the cap is reached, Stripe rejects further charges with a 402 Payment Required on the vault key, containing the blast radius to a known maximum regardless of how many retries or pods are in flight.