Akka HTTP and Stripe Integration: Restricted API Keys, Spend Caps, and Agent Governance

Akka HTTP’s Future-based request pipeline and reactive streaming model create three Stripe billing failure modes that each produce a duplicate charge from a different idempotency key: a recursive Future retry that calls UUID.randomUUID() on each invocation; a RetryFlow wrapping a cachedHostConnectionPool that re-builds the request per retry with a fresh idempotency key; and a withRequestTimeout directive that fires a 503 before the Stripe call completes, prompting the upstream load balancer to retry the request on a second pod that generates its own UUID in the billing handler.

This post covers all three failure modes with Scala Akka HTTP 10.x code, content-hash idempotency keys stable across recursive retries, RetryFlow re-materializations, and cross-pod upstream retries, pre-flight PostgreSQL ON CONFLICT DO NOTHING checks — and per-billing-period vault keys via a spend-cap proxy as a hard backstop. For related failure modes in the Akka actor model and persistent actor recovery, see the Akka Typed EventSourcedBehavior and Stripe Integration post. For stream-level materialization retries and Akka Streams restart supervision, see the Akka Streams and Stripe Integration post.

Failure mode 1: recursive Future retry calls UUID.randomUUID() on each invocation — the original Stripe call completes (ch_A created before network timeout) — the retry generates a new UUID — Stripe creates ch_B for a customer already charged

The idiomatic Akka HTTP retry pattern for a Http().singleRequest() call that may fail due to network errors or transient 5xx responses is a recursive Future function with an attempt counter. The billing function constructs the Stripe HTTP request, issues it via Http().singleRequest(), and recurses on failure up to a configured retry limit:

// Akka HTTP 10.x — billing with recursive Future retry
// UNSAFE: UUID generated per invocation of chargeCustomer()
import java.util.UUID
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.http.scaladsl.model.headers.RawHeader

def chargeCustomer(
    customerId: String,
    amount:     Long,
    currency:   String,
    billingPeriod: String,
    attempt:    Int = 0
)(implicit system: ActorSystem, ec: ExecutionContext): Future[String] = {

  // UNSAFE: UUID.randomUUID() is evaluated fresh on every call to chargeCustomer().
  // Original call (attempt = 0): key = "7e3f9a1b-2c4d-4e5f-8a0b-1c2d3e4f5a6b"
  // Retry    (attempt = 1): key = "1d2e3f4a-5b6c-4d7e-9f0a-2b3c4d5e6f7a" (different)
  val idempotencyKey = UUID.randomUUID().toString

  val formData = Map(
    "amount"      -> amount.toString,
    "currency"    -> currency,
    "customer"    -> customerId,
    "description" -> s"Billing period $billingPeriod"
  )

  val request = HttpRequest(
    method = HttpMethods.POST,
    uri    = "https://api.stripe.com/v1/charges",
    headers = List(
      RawHeader("Authorization",   s"Bearer $stripeSecretKey"),
      RawHeader("Idempotency-Key", idempotencyKey)
    ),
    entity = FormData(formData).toEntity
  )

  Http().singleRequest(request).flatMap { response =>
    response.status match {
      case StatusCodes.OK | StatusCodes.Created =>
        Unmarshal(response.entity).to[StripeChargeResponse].map(_.id)

      case StatusCodes.TooManyRequests | StatusCodes.InternalServerError
           | StatusCodes.BadGateway    | StatusCodes.ServiceUnavailable if attempt < 3 =>
        // Retry with the same function — generates a new UUID on next invocation.
        // If the Stripe call already reached the network and ch_A was created before
        // the 5xx response was returned (e.g. Stripe processed the charge but had an
        // internal error writing the response), the retry creates ch_B.
        response.entity.discardBytes()
        after(Duration(500L * math.pow(2, attempt).toLong, "ms"))(
          chargeCustomer(customerId, amount, currency, billingPeriod, attempt + 1)
        )

      case _ =>
        response.entity.discardBytes()
        Future.failed(new RuntimeException(s"Stripe error: ${response.status}"))
    }
  }.recoverWith {
    // Network-level failure (TCP reset, connection timeout) — retry if attempts remain.
    // Same problem: chargeCustomer() generates a new UUID on re-entry.
    case _: akka.stream.StreamTcpException | _: java.net.ConnectException if attempt < 3 =>
      after(Duration(500L * math.pow(2, attempt).toLong, "ms"))(
        chargeCustomer(customerId, amount, currency, billingPeriod, attempt + 1)
      )
  }
}

The failure scenario: chargeCustomer("cust_123", 9900, "usd", "2026-08") is called at attempt 0. UUID.randomUUID() returns "7e3f9a1b...". The request reaches Stripe’s API server. Stripe processes the charge and creates ch_A (ch_Af21c). Before Stripe sends the 201 response, Stripe’s API tier experiences an internal write timeout and returns a 500 instead. Akka HTTP receives the 500, discards the entity, and — because attempt < 3 — recurses into chargeCustomer(..., attempt = 1). A new UUID.randomUUID() is evaluated: "1d2e3f4a...". Stripe has never seen this key. The retry request reaches Stripe. ch_B (ch_B88a1) is created. Customer 123 now has two charges for August 2026.

The failure is especially hard to reproduce in integration tests because it requires a Stripe API call that both completes successfully on Stripe’s side and returns a 5xx to the caller. This combination — a Stripe internal error on the response path after processing — is rare but documented in Stripe’s own guidance on idempotency: “If you get a network error, you don’t know whether the request made it to Stripe. Retry with the same idempotency key.” The recursive pattern does not implement this guidance correctly because it generates a new key per call rather than per billing intent.

The same failure occurs when the network connection drops after Stripe begins processing but before the response arrives — Http().singleRequest() throws a StreamTcpException, the recoverWith block catches it, and the recursive call generates a new UUID. In high-load environments with connection pool exhaustion or TLS renegotiation latency, TCP resets during Stripe’s processing window are common enough to trigger this path multiple times per day.

The fix for failure mode 1

The idempotency key must be computed once before the retry loop begins and passed into each retry attempt unchanged. A content-hash key derived from stable business fields — customerId, billingPeriod, and a static salt — is identical on every invocation regardless of attempt number. The pre-flight database check separates the billing intent from the Stripe call: a database row is claimed before calling Stripe, so a retry that finds an existing row skips Stripe entirely:

// Safe Akka HTTP billing — content-hash key, pre-flight DB check
import java.security.MessageDigest

def stableKey(customerId: String, billingPeriod: String): String = {
  val digest = MessageDigest.getInstance("SHA-256")
  val hash   = digest.digest(s"$customerId:$billingPeriod:akka-http-billing".getBytes("UTF-8"))
  hash.map("%02x".format(_)).mkString.take(32)
}

def chargeCustomer(
    customerId:    String,
    amount:        Long,
    currency:      String,
    billingPeriod: String,
    attempt:       Int = 0
)(implicit system: ActorSystem, ec: ExecutionContext): Future[String] = {

  // Content-hash key — identical on every invocation for the same customerId + billingPeriod.
  // Must NOT include: UUID.randomUUID() (new per call),
  //   System.currentTimeMillis() (different ms per call),
  //   attempt (increments per retry — produces different key per attempt),
  //   System.nanoTime() (nanosecond clock, always different),
  //   hostname (changes across Kubernetes pods on restart).
  val idempotencyKey = stableKey(customerId, billingPeriod)

  // Pre-flight: claim the billing slot before calling Stripe.
  // Returns true if inserted (first caller), false if row already exists (retry or duplicate).
  billingRepo.insertIfAbsent(customerId, billingPeriod, idempotencyKey).flatMap {
    case false =>
      // Billing record exists — original call succeeded or is in-flight.
      // Retrieve the existing charge ID; do not call Stripe.
      billingRepo.findChargeId(customerId, billingPeriod).flatMap {
        case Some(chargeId) => Future.successful(chargeId)
        case None           =>
          // Record exists but no charge ID yet — original call is still in-flight.
          // Wait and retry the pre-flight check; do not issue a second Stripe call.
          after(2.seconds)(chargeCustomer(customerId, amount, currency, billingPeriod, attempt + 1))
      }

    case true =>
      // No existing record — proceed with Stripe call using the stable key.
      val request = HttpRequest(
        method  = HttpMethods.POST,
        uri     = "https://api.stripe.com/v1/charges",
        headers = List(
          RawHeader("Authorization",   s"Bearer $stripeSecretKey"),
          RawHeader("Idempotency-Key", idempotencyKey)
        ),
        entity  = FormData(Map(
          "amount"      -> amount.toString,
          "currency"    -> currency,
          "customer"    -> customerId,
          "description" -> s"Billing period $billingPeriod"
        )).toEntity
      )

      Http().singleRequest(request).flatMap { response =>
        response.status match {
          case StatusCodes.OK | StatusCodes.Created =>
            Unmarshal(response.entity).to[StripeChargeResponse].flatMap { charge =>
              billingRepo.markCompleted(customerId, billingPeriod, charge.id).map(_ => charge.id)
            }

          case StatusCodes.TooManyRequests | StatusCodes.InternalServerError
               | StatusCodes.BadGateway    | StatusCodes.ServiceUnavailable if attempt < 3 =>
            response.entity.discardBytes()
            // Retry with the SAME idempotency key — Stripe deduplicates on its side
            // if ch_A was created before the error response was returned.
            after(Duration(500L * math.pow(2, attempt).toLong, "ms"))(
              chargeCustomer(customerId, amount, currency, billingPeriod, attempt + 1)
            )

          case _ =>
            response.entity.discardBytes()
            billingRepo.markFailed(customerId, billingPeriod)
              .flatMap(_ => Future.failed(new RuntimeException(s"Stripe error: ${response.status}")))
        }
      }.recoverWith {
        case _: akka.stream.StreamTcpException | _: java.net.ConnectException if attempt < 3 =>
          after(Duration(500L * math.pow(2, attempt).toLong, "ms"))(
            chargeCustomer(customerId, amount, currency, billingPeriod, attempt + 1)
          )
      }
  }
}

-- Pre-flight table
CREATE TABLE billing_records (
    customer_id      TEXT        NOT NULL,
    billing_period   TEXT        NOT NULL,
    idempotency_key  TEXT        NOT NULL,
    status           TEXT        NOT NULL DEFAULT 'pending',
    charge_id        TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    CONSTRAINT billing_records_pk PRIMARY KEY (idempotency_key),
    CONSTRAINT billing_records_uq UNIQUE (customer_id, billing_period)
);

-- insertIfAbsent: returns 1 row on first call, 0 rows if record already exists
INSERT INTO billing_records (customer_id, billing_period, idempotency_key)
VALUES ($1, $2, $3)
ON CONFLICT (customer_id, billing_period) DO NOTHING
RETURNING 1;

With this fix, the Stripe retry always uses the same content-hash key. If ch_A was created by the original call before Stripe returned a 500, the retry sends the same idempotency key and Stripe returns the cached ch_A response from its idempotency store — no second charge. The pre-flight check short-circuits the Stripe call entirely if the billing record already has a charge_id, protecting against cases where the Stripe call succeeded but the billing record update failed before the charge ID was persisted.

Failure mode 2: Http().cachedHostConnectionPool() wrapped in RetryFlow.withBackoff() — the retry callback rebuilds the request with a new UUID.randomUUID() — the connection pool’s internal retry fires transparently before the outer RetryFlow — ch_A exists from the pool-internal first attempt — ch_B created on the outer retry with the rebuilt request

Akka HTTP’s host connection pool (Http().cachedHostConnectionPool()) is a Flow[(HttpRequest, T), (Try[HttpResponse], T), HostConnectionPool]. It manages a pool of TCP connections to a single host and multiplexes requests across them. The experimental RetryFlow.withBackoff() (available from Akka HTTP 10.4+ or via akka-http-caching) wraps the pool flow and retries failed responses by feeding a new (HttpRequest, T) tuple back into the pool flow when the retry predicate returns Some:

// Akka HTTP 10.x — connection pool + RetryFlow
// UNSAFE: request builder called per retry, generates new UUID each time
import akka.http.scaladsl.util.FastFuture
import akka.stream.contrib.RetryFlow

val poolSettings = ConnectionPoolSettings(system)
  .withMaxConnections(10)
  .withMaxRetries(0)    // disable pool-internal retries — outer RetryFlow handles retry logic

val poolFlow: Flow[(HttpRequest, String), (Try[HttpResponse], String), HostConnectionPool] =
  Http().cachedHostConnectionPool[String]("api.stripe.com", 443, poolSettings)

val retryFlow: Flow[(HttpRequest, String), (Try[HttpResponse], String), NotUsed] =
  RetryFlow.withBackoff(
    minBackoff    = 500.millis,
    maxBackoff    = 30.seconds,
    randomFactor  = 0.2,
    maxRetries    = 3,
    flow          = poolFlow
  ) {
    case ((_, correlationId), (Failure(_), _)) =>
      // UNSAFE: buildStripeRequest() is called again, generating a new UUID.
      // If the original request reached Stripe and ch_A was created before the
      // pool returned Failure (TCP reset, idle-timeout socket close mid-response),
      // this retry sends a different idempotency key and creates ch_B.
      Some((buildStripeRequest(correlationId), correlationId))

    case ((_, correlationId), (Success(response), _))
        if response.status.isServerError =>
      response.entity.discardBytes()
      Some((buildStripeRequest(correlationId), correlationId))

    case _ => None
  }

def buildStripeRequest(correlationId: String): HttpRequest = {
  // UNSAFE: UUID generated per buildStripeRequest() call.
  // Called once for the original request and once per retry — different UUID each time.
  val idempotencyKey = UUID.randomUUID().toString
  HttpRequest(
    method  = HttpMethods.POST,
    uri     = Uri("https://api.stripe.com/v1/charges"),
    headers = List(
      RawHeader("Authorization",   s"Bearer $stripeSecretKey"),
      RawHeader("Idempotency-Key", idempotencyKey)
    ),
    entity  = billingFormData(correlationId).toEntity
  )
}

// Usage: enqueue billing requests into the retry-wrapped pool flow
val billingSource: Source[(HttpRequest, String), NotUsed] =
  Source(pendingBillingRequests.map { req =>
    (buildStripeRequest(req.correlationId), req.correlationId)
  })

billingSource.via(retryFlow).runForeach {
  case (Success(response), correlationId) => handleSuccess(response, correlationId)
  case (Failure(ex), correlationId)       => handleFailure(ex, correlationId)
}

The failure scenario: a billing request for correlationId = "bill_cust_123_2026Q3" is fed into the pool. buildStripeRequest("bill_cust_123_2026Q3") generates UUID = "4a5b6c7d...". The pool assigns the request to an idle connection. The request reaches Stripe’s API server. Stripe creates ch_A. The connection has been idle long enough that the upstream load balancer (or a NAT device) has silently closed it on its side while Akka HTTP’s pool still considers it alive — this is the “stale-connection” problem common in HTTP keep-alive environments. The pool receives a TCP RST when it tries to read the response. The pool emits a Failure(StreamTcpException). RetryFlow’s predicate fires. The retry callback calls buildStripeRequest("bill_cust_123_2026Q3") again. A new UUID.randomUUID() returns "8e9f0a1b...". The retry request is fed into the pool. A new connection is established. The request reaches Stripe with a new idempotency key. Stripe has never seen "8e9f0a1b...". ch_B is created.

The stale-connection failure is not hypothetical. AWS Application Load Balancers close idle keep-alive connections after 60 seconds by default. Akka HTTP’s pool connection idleTimeout defaults to 60 seconds as well — a race condition between the ALB’s 60-second close and Akka’s 60-second idle timeout means some connections are used immediately after the ALB has closed them. The pool returns Failure, RetryFlow retries, and if buildStripeRequest is called inside the retry callback, a new UUID is in play.

The fix for failure mode 2

The idempotency key must be derived once and passed with the correlation ID through the flow, so that the retry callback can retrieve the already-computed key rather than generating a new one. The correlation ID becomes a container for both the business identifier and the stable idempotency key. The retry callback reconstructs only the HTTP request structure — headers, entity, method — while re-using the same pre-computed key:

// Safe connection-pool billing — stable key passed through correlation context
case class BillingContext(correlationId: String, idempotencyKey: String)

def buildStripeRequest(ctx: BillingContext): HttpRequest =
  HttpRequest(
    method  = HttpMethods.POST,
    uri     = Uri("https://api.stripe.com/v1/charges"),
    headers = List(
      RawHeader("Authorization",   s"Bearer $stripeSecretKey"),
      // Re-uses ctx.idempotencyKey — same value on every retry
      RawHeader("Idempotency-Key", ctx.idempotencyKey)
    ),
    entity  = billingFormData(ctx.correlationId).toEntity
  )

val retryFlow: Flow[(HttpRequest, BillingContext), (Try[HttpResponse], BillingContext), NotUsed] =
  RetryFlow.withBackoff(
    minBackoff   = 500.millis,
    maxBackoff   = 30.seconds,
    randomFactor = 0.2,
    maxRetries   = 3,
    flow         = Http().cachedHostConnectionPool[BillingContext]("api.stripe.com", 443)
  ) {
    case ((_, ctx), (Failure(_), _)) =>
      // Rebuild the request with the SAME idempotency key from ctx — no new UUID.
      Some((buildStripeRequest(ctx), ctx))

    case ((_, ctx), (Success(response), _)) if response.status.isServerError =>
      response.entity.discardBytes()
      Some((buildStripeRequest(ctx), ctx))

    case _ => None
  }

// Prepare billing contexts before feeding into the flow
val billingSource: Source[(HttpRequest, BillingContext), NotUsed] =
  Source(pendingBillingRequests.map { req =>
    // Compute the stable key once per billing intent, before the flow.
    val key = stableKey(req.customerId, req.billingPeriod)
    val ctx = BillingContext(
      correlationId  = s"bill_${req.customerId}_${req.billingPeriod}",
      idempotencyKey = key
    )
    // Pre-flight: claim the billing slot before entering the flow.
    // Source is lazy — this runs at materialization time.
    (buildStripeRequest(ctx), ctx)
  })

billingSource.via(retryFlow).runForeach {
  case (Success(response), ctx) => handleSuccess(response, ctx)
  case (Failure(ex), ctx)       => handleFailure(ex, ctx)
}

The pre-flight database check should run before the billing context is created and the request is fed into the flow. A mapAsync stage upstream of retryFlow can issue the INSERT ... ON CONFLICT DO NOTHING for each pending billing request and filter out records that already exist, so that only genuinely pending billing intents enter the pool flow. This ensures that even if the same billing request is enqueued twice — due to a restart of the upstream scheduler or a double-invocation from a background job — only one Stripe call is issued.

Failure mode 3: withRequestTimeout fires a 503 before the Stripe call completes — the upstream load balancer retries the request to a second Akka HTTP pod — the second pod generates a new UUID.randomUUID() in its billing route — Stripe creates ch_B while ch_A is still processing on the first pod

The Akka HTTP server-side directive withRequestTimeout(duration) wraps a route and fires the timeout response if the route’s Future does not complete within duration. When a billing route calls Stripe via Http().singleRequest() and Stripe’s API latency exceeds the configured timeout, the server returns a 503 (or the timeout response configured via withRequestTimeoutResponse) before the Stripe call completes. If the upstream caller has retry-on-5xx configured — as AWS ALB proxy_next_upstream rules, nginx proxy_next_upstream timeout, or an upstream gRPC gateway will have — the identical request is retried on a second Akka HTTP pod:

// Akka HTTP 10.x — billing route with withRequestTimeout
// UNSAFE: UUID generated per route invocation, not per billing intent
val billingRoute: Route =
  path("v1" / "billing") {
    post {
      withRequestTimeout(15.seconds) {
        entity(as[BillingRequest]) { req =>
          complete {
            // UNSAFE: UUID.randomUUID() generated at route-invocation time.
            // Pod A (attempt 1): key = "b1c2d3e4-f5a6-4b7c-8d9e-0f1a2b3c4d5e"
            // Pod B (attempt 2): key = "c2d3e4f5-a6b7-4c8d-9e0f-1a2b3c4d5e6f" (different pod, different UUID)
            val idempotencyKey = UUID.randomUUID().toString

            stripeService.chargeCustomer(
              customerId    = req.customerId,
              amount        = req.amount,
              billingPeriod = req.billingPeriod,
              idempotencyKey = idempotencyKey
            ).map { chargeId =>
              HttpResponse(
                status = StatusCodes.OK,
                entity = HttpEntity(ContentTypes.`application/json`,
                  s"""{"charge_id":"$chargeId","status":"success"}""")
              )
            }
          }
        }
      }
    }
  }

The failure scenario: an agent issues a POST /v1/billing for cust_456, September 2026 billing period. The request lands on Pod A. Pod A’s route invocation generates UUID = "b1c2d3e4...". Pod A calls Stripe with this key. Stripe begins processing the charge. Stripe’s API latency on this request is 18 seconds — within Stripe’s documented tail latency envelope, but exceeding Pod A’s withRequestTimeout(15.seconds). After 15 seconds, Pod A’s timeout handler fires and returns a 503 to the upstream load balancer. The Stripe call is still in-flight on the JVM thread pool; Pod A’s Future will complete in 3 more seconds and create ch_A.

The upstream ALB has a routing rule: retry 5XX responses on a different target. It selects Pod B and re-sends the identical POST /v1/billing body. Pod B’s route invocation generates UUID = "c2d3e4f5...". Pod B calls Stripe with this key. Stripe has never seen "c2d3e4f5..." — ch_A is not yet committed to Stripe’s idempotency store because it is still processing. Pod B’s call creates ch_B. Three seconds later, Pod A’s Stripe call completes and ch_A is committed. Customer 456 is charged twice for September 2026.

The failure is particularly insidious because both pods believe they acted correctly. Pod A timed out and returned a 503 as designed. Pod B received a legitimate billing request and processed it as designed. No error logs appear on either pod. The duplicate appears only in Stripe’s dashboard or in a billing reconciliation query that counts charges per customer per period.

The fix for failure mode 3

The idempotency key must come from the upstream caller, not from the server-side route. The correct architecture is for the caller to compute a stable key per billing intent and send it as a request header or field. The server route extracts the caller-supplied key and passes it to Stripe unchanged. If the ALB retries to Pod B, Pod B extracts the same key from the request (the ALB re-sends the full request body and headers), and Stripe deduplicates using the key already in its idempotency store from Pod A’s in-flight call:

// Safe billing route — idempotency key from caller, pre-flight DB check
val billingRoute: Route =
  path("v1" / "billing") {
    post {
      // The request timeout should be longer than Stripe's p99 latency.
      // Stripe's documented p99 for charges is ~8s; set timeout to 30s minimum.
      withRequestTimeout(30.seconds) {
        entity(as[BillingRequest]) { req =>
          // Extract caller-supplied idempotency key — sent by the upstream agent
          // as Idempotency-Key: sha256(customerId:billingPeriod:akka-http-billing)[:32].
          // If not supplied (legacy callers), derive it server-side from req fields.
          headerValueByName("Idempotency-Key") { callerKey =>
            complete {
              // Pre-flight: claim billing slot before calling Stripe.
              // If Pod B processes the ALB retry, it inserts the same key
              // and finds that Pod A's record already exists (even if still 'pending').
              billingRepo.insertIfAbsent(req.customerId, req.billingPeriod, callerKey)
                .flatMap {
                  case 0 =>
                    // Record exists — Pod A already claimed this billing intent.
                    // Return the existing result or wait for Pod A's Stripe call to complete.
                    billingRepo.findChargeId(req.customerId, req.billingPeriod).map {
                      case Some(chargeId) =>
                        HttpResponse(StatusCodes.OK,
                          entity = HttpEntity(ContentTypes.`application/json`,
                            s"""{"charge_id":"$chargeId","status":"success","deduplicated":true}"""))
                      case None =>
                        // Pod A's Stripe call is still in-flight.
                        // Return 202 Accepted; caller should poll for completion.
                        HttpResponse(StatusCodes.Accepted,
                          entity = HttpEntity(ContentTypes.`application/json`,
                            s"""{"status":"processing","message":"Billing in progress"}"""))
                    }

                  case _ =>
                    // First caller for this billing intent — proceed with Stripe.
                    stripeService.chargeCustomer(
                      customerId     = req.customerId,
                      amount         = req.amount,
                      billingPeriod  = req.billingPeriod,
                      idempotencyKey = callerKey  // pass caller's stable key to Stripe
                    ).flatMap { chargeId =>
                      billingRepo.markCompleted(req.customerId, req.billingPeriod, chargeId)
                        .map { _ =>
                          HttpResponse(StatusCodes.OK,
                            entity = HttpEntity(ContentTypes.`application/json`,
                              s"""{"charge_id":"$chargeId","status":"success"}"""))
                        }
                    }
                }
            }
          } ~ complete {
            // No Idempotency-Key header — derive from request body fields.
            // This path is safe as long as the derived key is stable across retries.
            val derivedKey = stableKey(req.customerId, req.billingPeriod)
            // ... same logic as above with derivedKey
            Future.successful(HttpResponse(StatusCodes.BadRequest,
              entity = HttpEntity(ContentTypes.`application/json`,
                """{"error":"Idempotency-Key header required"}""")))
          }
        }
      }
    }
  }

The upstream request timeout must also be increased. If withRequestTimeout(30.seconds) still fires before Stripe completes on high-tail-latency days, the caller should be designed to handle 202 Accepted and poll for the billing result rather than treating any non-200 as a failure to retry. The correct retry surface for a billing operation is a separate idempotent GET /v1/billing/{billingPeriod} endpoint that reads from the billing record table — not a retry of the original POST that would re-enter the billing route with any hope of a different outcome.

The vault-key backstop

Content-hash keys and pre-flight database checks close each of these failure modes at the application layer. They require correct implementation in every billing path, every retry handler, and every framework integration. A single missed UUID.randomUUID() in a code path added six months later re-opens the vulnerability without triggering any test or static analysis warning.

A spend-cap proxy at the vendor API layer is the hard backstop that does not depend on application-layer correctness. Keybrake issues a scoped vault key per billing period: vault_key_cust_123_2026_09 with a policy {"vendor":"stripe","daily_usd_cap":99,"period_usd_cap":110,"expires_at":"2026-10-01T00:00:00Z"}. All Akka HTTP billing calls — from recursive Future retries, from RetryFlow, from timeout-retried cross-pod calls — go through the proxy URL rather than directly to api.stripe.com. The proxy enforces the cap in real time: the 101st dollar in a $99/period billing period is blocked at the network layer before it reaches Stripe, regardless of how many idempotency keys were generated by which pod’s retry handler.

The cap is set at expected_total × 1.10 — ten percent over the legitimate billing amount — to allow Stripe’s own idempotency deduplication to absorb the first retry before the cap fires. If a content-hash key retry reaches Stripe and Stripe returns the cached ch_A response, the proxied total stays at one charge. If both calls reach Stripe with different keys and ch_A and ch_B are both created, the cap fires on the second charge. A billing period that was $99 stops at $108.90 instead of continuing until all three recursive retry attempts complete.

Summary

The pattern across all three failure modes is the same as in Akka Streams, Akka Typed EventSourcedBehavior, and Netty Pipeline Handlers: the framework gives you retry, reconnect, and timeout primitives that are natural for HTTP I/O handling but dangerous for billing, because they re-invoke your billing code at exactly the moment when a per-invocation value in the idempotency key produces a different Stripe key. A content-hash key derived from business fields only — sha256(customerId + ":" + billingPeriod + ":akka-http-billing")[:32] — is identical across all invocations for the same billing intent, regardless of which pod handles the request, how many recursive retries fire, or what the connection pool’s internal state is. The pre-flight ON CONFLICT DO NOTHING database check is the authoritative gate. The vault key is the backstop when both fail.