@RefreshScope bean destruction clears in-flight key cache: developer stores in-flight idempotency keys in a ConcurrentHashMap instance field on a @RefreshScope-annotated BillingService; a Spring Cloud Bus RefreshRemoteApplicationEvent fires between a timed-out Stripe call (which committed ch_A) and the scheduled retry; ContextRefresher.refresh() marks the bean stale; the proxy re-creates the bean; the new instance has an empty inflightKeys map; computeIfAbsent() runs UUID.randomUUID() → UUID_B; Stripe commits ch_B alongside ch_A; developer misconception: “@RefreshScope updates injected @Value fields but does not touch instance state I manage myself — @RefreshScope does a property patch, not a full reconstruction.” @Value-bound prefix changed on config refresh: developer derives Stripe idempotency key as idempotencyPrefix + “:” + subscriptionId + “:” + billingPeriod where idempotencyPrefix is a @Value("${billing.idempotency.prefix}") field; Config Server’s Git repository is updated from billing to billing-v2 between the first Stripe call and the retry; ContextRefresher.refresh() re-creates the bean with the new prefix; retry key is billing-v2:sub_001:2026-10 — Stripe’s idempotency cache has no entry for this string — Stripe processes it as a new charge and commits ch_B; developer misconception: “Stripe deduplicates based on customer, amount, and currency — two requests for the same customer and amount should be deduplicated even if the key strings differ.” {cipher} decryption failure returns sentinel: developer stores the HMAC signing secret used to derive idempotency keys as billing.idempotency.hmac-secret={cipher}AQBk...; Config Server keystore rotation causes decryption failure; with server-side fail-on-error=false (the default), Config Server returns hmac-secret=<n/a> instead of throwing a 5xx; client receives the sentinel as a valid string property; HMAC("<n/a>", subscriptionId:billingPeriod:amountCents) produces key_B ≠ key_A; Stripe commits ch_B; developer misconception: “spring.cloud.config.fail-fast=true catches config fetch failures including decryption failures — the application will fail loudly rather than silently accept a sentinel.” Fix for all three: derive key from stable business inputs only (subscriptionId + billingPeriod + amountCents, plain SHA-256) — no config-backed components, no in-flight cache; set spring.cloud.config.server.encrypt.fail-on-error=true; add @PostConstruct sentinel guard on hmacSecret field. Spring Cloud Config 2023.x + Spring Cloud Bus + Spring Retry + WireMock integration tests for all three modes.
@Retryable UUID reset after VersionConflictEngineException: developer generates UUID.randomUUID() at the top of a @Retryable method and calls Stripe before calling subscriptionRepo.save(); a concurrent ES write increments the document’s seq_no between load and save; Elasticsearch returns a 409 Conflict and Spring Data Elasticsearch throws VersionConflictEngineException after Stripe commits ch_A; @Retryable re-executes the method body from line 1 — UUID.randomUUID() generates UUID_B — Stripe creates ch_B alongside ch_A; developer misconception: “The OCC conflict means a concurrent billing attempt modified the document — the COMPLETED status will be visible on retry and the guard will catch it.” Near-real-time (NRT) search gap: developer writes an IdempotencyDocument to Elasticsearch before calling Stripe and queries it on recovery via findBySubscriptionIdAndBillingPeriod() (a SearchRequest); the 1-second default index.refresh_interval means the document is not yet visible to search when recovery runs within the NRT window; query returns Optional.empty(); recovery generates UUID_B and calls Stripe; ch_B committed alongside ch_A; developer misconception: “Elasticsearch acknowledged my write synchronously — the document is immediately searchable.” Auto-generated @Id lost in IndexResponse timeout: developer creates a PaymentDocument with null @Id, calls save() to get the ES-assigned document ID, then derives the Stripe key from that ID; a network timeout reading the IndexResponse prevents the application from receiving the auto-generated ID; retry creates a second document with a new auto-generated ID; Stripe key derived from the new ID is UUID_B; ch_B committed; developer misconception: “If save() throws an exception the document was not indexed — the retry starts fresh with a unique ID.” Fix for all three: derive Stripe key and document @Id from stable business inputs (subscriptionId + billingPeriod + amountCents) before any ES or Stripe call; use findById() (GetRequest, immediately consistent) for guard checks instead of search queries; optionally use RefreshPolicy.IMMEDIATE on guard writes if search queries are required. Spring Data Elasticsearch 5.x + Spring Retry + Testcontainers Elasticsearch 8.x + WireMock integration tests for all three modes.
RequestInterceptor.apply() on every Feign Retryer attempt: Feign calls clone().apply() before each retry — developer puts UUID.randomUUID() inside a RequestInterceptor bean — TCP drops after Stripe commits ch_A — RetryableException thrown — Retryer.Default fires — interceptor generates UUID_B on the retry attempt — Stripe creates ch_B alongside committed ch_A; developer misconception: “The Retryer retries on connection failures — if Stripe committed the charge it would have sent a 200 response before the TCP connection dropped.” Spring Cloud LoadBalancer retryOnAllOperations=true: LoadBalancer RetryLoadBalancerInterceptor re-issues the full POST request to the next service instance after a network timeout — sits above the Feign layer — the interceptor pipeline (including the UUID.randomUUID() interceptor) runs fresh for each instance attempt — billing-service-1 commits ch_A on Stripe before timing out — billing-service-2 receives a retried POST with UUID_B from the interceptor and commits ch_B; developer misconception: “Internal billing services are idempotent — retrying on the next instance is safe — the timeout means the first instance never charged the customer.” @FeignClientfallbackFactory UUID after TimeLimiter fires: Spring Cloud Circuit Breaker TimeLimiter fires at 3s deadline on the calling thread — Stripe finished processing and committed ch_A at t=1.8s but the response was in the socket buffer when the timer fired — FallbackFactory.create(TimeoutException) is called — fallback method generates UUID.randomUUID() assuming Stripe never responded — directStripeService.charge(UUID_B) commits ch_B; developer misconception: “A TimeoutException from the TimeLimiter means Stripe didn’t respond in time — no response before the deadline means no charge was committed.” Fix for all three: content-hash idempotency key from stable business inputs generated before any Feign call; passed as @RequestHeader parameter or embedded in request body; RequestInterceptor UUID generation removed entirely; fallback must reuse the idempotencyKey parameter received from the original method invocation, not generate a new one. @SpringBootTest + WireMock fault injection + Testcontainers PostgreSQL integration tests for all three modes asserting Idempotency-Key header stability (or instability in the bug-demonstration tests) across retry and fallback paths.
Three Spring Data JPA Hibernate Envers batch charging Stripe idempotency failure modes. AuditReader.getRevisions() as idempotency guard: developer queries the Envers revision list before each Stripe call — reasoning “if Stripe committed, a MOD revision would exist” — a JDBC transaction rolls back after Stripe commits ch_A (connection pool timeout, constraint violation), rolling back the Envers revision alongside the entity update — recovery queries AuditReader.getRevisions(Payment.class, id), finds only the original ADD revision, concludes “never charged”, generates UUID_B, Stripe creates ch_B alongside committed ch_A; developer misconception: “No MOD Envers revision ⇒ Stripe was never called — Envers and Stripe have the same transaction boundary.” @PrePersist UUID generation on @Audited entity: developer puts UUID.randomUUID() in a @PrePersist callback on the Payment entity — @PrePersist fires at persist() time; Hibernate queues the INSERT but defers the flush; Stripe call uses the in-memory UUID_A; a constraint violation at flush time rolls back the entity INSERT, discarding UUID_A from disk; next attempt constructs a fresh entity, @PrePersist generates UUID_B, Stripe creates ch_B; developer misconception: “The key is persisted with the entity — I can look it up by subscription ID and billing period on retry.” The lookup returns null because the INSERT never committed. OPTIMISTIC_FORCE_INCREMENT fires at flush after Stripe commits: developer uses LockModeType.OPTIMISTIC_FORCE_INCREMENT as a concurrency guard, believing the version check fires before side effects — Hibernate performs the UPDATE payments SET version=N+1 WHERE id=? AND version=N check at flush time, not at load time — a Stripe call between load and flush commits ch_A before the version check; a concurrent version increment causes the check to find zero matching rows and throw OptimisticLockException; the Envers COMPLETED revision is rolled back; recovery loads the payment, sees PENDING status (concurrent update left it PENDING), generates UUID_B, Stripe creates ch_B; developer misconception: “OPTIMISTIC_FORCE_INCREMENT acquires an exclusive-like lock at load time — if the lock fails, my transaction was aborted before reaching Stripe.” Fix for all three: content-hash idempotency key derived from stable business inputs (subscriptionId + billingPeriod + amountCents) before entity construction; key is deterministic across all retry attempts regardless of how many JDBC transactions fail. @SpringBootTest + WireMock + Testcontainers PostgreSQL integration tests for all three modes.
Three Spring Cloud Circuit Breaker Stripe idempotency failure modes using the spring-cloud-starter-circuitbreaker-resilience4j factory API. CircuitBreakerFactoryTimeLimiter timeout: the default 1-second TimeLimiterConfig fires TimeoutException on the calling thread after Stripe has already committed ch_A — the fallback returns FAILED — the caller retries with a new UUID.randomUUID() — ch_B created; developer misconception: “The TimeLimiter fired because Stripe did not respond in time — if a charge had been committed the HTTP response would have arrived before the timeout.” @CircuitBreaker AOP proxy ordering: Resilience4j’s CircuitBreakerAspect is registered at Integer.MAX_VALUE - 2 (outer), Spring’s TransactionInterceptor at Integer.MAX_VALUE (inner) — when paymentRepository.save() throws after Stripe commits ch_A, @Transactional rolls back and re-throws — @CircuitBreaker fires the fallback which queries the now-empty database and concludes “no charge committed” — retry generates UUID_B — ch_B; developer misconception: “@Transactional rollback atomically undid the charge attempt — the fallback correctly saw no committed state in the database.” CLOSED-to-OPEN transition mid-batch: Resilience4j transitions synchronously inside the call that crosses the failure threshold — that call ran its supplier (including the Stripe HTTP call) before the transition — subsequent calls receive CallNotPermittedException — developer re-queues all failed items including the threshold-crosser with new UUIDs — ch_B for the threshold-crosser; developer misconception: “CallNotPermittedException means the circuit was already OPEN when my call was rejected — no HTTP call was ever made.” Fix for all three: content-hash idempotency key from stable business inputs. @SpringBootTest + WireMock + Testcontainers PostgreSQL integration tests.
Three Kafka Streams DSL Stripe idempotency failure modes distinct from EOS semantics. suppress(untilWindowCloses) crash-restore: RocksDB state store replays past the checkpoint watermark, pushing the window watermark past the end again; suppress() emits the same window record twice to the output topic; billing consumer processes the duplicate with UUID.randomUUID() — ch_B alongside committed ch_A; developer misconception: “suppress() guarantees at-most-one emission per window — each window record is emitted exactly once regardless of crashes.” EXACTLY_ONCE_V2 + .foreach() Stripe call: the Kafka transaction covers only Kafka producer writes and consumer offset commits; the Stripe HTTP call inside .foreach() happens outside the transaction boundary; a producer epoch fence causes the Streams runtime to abort the transaction and re-execute the processing step; .foreach() fires again with a fresh UUID — ch_B; developer misconception: “EXACTLY_ONCE_V2 means the entire topology executes exactly once, including all side effects.” STREAM_TIME punctuator multi-firing: the 24-hour billing punctuator advances with event timestamps; a 66-hour batch of backlogged records advances stream time past three 24-hour boundaries in one processing loop; the Streams runtime fires the punctuator three times — each invocation calls Stripe with UUID.randomUUID() — three charges created; developer misconception: “STREAM_TIME advances one day per day of event data — a 66-hour batch fires the daily punctuator once at the end.” Content-hash key from stable billing-period inputs prevents all three failure modes.
Three Spring Boot Actuator Stripe idempotency failure modes. Custom @Endpoint@WriteOperation: developer builds POST /actuator/billing that generates UUID.randomUUID() per invocation — an HTTP client retry on SocketTimeoutException (first response lost in transit) sends a second POST — Actuator has no per-endpoint idempotency layer — second invocation generates UUID_B — Stripe creates ch_B alongside committed ch_A; developer misconception: “The client retries only on 5xx — if the endpoint returned 200 the client won’t retry.” ApplicationStartedEvent PENDING recovery: developer scans for PENDING charge rows on startup and calls stripe.charges().create() with a fresh UUID.randomUUID() — PENDING was written BEFORE the Stripe call — a JVM crash after Stripe committed ch_A but before billingRepository.save(COMPLETED) leaves the row in PENDING — recovery interprets PENDING as “Stripe never called” and creates ch_B; developer misconception: “PENDING means the charge was not committed — if Stripe had succeeded the row would have been updated to COMPLETED.” @SchedulerLocklockAtMostFor TTL shorter than batch runtime: ShedLock’s lockUntil timestamp expires while the first node is still processing — second node acquires the lock — re-queries the charge window — finds customers where chargedAt IS NULL because the first node has not yet committed — calls Stripe for each with UUID.randomUUID() — ch_B created alongside in-flight ch_A; developer misconception: “lockAtMostFor is a crash-protection safeguard — a live node running the task holds the lock until it releases it.” Fix for all three: content-hash idempotency key stored in the PENDING row before Stripe call; on recovery, reuse stored key. @SpringBootTest + WireMock + Testcontainers PostgreSQL tests.
Three Spring Kafka Streams EOS Stripe idempotency failure modes. StreamsUncaughtExceptionHandler.REPLACE_THREAD: the replacement StreamThread restores tasks from the last committed EOS checkpoint and re-processes all records in the uncommitted batch — processor code (including UUID.randomUUID()) re-executes from the beginning — UUID_B generated — Stripe creates ch_B alongside committed ch_A; developer misconception: “processing.guarantee=exactly_once_v2 guarantees each input record triggers exactly one Stripe charge.” Processor API KeyValueStore guard: developer writes a state store entry after calling Stripe and checks the store before calling Stripe — EOS rolls back the state store entry atomically with the consumer offset advance when the task fails — on re-delivery the guard is empty (was rolled back) and Stripe is called again with UUID_B; developer misconception: “If the state store entry is absent, the EOS transaction was rolled back — no prior Stripe call committed.” TaskMigratedException during cooperative rebalance: cooperative partition revocation races with an in-flight EOS commit — CommitFailedException (wrapped as TaskMigratedException) aborts the active EOS transactions — state store writes and output records rolled back — new task owner restores from changelog (no entry) and re-processes with UUID_B; developer misconception: “Cooperative rebalancing commits in-flight work before relinquishing partition ownership — the new task owner starts from a committed state.” Content-hash key from stable PaymentCommand fields; relational DB PENDING row (in its own transaction, not governed by Kafka Streams EOS) written before Stripe call — survives EOS aborts; fast-path KeyValueStore guard for post-commit lookups. Testcontainers Kafka EOS + WireMock + PostgreSQL integration tests asserting Idempotency-Key header stability across REPLACE_THREAD and EOS-abort re-processing. Comparison table (root cause, trigger, developer misconception, detection signal, fix).
Three Spring Kafka EOS Stripe idempotency failure modes. DefaultErrorHandler seek-and-re-poll: when the @KafkaListener method throws DataAccessException after Stripe already committed ch_A, DefaultErrorHandler calls consumer.seek(TopicPartition, offset) to reset the partition position to the failed record — on re-poll the container calls the listener method from line 1 — UUID.randomUUID() at method entry generates UUID_B — Stripe creates ch_B alongside committed ch_A; developer misconception: “DefaultErrorHandler retries the failed DB write step — the Stripe call at the top of the method already succeeded and will not be re-executed.” @Retryable + @Transactional(kafkaTransactionManager) AOP ordering: RetryOperationsInterceptor (order Integer.MAX_VALUE-5, outer) calls MethodInvocation.proceed() on retry, re-invoking TransactionInterceptor (inner), which opens a new KafkaTransactionManager transaction and re-executes the full method body from line 1 — UUID.randomUUID() generates UUID_B — Stripe creates ch_B; developer misconception: “@Retryable retries the DataAccessException JDBC error — the KafkaTransactionManager wraps the Kafka send, not the Stripe call — the Stripe call is outside the retry scope.” ChainedKafkaTransactionManager sequential commit: CKTM commits DataSourceTransactionManager first, then KafkaTransactionManager — if the Kafka commit throws KafkaException after JDBC has committed, the payment_records row with UUID_A is permanently durable (committed JDBC transactions cannot be rolled back) while the consumer offset is not advanced — container seeks the partition back — listener re-called from line 1 — UUID.randomUUID() generates UUID_B — Stripe creates ch_B; developer misconception: “CKTM provides atomic commit for both Kafka and JDBC — Kafka commit failure rolls back the JDBC write.” Content-hash key, PENDING row INSERT in REQUIRES_NEW before Stripe, existence guard at listener entry. @SpringBootTest + WireMock + Testcontainers Kafka + PostgreSQL tests for all three modes. Comparison table (root cause, trigger, developer misconception, detection signal, fix).
Three Micronaut Kafka EOS Stripe idempotency failure modes distinct from all prior Micronaut Kafka posts. Producer fencing: the transactional producer calls beginTransaction(), calls Stripe (ch_A committed), calls producer.send(), then is fenced by the Kafka coordinator — abortTransaction() cancels the Kafka message but Stripe’s HTTP commit is not rolled back — the new producer epoch replays the same message — UUID.randomUUID() at method entry generates UUID_B — Stripe creates ch_B alongside committed ch_A; developer misconception: “abortTransaction() rolls back the entire unit of work including external API calls — the Stripe charge is cancelled when the Kafka transaction is aborted.” read_committed concurrent consumer guard race: partition rebalance assigns the same record to two consumer instances — the first consumer inserts a PENDING row (uncommitted) and calls Stripe — the second consumer’s existsByIdempotencyKey() check runs under READ COMMITTED which does not see the first consumer’s uncommitted PENDING row — both consumers call Stripe with UUID.randomUUID() — ch_A and ch_B both committed; developer misconception: “isolation.level=read_committed on the Kafka consumer means each partition record is processed by exactly one consumer instance at a time.” sendOffsetsToTransaction() consume-transform-produce: Stripe is called inside the EOS loop before commitTransaction() — JVM crash between producer.send() and commitTransaction() causes the Kafka coordinator to abort the transaction after timeout — the produced message on the output topic is rolled back — the consumer offset for the input topic is rolled back — consumer re-reads the record — UUID.randomUUID() generates UUID_B — ch_B committed alongside ch_A; developer misconception: “sendOffsetsToTransaction() makes the consume-transform-produce pipeline fully atomic — aborting the transaction rolls back the Stripe charge.” Content-hash key from stable message fields, PENDING row INSERT in REQUIRES_NEW before Stripe, Stripe call outside beginTransaction()/commitTransaction() boundary. @MicronautTest + WireMock + Testcontainers Kafka EOS + PostgreSQL tests for all three modes.
Three Micronaut Kafka Stripe idempotency failure modes. @KafkaListener with @Retryable method body re-entry: Micronaut’s RetryInterceptor calls MethodInvocationContext.proceed() again on each retry attempt — the annotated method executes from line 1 — UUID.randomUUID() at method entry generates UUID_B — Stripe creates ch_B alongside committed ch_A; developer misconception: “@Transactional rolled back the DB write — no record of ch_A — safe to retry via @Retryable.” @KafkaClient non-transactional inside @Transactional: @KafkaClient uses the Kafka producer’s NetworkClient, which is not a JTA resource — message reaches the broker before @Transactional commits or rolls back — JDBC rollback does not undo the send — @Retryable sends a second message on retry — consumer processes both messages with UUID.randomUUID() per invocation — ch_A and ch_B on Stripe; developer misconception: “@Transactional wraps the @KafkaClient send — rolling back prevents the message from reaching the broker.” batch=true mid-batch failure redelivers the entire batch: Micronaut Kafka does not commit the Kafka consumer group offset until the entire batch method returns successfully — a DataAccessException on record N causes the method to throw — offset not committed — Kafka redelivers all records from the last committed offset — records 1–N-1 were already charged on Stripe — redelivery invokes the method again — UUID.randomUUID() per record generates new keys — ch_B for each; developer misconception: “Records 1–N-1 were saved successfully to the DB — Kafka will redeliver only from offset N.” Content-hash key from stable message fields, PENDING row INSERT before Stripe call, and idempotent consumer logic. @MicronautTest + WireMock + Testcontainers Kafka + PostgreSQL tests for all three modes. Comparison table (root cause, trigger, developer misconception, detection signal, fix).
Three Spring Data JPA Hibernate Envers Stripe idempotency failure modes. @Retryable + @Transactional + Envers _AUD rollback: @Transactional rolls back both the payment_records base-table INSERT and the Envers payment_records_aud revision INSERT, leaving no audit trail for key_A or ch_A — developer concludes the system is clean and safe to retry — UUID.randomUUID() at method entry generates UUID_B on @Retryable attempt 2 — Stripe creates ch_B alongside committed ch_A; developer misconception: “No Envers revision for this entity ID means no Stripe charge was committed.” AuditReader.forRevisionsOfEntity() concurrent race: two concurrent requests both call forRevisionsOfEntity() inside READ COMMITTED@Transactional before either has committed an Envers revision — both see an empty result list — both proceed to Stripe with distinct UUID keys — ch_A and ch_B both committed — second paymentRecordRepository.save() hits UNIQUE constraint; developer misconception: “READ COMMITTED means I see all committed rows — if AuditReader returns empty, no prior charge was committed.” AuditReader.getCurrentRevision(true) sequence gap: getCurrentRevision(true) immediately inserts a REVINFO row and consumes a PostgreSQL SEQUENCE value — on @Transactional rollback the REVINFO row is rolled back but the sequence counter permanently advances from 1000 to 1001 (PostgreSQL sequences are non-transactional by specification) — on @Retryable retry, getCurrentRevision(true) allocates sequence value 1001 — developer using "rev-" + revisionId as the Stripe key sends rev-1001 instead of rev-1000 — Stripe creates ch_B; developer misconception: “getCurrentRevision() is transaction-scoped — rollback reverts the revision ID — retry reuses the same ID.” Content-hash key from stable business inputs, PENDING row INSERT before Stripe call, and UNIQUE constraint as concurrent guard. @SpringBootTest + WireMock + Testcontainers PostgreSQL integration tests for all three modes. Comparison table (root cause, trigger, developer misconception, detection signal, fix).
Three Quarkus SmallRye Fault Tolerance Stripe idempotency failure modes. @Retry CDI method body re-entry: @Retry intercepts at the CDI proxy boundary and re-invokes the entire annotated method from line 1 when billingRepository.persist() throws PersistenceException after Stripe already created ch_A — UUID.randomUUID() at method entry generates UUID_B on each attempt — ch_B created alongside committed ch_A; developer misconception: “@Transactional rollback means the Stripe charge was undone — retry is safe.” @Fallback with fresh UUID after timeout-masked committed charge: @Retry exhaustion triggers @Fallback method — fallback generates UUID.randomUUID() assuming all retry attempts threw before Stripe responded — a read-timeout StripeException can mean Stripe processed the charge and sent HTTP 200 but the TCP response was lost — fallback creates ch_C for a customer who may already have ch_A and ch_B; developer misconception: “StripeException wrapping SocketTimeoutException means Stripe never received the request.” @Timeout fires after Stripe responds but before DB write commits: Stripe HTTP call completes at t=1800ms within the 2000ms budget — billingRepository.persist() runs at t=1800ms under secondary-index write pressure — @Timeout fires at t=2000ms during the database write phase — @Transactional rolls back only the JDBC write — @Retry re-executes the method body — UUID_B generated — ch_B created; developer misconception: “@Timeout cancelled the operation — the Stripe charge was not committed.” Content-hash key, pre-Stripe PENDING row insert, stable key in @Fallback, and narrowing @Retry scope to idempotency-safe operations. @QuarkusTest + WireMock + Testcontainers PostgreSQL integration tests for all three modes asserting Idempotency-Key header stability (or instability) across retry attempts. Comparison table (root cause, trigger, developer misconception, detection signal, fix).
Three Micronaut Data JDBC–specific Stripe idempotency failure modes distinct from all prior Micronaut posts. saveAll() batch retry: service builds BillingAttempt entities with UUID.randomUUID() per item, calls Stripe per item, then saveAll() — if saveAll() throws DataAccessException the database rolls back but Stripe charges already committed — @Retryable re-enters method body — new UUID per item — ch_B for every customer already charged; developer misconception: “saveAll() threw so no charges went through.” @Query ON CONFLICT DO NOTHING guard bypass: guard is semantically correct but UUID.randomUUID() guarantees every invocation uses a key that has never been inserted — ON CONFLICT can never trigger — Stripe called on every invocation; developer misconception: “ON CONFLICT DO NOTHING detects a prior attempt and skips Stripe.” READ COMMITTED TOCTOU race: two concurrent requests compute the same content-hash key — both call findByIdempotencyKey(key) — both see empty (neither has committed yet) — both proceed to Stripe — second save hits uniqueness constraint; developer misconception: “READ COMMITTED is fine because once the first request commits the second will see the row.” Content-hash key, pre-insert PENDING row before Stripe, and DataIntegrityViolationException as the concurrent-safe guard. Comparison table (root cause, trigger, developer misconception, detection signal, fix). @MicronautTest + WireMock + Testcontainers PostgreSQL integration tests for all three modes.
Three Spring Data R2DBC advanced-pattern Stripe idempotency failure modes. @Transactional reactive method + .retryWhen() on the caller: Spring’s ReactiveTransactionInterceptor wraps the method in Mono.defer() — each .retryWhen() re-subscription re-runs the method body — UUID.randomUUID() generates UUID_B — ch_B alongside committed ch_A; developer misconception: “.retryWhen() retries the Stripe call, not the @Transactional boundary — UUID_A stays stable.” Flux.merge() with downstream retryWhen(): merge() subscribes all source publishers simultaneously — retryWhen() re-subscribes the entire merged Flux — all per-customer billing Monos restart — UUID_B for every customer including those already charged; developer misconception: “retryWhen() retries only the failed element, not all concurrent sources.” Mono.cache() per service method invocation: Mono.fromCallable(() -> UUID.randomUUID().toString()).cache() memoizes UUID_A within one instance — caller retry calls the service method again — new Mono.cache() instance — no memoized value — callable re-runs — UUID_B; developer misconception: “Mono.cache() locks in UUID_A for the duration of the billing operation.” Content-hash key derivation, DB guard, per-element retryWhen() fixes. @DataR2dbcTest + WireMock integration tests for all three modes. Comparison table (root cause, trigger, developer misconception, detection signal, fix).
Three Quarkus SmallRye Reactive Messaging Stripe idempotency failure modes. @Incoming + @Transactional consumer: Stripe called inside @Transactional boundary — DB write fails after Stripe call — @Transactional rolls back the DB write but not the Stripe charge — SmallRye nacks the message — broker redelivers — consumer re-invoked — UUID.randomUUID() at method entry generates UUID_B — ch_B alongside committed ch_A; developer misconception: “@Transactional on the consumer makes the whole processing atomic — rollback means the retry starts clean.” Mutiny Uni<Void> chain deferred UUID: UUID.randomUUID() inside chain() callback is deferred to subscription time — SmallRye nacks failed Uni — consumer method called again — new Uni subscribed — chain() callback re-executes — UUID_B; developer misconception: “UUID is generated when the consumer method is called, not when SmallRye subscribes to the Uni.” @Outgoing Kafka emitter inside @Transactional: Kafka producer not enrolled in JTA/JPA transaction — message committed to broker even when application @Transactional rolls back — consumer receives the phantom message — UUID generated — Stripe call ch_A — application retries and re-emits — consumer invoked again — UUID_B — ch_B; developer misconception: “@Transactional ensures the Kafka message is only sent if the DB write commits.” Content-hash key derivation, DB idempotency guard, Kafka Transactions extension, and Transactional Outbox pattern fixes. @QuarkusTest + SmallRye in-memory channel test patterns for all three modes. Comparison table (root cause, trigger, developer misconception, detection signal, fix).
Three Spring Boot Testcontainers Stripe idempotency testing blind spots. @Transactional test + @Transactional(REQUIRED) service: the test’s outer transaction is shared with the service call — UUID is generated once and Stripe is called once — test passes — but in production two concurrent HTTP requests spawn two independent transactions each generating UUID_A and UUID_B — two Stripe charges; developer misconception: “@Transactional test isolation means my service is isolated too.” WireMock stub default URL matching: stubFor(post(urlEqualTo(“/v1/payment_intents”))) matches URL and method only — does not inspect Idempotency-Key header — @Retryable re-executes method body on retry — UUID_B sent to WireMock — stub returns 200 — test passes — WireMock request journal contains UUID_A on attempt 1 and UUID_B on attempt 2 but test never calls findAll() to inspect the journal — in production Stripe receives both keys and creates ch_A and ch_B; developer misconception: “WireMock validates that my Stripe call is correct — if the test passes, the idempotency key is stable.” @TransactionalEventListener(AFTER_COMMIT) + @Transactional test: listener fires only after the publishing transaction commits — @Transactional test rolls back — listener never fires — test asserts event was published (it was) but never validates Stripe was called or that the key is stable — UUID generated inside the listener on each invocation is never tested; developer misconception: “the event is published, the listener handles the rest.” Fix for Mode 1: concurrent test with two threads and CountDownLatch assert exactly one charge. Fix for Mode 2: WireMock request journal assertion assertThat(keyAttempt1).isEqualTo(keyAttempt2). Fix for Mode 3: @Commit on test method with explicit @BeforeEach cleanup; listener re-invocation idempotency test; idempotency key carried in event payload, not re-generated in listener. Comparison table (3 blind spots × why test passes / production failure / missing assertion). Checklist of five required idempotency assertions for any Testcontainers Stripe test.
Three Spring Data MongoDB Stripe idempotency failure modes. @Retryable + @Transactional(transactionManager = “reactiveMongoTransactionManager”): Spring Retry’s outer AOP proxy (order MAX-5) calls MethodInvocation.proceed() which re-invokes the @Transactional interceptor — ReactiveMongoTransactionManager opens a new reactive ClientSession — the method body re-executes from line 1 — UUID.randomUUID() at method entry generates UUID_B — ch_B alongside committed ch_A; MongoDB-specific nuance: the BillingAttempt record for UUID_A is rolled back (replica set oplog), leaving only the UUID_B record in MongoDB while Stripe shows both ch_A and ch_B — the audit gap between systems is the detection signal; developer misconception: “@Transactional with ReactiveMongoTransactionManager owns the reactive MongoDB session lifecycle — @Retryable retries the Stripe HTTP call; UUID at method entry is outside the retry scope.” ReactiveMongoTemplate.inTransaction() + Mono.defer{} UUID inside callback + retryWhen() on outer Mono: retryWhen() re-subscribes inTransaction()’s Mono — inTransaction() re-invokes the callback Function<ReactiveMongoOperations, Publisher<T>> to get a new Publisher — callback body re-executes — Mono.defer{} factory re-evaluates — UUID_B — ch_B; fix: UUID as final local variable in enclosing method, outside inTransaction() call, captured by the callback lambda; same pattern for Mono.fromCallable{} inside the callback; developer misconception: “inTransaction() executes the callback once and commits a MongoDB multi-document transaction — retryWhen() retries the Stripe call after the transaction; the callback is done and UUID is stable.” ReactiveMongoRepository.findAll() cold Flux<Customer> + outer retryWhen() in batch billing: findAll() backed by MongoDB reactive driver cursor — each new subscription opens a new cursor and re-queries from collection start — outer retryWhen() re-subscribes on any element failure — all customers re-emitted — UUID inside flatMap regenerates — blast radius = entire batch; fix: move retryWhen() inside flatMap lambda for per-element retry scope — content-hash UUID captured as final variable before per-element retry scope; checkpoint-based resumable batch pattern using MongoDB upsert for idempotent writes; developer misconception: “retryWhen() retries from the failed element — customers already charged are not re-emitted from findAll() because MongoDB already returned their documents.” Comparison table (MongoDB vs. JPA reactive failure modes), detection signals per mode, and JUnit 5 + WireMock + Testcontainers MongoDB test patterns for all three modes asserting Idempotency-Key header stability across retry attempts and per-element retry scope.
Three Quarkus Hibernate Reactive Panache Stripe idempotency failure modes. MicroProfile @Retry + @Transactional CDI method: SmallRye FT @Retry interceptor is the outer CDI interceptor — on a Stripe transient failure, SmallRye FT calls InvocationContext.proceed() which re-invokes the @Transactional interceptor (opens new transaction) and re-executes the full method body — UUID.randomUUID() at method entry generates UUID_B — ch_B alongside committed ch_A; developer misconception: “@Retry retries the Stripe HTTP call — @Transactional manages the Hibernate Reactive session lifecycle; UUID is generated once at the start of the logical operation.” Mutiny Uni.createFrom().item(Supplier<T>) lazy supplier inside onFailure().retry().atMost(): the Supplier overload defers UUID evaluation to subscription time — retry() re-subscribes the upstream Uni from the source — the supplier executes again — UUID_B — ch_B; developer misconception: “retry() re-subscribes from the failing flatMap, not from the item(Supplier) source at the chain head.” Panache Entity.streamAll() returning cold Multi<Entity> + outer onFailure().retry(): streamAll() is backed by a Hibernate Reactive scroll cursor — each new subscription opens a new cursor from the beginning of the result set — outer retry re-subscribes from the source — all customers re-streamed — UUID generated inside flatMap re-generated for every customer — blast radius = entire batch; developer misconception: “retry() on a Multi retries the failed element, not the entire stream from the beginning.” Content-hash key patterns, per-element retry inside flatMap, CDI self-injection for @Retry + @Transactional bypass, and @QuarkusTest + WireMock test patterns for all three modes.
Three Project Reactor and Kotlin Flow Stripe idempotency failure modes compared side by side. Reactor Mono.defer{} inside retryWhen(): defer{} defers evaluation to subscription time — retryWhen() re-subscribes the entire upstream chain — defer{} factory re-executes — UUID.randomUUID() generates UUID_B — ch_B alongside committed ch_A; same problem with Mono.fromCallable{}; developer misconception: “Mono.defer{} lazily wraps UUID — retryWhen() retries from the failed operator, not the deferred source that had already evaluated.” Kotlin Flow cold flow{} builder + outer retryWhen{} in batch billing: flow{} is cold — outer retryWhen{} causes re-collection from the beginning of the builder block — UUID.randomUUID() at emit() regenerates for every customer — blast radius = entire batch; fix: pre-compute content-hash keys before the flow builder and move retryWhen{} inside the flatMapMerge lambda for per-element retry scope; same problem with channelFlow{}. TransactionalOperator (Reactor) vs. @Transactional suspend fun (Kotlin) with outer @Retryable: proceed() re-subscribes the transactional Mono (Reactor) or creates a new coroutine invocation (Kotlin) — UUID re-generates in both; developer misconception for Reactor: “TransactionalOperator.transactional() is a committed unit — retry enters from after the transaction”; for Kotlin: “Spring’s coroutine @Transactional support means @Retryable is coroutine-aware.” Fix for Reactor: use retryWhen() as a Reactor operator; fix for Kotlin: separate UUID into non-@Retryable facade, pass as stable parameter. Mono.just() vs. Mono.defer() vs. Mono.fromCallable() subscription semantics, List.asFlow() safe pattern, and StepVerifier + WireMock test patterns for all three modes.
Three Spring Data JPA + Stripe idempotency failure modes. @Retryable + @Transactional on a JPA service method: Spring Retry’s proxy order places @Retryable (order MAX-5) as the outer AOP proxy — proceed() re-invokes through @Transactional which opens a freshEntityManager AND new transaction — UUID.randomUUID() at method entry generates UUID_B on retry — ch_B alongside committed ch_A; JPA-specific nuance: the EntityManager lifecycle is tied to the @Transactional proxy — on rollback, EM1 closes and entity references become detached; developer misconception: “@Transactional is the outer wrapper that owns the JPA session lifecycle — @Retryable retries within the existing EntityManager context, not UUID generation at method entry.” @Version optimistic locking retry loop: UUID inside while loop — Stripe charged before save() — ObjectOptimisticLockingFailureException marks transaction rollback-only — developer catches exception and loops — UUID_B on next iteration — ch_B alongside committed ch_A; rollback-only complication: catching OLE inside the method avoids UnexpectedRollbackException, but developer typically sees that exception and misses the Stripe duplicate entirely; developer misconception: “ObjectOptimisticLockingFailureException is a DB version conflict — Stripe succeeded before the DB exception — new UUID on retry is fine because starting from scratch.” @TransactionalEventListener(AFTER_COMMIT) + @Retryable: UUID computed inside the listener — @Retryable’s proceed() re-invokes the listener method from line 1 — UUID_B — ch_B; AFTER_COMMIT semantics do not create any UUID isolation boundary — they control when the listener fires, not what proceed() re-invokes; developer misconception: “AFTER_COMMIT fires the listener after the outer transaction commits — @Retryable retries only the Stripe HTTP call, not UUID generation at handler entry.” Content-hash key patterns, DB-write-before-Stripe-call ordering, UUID in event payload, and JUnit 5 + WireMock test patterns for all three modes.
Three Spring JdbcTemplate + Stripe idempotency failure modes. @Transactional + @Retryable on the same method: Spring Retry’s default proxy order places @Retryable (order MAX-5) as the outer AOP proxy and @Transactional (order MAX) as the inner — proceed() re-invokes through @Transactional (new transaction) and then re-executes the method body from line 1 — UUID.randomUUID() at method entry generates UUID_B on retry — ch_B alongside committed ch_A; developer misconception: “@Transactional is the outer wrapper that manages the transaction lifecycle — @Retryable retries the transaction, not UUID generation at method entry.” TransactionTemplate.execute() inside @Retryable: developer switches to programmatic transaction management to fix AOP ordering ambiguity — places UUID.randomUUID() before transactionTemplate.execute() thinking it is outside the retry scope — @Retryable.proceed() re-invokes the method including the UUID line before transactionTemplate.execute() — UUID_B — ch_B; developer misconception: “@Retryable retries the transactionTemplate.execute() call that threw, not my UUID computation that comes before it.” jdbcTemplate.batchUpdate() + @Retryable(DeadlockLoserDataAccessException.class): batch args built in a .stream().map() step that generates UUID and calls Stripe per customer — batchUpdate() deadlocks — proceed() re-runs the entire method including the map step — UUID_B per customer — ch_B for all customers whose Stripe charges completed in attempt 1’s map phase; blast radius = entire batch; developer misconception: “batchUpdate() is the failing DB write step — the map/Stripe phase is already done and is not retried.” Phase-separation fix, content-hash key patterns, BatchPreparedStatementSetter alternative, and JUnit 5 + WireMock test patterns for all three modes.
Three jOOQ + Spring Boot + Stripe idempotency failure modes. @Retryable on a method containing dsl.transactionResult(): Spring Retry’s AOP proceed() re-invokes the entire method body including UUID.randomUUID() at method entry — UUID_B on retry — ch_B alongside committed ch_A; developer misconception: “UUID is outside the jOOQ transaction lambda so it’s stable across retries.” jOOQ optimistic locking retry loop: UUID inside the while loop — Stripe charged inside dsl.transaction{} before the version-check UPDATE — DataChangedException rolls back the JDBC transaction but not the already-completed Stripe HTTP call — UUID_B on next iteration — ch_B; developer misconception: “dsl.transaction{} is ACID — a rollback undoes the Stripe charge inside the lambda too.” @Retryable(DeadlockLoserDataAccessException) on a batch billing method: batchInsert() throws on deadlock — proceed() re-runs the entire method including the .stream().map() step that calls Stripe per customer — UUID_B per customer — ch_B for all customers charged in attempt 1’s map step; developer misconception: “batchInsert() is the failing step — the map/Stripe step is not retried.” Fix: content-hash keys pre-computed outside every retried boundary; Stripe charging moved before batchInsert() into a non-@Retryable caller; @Retryable applied only to the DB-write method receiving records as a parameter. WireMock + JUnit 5 test patterns for all three modes.
Three Spring Data R2DBC + Kotlin Coroutines + Stripe idempotency failure modes. @Transactional suspend fun + @Retryable: Spring Retry’s AOP proceed() creates a new Kotlin coroutine and a new reactive context on each retry — the R2DBC transaction manager obtains a fresh connection — UUID.randomUUID() inside the suspend fun generates UUID_B — ch_B alongside committed ch_A; developer misconception: “@Retryable might be coroutine-aware like @Transactional in Spring — the retry re-enters the existing reactive context.” Flux.flatMap { val key = UUID.randomUUID() }.retryWhen() outside flatMap: retryWhen() re-subscribes the entire upstream Flux including all flatMap bodies — UUID_B per customer — ch_B for every customer processed in the first subscription; developer misconception: “retryWhen() retries only the failed element, not the entire Flux.” TransactionalOperator.transactional(Mono.fromCallable { UUID.randomUUID() }.flatMap { save(key).then(charge(key)) }).retryWhen(): retryWhen() after transactional() re-subscribes the transactional Mono — new transaction — UUID.randomUUID() re-executes in the new subscription — UUID_B — ch_B; developer misconception: “retryWhen() retries from after the committed transactional() block — the DB save with UUID_A is stable.” Mono.fromCallable vs. Mono.just for key generation, per-element retryWhen() placement inside flatMap, Kotlin Flow + R2DBC retry patterns, and StepVerifier + WireMock test patterns for all three modes.
Three Spring Boot 3 virtual threads and Stripe idempotency failure modes. @Retryable on a virtual-thread-backed service (spring.threads.virtual.enabled=true) re-invokes the method body on each retry attempt via AOP proceed() — UUID inside the method generates UUID_B — ch_B alongside committed ch_A; developer misconception: “Virtual threads use continuations like Kotlin coroutines — retry might resume the original continuation with preserved state.” Executors.newVirtualThreadPerTaskExecutor() + manual while loop with Thread.sleep() backoff — UUID inside the loop body generates UUID_B on each iteration — ch_B per already-charged customer; developer misconception: “New UUID on retry is safer because we don’t know if ch_A committed — Stripe returns the same result for a committed charge.” StructuredTaskScope.ShutdownOnFailure + outer retry loop — scope shutdown cancels running forks but not completed forks — outer retry creates a new scope with new fork tasks for all customers — UUIDs regenerate — ch_B per customer whose fork committed ch_A before the failing fork triggered shutdown; developer misconception: “ShutdownOnFailure manages task lifecycles — it knows which forks completed and won’t re-run them.” Virtual thread continuation mechanics vs. AOP retry, content-hash key fixes, and JUnit 5 + WireMock test patterns for all three modes.
Three Spring Batch + Stripe idempotency failure modes. FaultTolerantStep retry calls ItemWriter.write() again with the same chunk — UUID.randomUUID() inside write() generates UUID_B on the retry invocation — ch_B alongside committed ch_A; developer misconception: “Spring Batch retries the chunk, not write() — same invocation re-attempted.” @Scheduled + jobLauncher.run(job, new JobParametersBuilder().addLong(“run.id”, currentTimeMillis())) creates a new JobInstance instead of restarting the failed execution — the new instance processes all items from scratch — customers already charged in the failed run receive ch_B; developer misconception: “Adding run.id is the standard Spring Batch restart pattern.” FaultTolerantStep skip-scan logic re-processes items through ItemProcessor one at a time to identify the skip candidate — UUID.randomUUID() computed in ItemProcessor.process() generates UUID_B on the re-processing pass — write() sends ch_B alongside committed ch_A; skip-scan is distinct from retry: retry does not re-invoke the processor, skip-scan does. Content-hash key placement rules for Spring Batch, JobOperator.restart() vs. new JobInstance patterns, Spring Batch metadata table queries to detect which mode fired, and Spring Batch Test + WireMock test patterns for all three modes.
Three Kotlin Flow and Exposed Stripe idempotency failure modes. Exposed’s transaction{} DSL retries the lambda body automatically on database serialization failures (SQLState 40001 — deadlock detected) via DEFAULT_REPETITION_ATTEMPTS — UUID inside the block generates UUID_B on the silent database-layer retry — ch_B; developer misconception: “I’m not retrying Stripe — the database is retrying a concurrency conflict.” Flow.retry{} re-collects a cold flow{} builder on failure — flow{} is a cold stream — re-collection re-executes the builder block — UUID inside the builder generates UUID_B — ch_B; Exposed transaction rolled back for attempt 1 (DB record undone) while ch_A is already committed in Stripe — two charges, one DB record. Flow.flatMapLatest{} cancels the previous inner flow when the upstream emits a new value — the Stripe HTTP request is already in-flight — Stripe commits ch_A — the coroutine is cancelled before receiving the response — Exposed transaction rolls back — next emission starts a new inner flow with UUID_B — ch_B — two Stripe charges, zero DB records; cancellation of a coroutine does not retract an HTTP request already delivered to Stripe. Content-hash key fixes, flatMapMerge vs. flatMapLatest guidance, and kotlinx.coroutines.test + WireMock test patterns for all three modes.
Three Micronaut Kotlin coroutine Stripe idempotency failure modes. Micronaut’s compile-time KSP-generated @Retryable interceptor calls proceed() per retry on a suspend function — each proceed() is a fresh coroutine invocation with a new activation frame — UUID.randomUUID() at the top of the function body regenerates — UUID_B — ch_B; compile-time vs. runtime AOP distinction doesn’t change the runtime re-invocation behavior; @Recover fires too late — N−1 duplicate charges already exist by the time it’s called. Manual while (attempt < maxAttempts) retry inside a @TransactionalEventListener: UUID.randomUUID() inside the loop body is a new Kotlin val binding per iteration — UUID_B on iteration 2 — ch_B; the @TransactionalEventListener rollback-only complication: transaction rolls back the DB record but cannot roll back committed Stripe charges — customer charged once or twice, DB shows nothing. supervisorScope{} + async{} parallel batch billing with outer @Retryable: proceed() re-invokes the batch method body — a new supervisorScope{} is created — all async{} blocks recreated for every customer — UUIDs regenerate for already-charged customers — ch_B per customer — blast radius proportional to batch progress at failure time; fix: per-customer @Retryable on a separate bean method, content-hash key passed as parameter. All three with Micronaut Kotlin code, interceptor ordering analysis, and @MicronautTest + WireMock test patterns.
Three Spring WebFlux functional endpoint (RouterFunction/HandlerFunction) Stripe retry failure modes. request.bodyToMono().cache() fixes the body-consumed-once problem but the outer retryWhen() re-invokes the flatMap lambda — UUID.randomUUID() inside the lambda generates UUID_B — ch_B alongside committed ch_A. WebClient .headers{ } customizer lambda executes on every HTTP request assembly per subscription — retryWhen() re-subscribes — WebClient reassembles the request — UUID inside the customizer generates UUID_B — ch_B; subtler: UUID moved outside .headers{} but inside the helper method — helper called from a flatMap lambda subject to outer retryWhen() — helper re-invoked per lambda re-execution — UUID_B. Mono.fromCallable() inside a per-item batch flatMap re-executes its callable on every subscription — per-item retryWhen() re-subscribes — UUID inside the callable generates UUID_B — ch_B; outer Flux.retryWhen() variant restarts the entire batch on any failure — all already-charged customers’ flatMap lambdas re-execute — blast radius scales with already-processed count. Content-hash key fixes and WebTestClient + WireMock test patterns for all three modes.
Three Ktor-specific Stripe retry failure modes. A suspend retry HOF calls its block lambda again on each attempt — UUID.randomUUID() inside the block generates UUID_B — ch_B. Exposed newSuspendedTransaction{} with a while-loop retry: UUID moved “above the transaction block” but still inside the loop body — Kotlin val inside a while-loop body is a new binding per iteration — UUID regenerates — UUID_B — ch_B. Kotlin Flow.retryWhen{} batch billing in a Ktor route handler with flatMapMerge: flow{} builder lambda re-executes on re-collection — UUID inside the builder generates UUID_B — ch_B — blast radius scales with the number of customers whose first attempt fails. All three with Ktor HttpClient code, content-hash key fixes, and WireMock test patterns asserting Idempotency-Key equality across retry attempts.
Three Kotlin coroutine retry failure modes that generate new Stripe idempotency keys. Arrow Resilience retry{Schedule} calls its block lambda again on each retry attempt — every statement inside the block re-executes including UUID.randomUUID() — UUID_B — ch_B. Spring @Retryable on a Kotlin suspend function calls proceed() again per retry — a fresh method invocation with a new activation frame — all local variables including UUID at the very top of the function scope reinitialize per proceed() call — UUID_B — ch_B — plus interceptor ordering hazard: if @Transactional (outer) wraps @Retryable (inner), a DB failure marks the coroutine transaction rollback-only and all subsequent retry Stripe charges commit without a DB record; fix: service decomposition. Kotlin Flow.retryWhen{} re-collects the upstream Flow on matching exceptions — Kotlin’s cold flow{} builder re-executes its lambda on every collection — UUID.randomUUID() inside the builder generates UUID_B — ch_B; developer misconception: “Kotlin Flows don’t have subscriptions like Reactor” — retryWhen{} internally re-collects (equivalent to re-subscribing in Reactor terms). All three with Kotlin code, content-hash key fixes, nested-retry nesting trap analysis, and WireMock test patterns capturing Idempotency-Key headers across retry attempts.
Three Spring Boot CompletableFuture batch billing failure modes that generate new Stripe idempotency keys on retry. RetryTemplate.execute(RetryCallback) re-invokes the callback lambda body on each retry — UUID.randomUUID() inside the callback body (outside supplyAsync()) regenerates per RetryTemplate attempt — UUID_B — ch_B. CompletableFuture.allOf() batch billing with UUID inside each per-customer supplyAsync() supplier — @Retryable batch method re-invokes and recreates all N CompletableFutures — all supplier UUIDs regenerate — blast radius: N−1 duplicate charges for a batch of N. @Transactional + @Retryable batch method where developer moved UUID to method body scope outside all lambdas — @Retryable AOP proceed() reinitializes all local variables including UUID results — UUID_B — ch_B — the fix requires content-hash keys, not just placement outside lambdas.
Three Micronaut ReactorHttpClient retry failure modes that generate new Stripe idempotency keys on retry. Mono.create() is Reactor’s publisher-builder API for wrapping callback-based async code such as Stripe’s Charge.createAsync() — the MonoSink consumer executes on every subscription — retryWhen() re-subscribes — UUID.randomUUID() inside the consumer generates UUID_B — ch_B alongside committed ch_A. Flux.fromIterable(customerIds).flatMap() batch billing with UUID inside the flatMap() mapper and Flux.retryWhen() on the outer Flux — a transient failure on any customer triggers a full batch restart — Flux.fromIterable() re-emits all customers from the beginning — the mapper re-executes for every already-charged customer — all UUIDs regenerated — all customers charged twice; fix: move retry inside the flatMap() mapper as per-item retry with stable content-hash keys. Mono.defer() deferred request construction inside retryWhen(Retry.fixedDelay().filter(predicate)) — retryWhen() re-subscribes to the outer Mono.defer() on filter-matching failures — the defer factory re-evaluates UUID.randomUUID() per re-subscription — the custom filter predicate limits which exceptions trigger retry but does not prevent UUID regeneration when retry does fire — UUID_B — ch_B. All three with Java code, content-hash key fixes, per-item retry restructuring for batch billing, and @MicronautTest + WireMock + StepVerifier test patterns.
Three Quarkus reactive @Transactional failure modes that generate new Stripe idempotency keys on retry. Panache.withTransaction() creates a cold, deferred Uni that executes its supplier lambda on each subscription — when the caller chains onFailure().retry() on the returned Uni, each retry re-subscribes and re-executes the lambda — UUID.randomUUID() inside the lambda generates UUID_B — ch_B alongside committed ch_A. SmallRye FT @Retry on a @ReactiveTransactional CDI method re-invokes the method body per retry attempt via AOP proceed() — UUID at method scope regenerates — UUID_B — ch_B — plus interceptor ordering hazard: @ReactiveTransactional (outer, priority 200) wraps @Retry (inner, priority ~2000) — a DB failure in attempt 1 marks the reactive transaction rollback-only — all subsequent @Retry attempts issue new Stripe charges that commit in Stripe but cannot be recorded in the poisoned transaction boundary. UUID.randomUUID() inside a flatMap() mapper in a @ReactiveTransactional method with onFailure().retry() chained inline — each retry re-subscribes to the upstream including the flatMap() mapper — mapper re-executes per re-subscription — UUID_B — ch_B — plus the same rollback-only poisoning when inline retry shares the outer transaction boundary. All three with Quarkus Panache Java code, content-hash fixes, service-decomposition patterns for correct interceptor ordering, and @QuarkusTest + WireMock test patterns capturing all Idempotency-Key headers across retry attempts.
When Spring Boot uses @Async to offload Stripe calls to a thread pool and adds retry for resilience, three structurally distinct mechanisms each silently generate a new idempotency key on every retry attempt. A caller retry loop calls the @Async service method again per attempt — each call creates a new CompletableFuture.supplyAsync() task — the supplier lambda re-evaluates — UUID.randomUUID() inside the lambda generates UUID_B — ch_B alongside committed ch_A. @Retryable on an outer @Transactional service that calls an @Async inner service — @Retryable’s proceed() re-invokes the outer method body per attempt — outer method calls the @Async service again — each @Async dispatch is a new method body execution in the async thread pool — UUID at @Async method scope regenerates — UUID_B — ch_B; @Async also breaks @Transactional propagation — new thread = no inherited ThreadLocal transaction — inner Stripe call not covered by parent transaction rollback. exceptionallyCompose() recovery function calls the @Async service again — each recovery invocation is a new method body execution — UUID regenerates — UUID_B — ch_B — plus outer @Transactional rollback-only complication when CompletableFuture.get()TimeoutException corrupts the outer transaction boundary before the retry future completes. All three with Java code, content-hash key fixes, and WireMock integration test patterns capturing all Idempotency-Key headers across retry attempts.
Three Micronaut Data reactive @Transactional failure modes that generate new Stripe idempotency keys on retry. UUID.randomUUID() inside Mono.fromCallable() callable re-executes per Reactor retryWhen() re-subscription in a Micronaut reactive @Transactional service — Micronaut’s ReactorReactiveTransactionOperations binds the transaction to the Reactor subscriber context — each retryWhen() re-subscription creates a new subscriber context — fromCallable() calls its callable per subscription — UUID_B — ch_B. Micronaut @Retryable on a method returning Mono<T> — DefaultRetryInterceptor calls context.proceed() per retry attempt — proceed() re-invokes the entire method body (not re-subscribes) — UUID.randomUUID() at method scope before any reactive operator still regenerates — UUID_B — ch_B. Manual onErrorResume() retry with UUID inside Mono.defer() recovery factory — each onErrorResume() invocation evaluates the defer factory — UUID_B — ch_B — plus Micronaut reactive @Transactional(REQUIRED) rollback-only complication in the recovery path. All three with Micronaut-specific context, Java code, content-hash fixes, and @MicronautTest + WireMock + StepVerifier test patterns.
When a Spring Boot service uses WebClient to call Stripe and adds retry for resilience, three structurally distinct mechanisms each silently generate a new idempotency key. UUID.randomUUID() inside Mono.fromCallable() re-executes on every Reactor retryWhen() re-subscription — fromCallable() calls its callable per subscription — UUID_B — ch_B alongside committed ch_A. @Retryable wrapping a @Transactional service method that calls WebClient...block() — @Retryable (outer, order 2,147,483,642) re-invokes the @Transactional (inner) method body — UUID in method body generates UUID_B — ch_B — plus .block() in a reactive event-loop thread throws IllegalStateException after Stripe may have committed ch_A. Spring Data R2DBC reactive @Transactional with WebClient + retryWhen() — UUID inside Mono.defer() or flatMap() mapper — each retryWhen() re-subscription starts a new reactive transaction (correct for DB atomicity) but also re-evaluates the per-subscription operator — UUID_B — ch_B. All three with Java code, content-hash fixes, and WireMock + StepVerifier test patterns.
When a Spring Boot service is annotated with both @Retryable and @Transactional and calls a Feign client for Stripe, three structurally distinct retry mechanisms silently generate a new idempotency key. @Retryable (order 2,147,483,642, outer) re-invokes the @Transactional (inner) method body after each rollback — UUID.randomUUID() in the method body generates UUID_B — Feign (configured with Retryer.NEVER_RETRY to avoid compounding) calls Stripe with UUID_B — ch_B. A @Transactional(propagation = REQUIRES_NEW) inner service called from a try-catch retry loop — each REQUIRES_NEW call is an independent method invocation — UUID inside inner method body generates UUID_B on every retry call — ch_B. RetryTemplate.execute(callback) calls callback.doWithRetry() on each retry attempt — UUID.randomUUID() inside the lambda body executes per invocation — UUID_B — ch_B — the @Transactional proxy stays open across all RetryTemplate retries, accumulating audit writes in one transaction. Three @Transactional + Feign failure modes with Java code, content-hash key fixes, and WireMock integration tests.
Vert.x’s event-loop model requires blocking Stripe Java SDK calls to run inside vertx.executeBlocking() — UUID.randomUUID() inside the Callable body regenerates on every new executeBlocking() invocation — Future.recover() retry calls the method again — new Callable — UUID_B — ch_B alongside committed ch_A. UUID.randomUUID() inside a .compose() mapper function regenerates when Future.recover() retries the outer method, because retry rebuilds the entire pipeline from the first step and re-invokes every mapper — the developer reasoning (“the UUID is scoped to the charge step”) is correct within a single execution but fails under retry because recover() rebuilds from root, not from the step that failed. vertx.setTimer() is the Vert.x-idiomatic way to add backoff between retries without blocking the event loop — but the timer callback calls the billing method again — UUID.randomUUID() inside the method body generates UUID_B — ch_B — key fact: Vert.x Future is eager (hot), not cold like RxJava Single or Mutiny Uni — every Vert.x retry is unconditionally a new method invocation or a new executeBlocking() task — UUID must be computed outside the retry boundary in all three patterns. Three Vert.x native Future failure modes with code and fixes.
Spring AOP places @Retryable (order 2,147,483,642) as the outer interceptor and @Transactional (order 2,147,483,647) as inner by default — each @Retryable retry calls the method body again, opening a fresh @Transactional boundary — UUID.randomUUID() at method entry regenerates per attempt — if Stripe committed ch_A on attempt 1 before a transient StripeException, the retry carries UUID_B and Stripe commits ch_B alongside ch_A. A @Scheduled batch job that catches exceptions from @Transactional(REQUIRES_NEW) per-customer billing calls and retries failed customers re-invokes the billing method entirely — REQUIRES_NEW isolation guarantees each call is a fresh method invocation — UUID.randomUUID() inside the billing method regenerates on every retry call — UUID_B — ch_B — fix: compute the idempotency key before the first call and pass the same value as a parameter to both the initial call and all retries. @Retryable(include = OptimisticLockException.class) on a @Transactional billing method retries the Stripe call along with the @Version-checked DB write — the developer’s intent was to retry the DB write — the AOP interceptor retries the entire method body including the Stripe call that precedes the DB write — ch_A already committed before OptimisticLockException thrown by the DB flush — UUID_B on retry — ch_B — fix: split the Stripe call and the DB write into separate method boundaries with independent retry scopes. Three Spring @Transactional + @Scheduled failure modes with code and fixes.
RxJava3’s Single.fromCallable(callable) invokes the Callable on every new subscription — Single.retry(N) re-subscribes on failure — if UUID.randomUUID() lives inside the callable, each retry subscription generates a fresh UUID — if Stripe committed ch_A on attempt 1 before a transient failure, the retry arrives with UUID_B and Stripe commits ch_B alongside ch_A. UUID inside a flatMap(mapper) with an outer retry() causes the mapper to run again on re-subscription — UUID_B — ch_B — the fix is to scope the retry inside the mapper body, wrapping only the innermost Stripe call. Micronaut @Retryable on a method returning Single<T> re-invokes the method body on each attempt, so UUID.randomUUID() at method start — even outside the fromCallable() lambda — regenerates per @Retryable attempt — UUID_B — ch_B — the fix is to derive the key from stable method parameters or pass it as a parameter from the caller, outside the @Retryable boundary. Three Micronaut RxJava3 failure modes with code and fixes.
Quarkus Mutiny’s Uni is a cold, lazy reactive type: the upstream pipeline is re-evaluated on every new subscription. Uni.onFailure().retry() re-subscribes to the upstream Uni on each retry attempt — if UUID.randomUUID() lives inside a Uni.createFrom().completionStage(supplier) factory, a Uni.createFrom().item(supplier) factory, or a transformToUni(mapper) operator, each retry re-subscription evaluates that code again and generates a fresh UUID — if Stripe committed ch_A on attempt 1 before a transient network error, the retry arrives with UUID_B and Stripe commits ch_B alongside ch_A. MicroProfile Fault Tolerance @Retry on a method returning Uni<T> re-invokes the method body itself on each attempt, so UUID.randomUUID() at method start also regenerates per @Retry attempt — UUID_B — ch_B. Three failure modes with working code and fixes: compute the idempotency key from immutable business data before the Uni pipeline, scope the retry to the innermost Uni that wraps only the Stripe call, and pass a pre-generated key as a method parameter to @Retry-annotated methods rather than generating it in the method body.
Spring’s @Async and @Scheduled annotations interact with Stripe’s idempotency system in ways that produce three distinct billing failure modes. First: a @Scheduled billing job dispatches customer charges as @Async tasks; a StripeException thrown inside the thread pool is captured in the CompletableFuture’s exceptional state — the @Scheduled orchestrator already returned after the dispatch and never observes the exception; the developer adds a manual for-loop inside the @Async method body, calling UUID.randomUUID() per loop iteration — if Stripe committed ch_A on attempt 1 before the timeout, attempt 2 carries UUID_B — ch_B alongside ch_A; fix: compute a content-hash idempotency key once before the retry loop and use the same key on all attempts. Second: @Retryable and @Async placed on the same method — @Retryable’s RetryOperationsInterceptor (order 2,147,483,642) is the outer interceptor; @Async’s AsyncExecutionInterceptor (order 2,147,483,647) is the inner interceptor; when the proxy method is called, @Retryable fires, calls through to @Async, which dispatches the billing work to a thread pool and immediately returns a CompletableFuture to @Retryable — the proxy method returned without throwing, so @Retryable considers the call successful and exits; StripeException happens asynchronously in the thread pool and is never seen by @Retryable; the developer adds a .exceptionally() callback with a new UUID.randomUUID() — UUID_B — ch_B; fix: place @Retryable on a separate synchronous @Service bean; call it from within the @Async method body via cross-bean injection so the retry proxy is traversed inside the async thread. Third: CompletableFuture retry chain — the @Scheduled orchestrator chains .exceptionally() to retry a failed billing task by calling the @Async billing method again; the retry lambda calls chargeCustomerAsync() which generates a new UUID.randomUUID() inside the method body per invocation — UUID_B from the retry call — ch_B alongside committed ch_A; fix: generate the idempotency key from immutable business data before the CompletableFuture chain; pass the pre-computed key as a parameter to every invocation in the chain; all calls — initial and retry — carry the same key; Stripe deduplicates the retry.
Spring’s @EventListener and @Retryable annotations interact in ways that produce three distinct Stripe billing failure modes. First: UUID.randomUUID() placed inside the body of an @EventListener + @Retryable listener method re-evaluates when @Retryable re-invokes the listener from scratch on a transient StripeException — the same BillingRequestedEvent object is passed as the parameter on every retry attempt, but UUID.randomUUID() is in the method body, not read from the event, so it regenerates on every attempt — UUID_B — ch_B alongside already-committed ch_A; the developer’s mental model that “the listener handles one event so the UUID is generated once per event” is correct when no exception occurs, but wrong under @Retryable because @Retryable re-executes the entire method body including the UUID line on each attempt; fix: compute the idempotency key in the event publisher before publishEvent() and store it as a field in the immutable event object — all @Retryable retries read the same value from the same event object; prefer content-hash keys over UUID.randomUUID() even in the event constructor for crash-safety across JVM restarts. Second: @TransactionalEventListener(phase = AFTER_COMMIT) + @Retryable — @TransactionalEventListener is implemented by TransactionalApplicationListenerMethodAdapter, which registers a TransactionSynchronization with TransactionSynchronizationManager; when the publishing transaction commits, Spring calls TransactionSynchronization.afterCommit(), which invokes the listener method via Method.invoke(targetBean, event) — Java reflection directly on the raw bean target, not through the Spring AOP proxy — @Retryable’s RetryOperationsInterceptor is not in the call stack and is silently bypassed; StripeException propagates to the TransactionSynchronizationManager framework, which logs it at DEBUG level and continues — no re-attempt, no caller notification; the developer adds a manual retry loop inside the @TransactionalEventListener method with a new UUID.randomUUID() per loop iteration — UUID_B — ch_B if ch_A was committed on attempt 1; fix: extract the Stripe charge call to a separate @Service bean annotated with @Retryable; the cross-bean call from the @TransactionalEventListener method to the service bean traverses the proxy; @Retryable fires on StripeException; generate the idempotency key from event business data before the service call. Third: retry-by-republish pattern — the @EventListener catches StripeException and calls applicationEventPublisher.publishEvent(new BillingRequestedEvent(...)) to retry through the event bus; the motivation is to allow all event listeners (audit, notification, metrics) to observe the retry as a separate event; the bug: BillingRequestedEvent’s constructor calls UUID.randomUUID(), so the republished event object carries UUID_B; if Stripe committed ch_A on the first attempt before the StripeException, the republished event’s listener invocation sends a charge with UUID_B — Stripe has never seen UUID_B — ch_B alongside ch_A; fix: decouple idempotency key generation from event object construction; use a content-hash key derived from immutable business data ("billing:" + customerId + ":" + planId + ":" + billingPeriod); the same key is computed from both the original and republished events because the business data is identical, allowing Stripe to deduplicate the retry against the prior committed charge.
Spring’s @Scheduled and @Retryable annotations interact in ways that produce three distinct Stripe billing failure modes. First: UUID.randomUUID() placed inside the body of a @Retryable-annotated billing service method re-evaluates when @Retryable re-invokes the method from scratch on a transient StripeException — each re-invocation builds a new UUID_B that drives a second charge ch_B alongside already-committed ch_A; the developer’s mental model that “the service method runs once per billing cycle” is correct when no exception occurs but wrong under retry because @Retryable re-invokes the entire method body, not just the Stripe call site. Second: a @Scheduled method that calls a @Retryable-annotated method on the same bean as a direct method call (this.chargeAllUsers()) bypasses the Spring AOP proxy entirely — @Retryable’s interceptor never fires — StripeException propagates unretried — when the developer discovers this and adds a manual retry loop with a new UUID.randomUUID() per iteration, the timestamp-based or random key per attempt introduces the duplicate-charge bug that a stable key would have prevented; fix: split into two beans so all calls from the @Scheduled orchestrator to the @Retryable service pass through the proxy. Third: an idempotency key stored as a lazily-initialized private String instance field in the singleton @Scheduled bean — initialized on null check inside the scheduled method body — is correct within one scheduled run (stable across @Retryable retries), but the null check is false on the second JVM-continuous execution because the singleton bean still holds UUID_A from the prior run; Stripe matches UUID_A against the prior run’s committed charge payload, finds a mismatch with the current run’s payload (different billing period, different metadata), and returns a 422 idempotency_key_in_use error on every @Retryable attempt — all charges fail without billing any subscriber; fix: use a local variable scoped to the method body, or reset the instance field to null at the start of every scheduled execution before any service calls.
Spring’s @Transactional and @Async annotations each alter method execution in ways that interact non-trivially with @Retryable when Stripe API calls require a stable idempotency key. Three distinct failure modes arise from their combination. First: UUID.randomUUID() placed inside an @Async service method body re-evaluates when the caller’s @Retryable retries and calls the @Async method again — a new task is submitted to the AsyncTaskExecutor — a new method body execution runs in the thread pool — UUID_B — ch_B; the @Transactional context worsens the picture because the caller’s @Transactional and the @Async method’s @Transactional are in separate threads and therefore separate transactions — the developer adds @Transactional believing the DB write and Stripe charge are atomic; they are not. Second: UUID.randomUUID() placed in a thenApply() lambda inside a CompletableFuture chain built inside the @Async method re-evaluates when @Retryable calls the @Async method again — a new task is submitted — a new CompletableFuture chain is constructed from the first statement — a new thenApply() lambda instance is created — the lambda body evaluates UUID.randomUUID() fresh — UUID_B; the developer’s reasoning that “thenApply() runs once per chain” is correct within a single chain execution, but the chain is rebuilt per @Retryable attempt so “once per chain” equals “once per attempt.” Third: UUID.randomUUID() generated inside a TransactionSynchronizationManager.getResource() == null guard inside an @Async@Transactional method re-evaluates because @Async always runs in a thread-pool thread where ThreadLocal state is not inherited from the caller — @Transactional starts a fresh transaction with empty TSM state on each new thread — every @Retryable attempt creates a new @Async task, which starts a new transaction, which has empty TSM — TSM.getResource(key) always returns null — UUID.randomUUID() fires on every attempt — UUID_B — ch_B. Fix for all three: compute the idempotency key outside both the @Retryable and @Async boundaries and pass it as a stable parameter; for Mode 3, persist the key in a @Transactional(REQUIRES_NEW) step that commits independently before the @Async task is submitted.
Spring’s @RequestScope is an attractive pattern for carrying a stable per-request billing idempotency key through a chain of service calls, but it introduces three distinct failure modes when combined with @Retryable and Stripe. First: the developer places UUID.randomUUID() inside a CompletableFuture.supplyAsync() lambda in a @Retryable method as a fallback when ScopeNotActiveException is caught — ForkJoinPool.commonPool() threads have no Spring request context so the proxy always throws — the catch block looks like a one-time initializer; it is not: @Retryable re-invokes the method from the top on StripeException, creating a new CompletableFuture.supplyAsync() call with a new lambda instance, and the catch block in the new lambda evaluates UUID.randomUUID() again — UUID_B on attempt 2 — ch_B alongside committed ch_A. Second: a RequestContextHolder.getRequestAttributes() null-check with a UUID.randomUUID() fallback placed inside the @Retryable method body in a background billing job (where no HTTP request context is ever active): on every @Retryable attempt, getRequestAttributes() returns null, the else-branch runs, and UUID.randomUUID() evaluates again — UUID_B; the developer expected the null check to execute once and produce a stable fallback key, but it executes once per @Retryable attempt; additionally, UUID keys are unsafe for background jobs because a scheduler crash loses the UUID and the next restart generates a new one — content-hash keys (userId + ":" + billingPeriod + ":" + amountCents) survive restarts. Third: the @RequestScopeBillingContext.getIdempotencyKey() implementation calls UUID.randomUUID() without memoizing the result in an instance field — @RequestScope guarantees one bean instance per request, not idempotent return values from non-memoized methods — @Retryable calls getIdempotencyKey() again on the same bean instance — new UUID.randomUUID() call — UUID_B; fix: final field initialized in the constructor captures the UUID once per request-scope bean instantiation — all subsequent calls return the stored value. Fix pattern for all three: generate the idempotency key once, in a scope that @Retryable cannot re-enter, and pass it as a stable parameter to the @Retryable-annotated method.
Helidon SE 4.x introduces three Stripe billing failure modes that are structurally distinct from the CDI-based failure modes in the Helidon MicroProfile post. Helidon SE has no CDI container and no SmallRye Fault Tolerance — fault tolerance is entirely programmatic via Retry.builder(), Timeout.builder(), and FaultTolerance.builder(). First: UUID.randomUUID() inside an FtSupplier lambda body regenerates on every Retry re-invocation — retry.invoke(() -> { String key = ... + UUID.randomUUID(); ... }) re-calls the lambda from the opening brace on each retry attempt — UUID_B on attempt 2 — Stripe commits ch_B alongside ch_A; the developer’s mental model that the lambda describes a single operation that retry “transparently repeats” is incorrect — the lambda is a functional interface called anew per attempt. Second: FaultTolerance.builder().addRetry(retry).addTimeout(timeout).build() — first added = outermost, so Retry is outer and Timeout is inner (per-attempt) — Helidon SE’s Timeout fires Thread.interrupt() on the executing virtual thread via a ScheduledExecutorService task — Stripe commits ch_A at second 2.9 of a 3.0-second deadline — interrupt fires before the blocking HTTP read completes — TimeoutException propagates to outer Retry — Retry re-invokes the supplier — UUID_B — ch_B; adding TimeoutException to Retry’s abortOn list prevents the retry. Third: Retry.invokeCompletionStage(Supplier<CompletionStage<T>>) monitors the returned stage and on exceptional completion re-invokes the supplier to obtain a new stage for the next attempt — UUID.randomUUID() inside the stage supplier lambda generates UUID_B on supplier re-invocation — the developer’s reasoning that “CompletionStage is not a cold publisher and retrying it is not the same as re-subscribing” is correct in isolation but misidentifies what invokeCompletionStage retries: it retries the supplier, not the stage. Fix for all three: compute a stable content-hash idempotency key outside any FtSupplier or stage supplier lambda, capture it as a final variable, and never place UUID.randomUUID() inside any lambda body passed to a Retry handler.
Spring Boot introduces three Stripe billing failure modes when @Cacheable and @Retryable appear on the same billing service method. Default AOP advisor ordering places @Retryable (order = 1, outermost) outside @Cacheable (order = Integer.MAX_VALUE, innermost) — @Cacheable does not store exceptions by default — when StripeException propagates through CacheInterceptor, no cache entry is written — @Retryable retries the full proxy chain — @Cacheable sees another MISS — method body re-runs — UUID.randomUUID() at method entry generates UUID_B — Stripe commits ch_B alongside ch_A; reversing the ordering (@Cacheable outer, @Retryable inner via @EnableCaching(order=1)) does not fix the problem — @Retryable’s internal retry loop calls the method body directly, bypassing @Cacheable on each retry invocation — UUID_B on the internal retry — @Cacheable stores ch_B as the canonical result while ch_A is orphaned in Stripe; a SpEL key expression that includes T(java.util.UUID).randomUUID() as an Elvis fallback (“use caller’s X-Request-Id if present, else generate a UUID”) re-evaluates randomUUID() on every expression evaluation — each @Retryable retry invocation generates a new cache key UUID — permanent MISS — method body re-runs with a second independent UUID.randomUUID() in the body — UUID_B — ch_B. Fix for all three: stable content-hash idempotency key computed before both interceptors and passed as a stable method parameter; for Mode 2, extract key generation into a separate @Cacheable-annotated bean called from within @Retryable; for Mode 3, generate the fallback UUID once in the controller (per HTTP request) rather than in the SpEL expression.
Helidon MP 4.x introduces three Stripe billing failure modes distinct from the Quarkus and Micronaut posts in this series. In Helidon MP (Weld CDI + Narayana JTA + SmallRye FT), the CDI @Transactional interceptor (priority 200) is the outer interceptor and SmallRye FT’s composite interceptor (priority 1000) is the inner interceptor — @Retry retries the method body within the same transaction, but UUID.randomUUID() at method entry regenerates on every retry attempt regardless of the surrounding transaction context — a Stripe network timeout on attempt 1 commits ch_A, UUID_B on retry commits ch_B; the MicroProfile FT specification section 7.2 places @Retry outer to @Timeout (each attempt is subject to @Timeout independently), so @Timeout can fire before Stripe delivers its response — Stripe committed ch_A before the client-side interrupt — @Retry retries with UUID_B — Stripe creates ch_B — Helidon 4.x virtual thread interrupt mechanics make this mode particularly sharp because blocking Stripe SDK calls are idiomatic; and @Retry(retryOn = OptimisticLockException.class) with JPA @Version retries the entire method body on version conflict — Stripe call precedes the @Version-checked merge — UUID_B per contention retry — single-threaded @HelidonTest never triggers version conflicts. Fix for all three: stable content-hash key as method parameter; for Mode 2, add abortOn = { TimeoutException.class }; for Mode 3, separate the Stripe call (once, outside @Retry) from the @Version-checked JPA update (inside @Retry with REQUIRES_NEW).
Quarkus introduces three Stripe billing failure modes distinct from Micronaut and Spring Boot: in Quarkus CDI, the JTA @Transactional interceptor (Narayana, priority 200) is the outer interceptor and SmallRye FT @Retry (priority 1000) is the inner interceptor — the opposite of Micronaut’s default — @Retry retries the method body within the same transaction, but UUID.randomUUID() at method entry regenerates per retry regardless of the surrounding transaction context — a Stripe network timeout on attempt 1 commits ch_A, UUID_B on retry commits ch_B; when a service method returns Uni<T> and is annotated with @Retry, SmallRye FT re-invokes the method body per retry to get a fresh Uni — code that appears before the Uni chain assembly, including UUID.randomUUID(), re-executes per retry invocation — Uni.createFrom().deferred() does not prevent this; and @Retry(retryOn = OptimisticLockException.class) with a Panache @Version entity retries the entire method body on DB version conflict — Stripe call precedes the @Version-checked persist() — UUID_B per contention retry — single-threaded @QuarkusTest never triggers version conflicts so the double-charge path is untested. Fix: compute a stable content-hash idempotency key outside the @Retry boundary and pass it as a method parameter; for Mode 3, separate the Stripe call (no @Retry) from the @Version-checked DB write (inner @Retry(OptimisticLockException) with REQUIRES_NEW and chargeId as a stable parameter).
Micronaut Data introduces three Stripe billing failure modes absent from the Micronaut HTTP Client transport-layer retry post: RecoveryInterceptor occupies a higher position than TransactionalInterceptor in the Micronaut 4.x AOP chain — @Retryable is the outer interceptor and @Transactional is the inner interceptor when both annotations appear on the same method — each @Retryable retry is a new invocation from TransactionalInterceptor inward, opening a fresh transaction and re-evaluating UUID.randomUUID() at method entry with UUID_B; Micronaut Data generates repository implementations at compile time and the generated save() method carries @Transactional without the interface source showing it — a @Retryable service calls Stripe (ch_A committed), then billingRepository.save() throws DataAccessException, and the @Retryable retry generates UUID_B — adding @Transactional to the service method to fix propagation semantics does not remove the UUID regeneration; and @Retryable(includes = OptimisticLockException.class) is a common Micronaut Data pattern for @Version-annotated entity contention — it retries the entire method body including the Stripe call before the @Version-checked DB write — UUID.randomUUID() at method entry regenerates with UUID_B on every contention-triggered retry — integration tests miss this because single-threaded test scenarios never trigger a version conflict. The fix for all three: compute a deterministic content-hash idempotency key outside the @Retryable boundary and pass it as a stable method parameter; for Mode 3, separate the Stripe call (once, outside @Retryable) from the @Version-checked DB write (inside an inner @Retryable method that never calls UUID.randomUUID()).
Spring WebMVC async introduces three Stripe billing failure modes absent from synchronous Spring MVC: DeferredResult<T> executor threads do not receive SecurityContext via SecurityContextCallableProcessingInterceptor, which only fires for Callable<T> return types — a developer who submits billing to a TaskExecutor callback finds SecurityContextHolder.getContext().getAuthentication() is null on the executor thread — a defensive null-check generates UUID.randomUUID() per invocation — @Retryable produces UUID_B where Stripe creates ch_B. The common fix of setting SecurityContextHolder.setStrategyName(MODE_INHERITABLETHREADLOCAL) appears to solve the gap: InheritableThreadLocal propagates from parent to child thread at creation time — but thread-pool threads (ThreadPoolTaskExecutor, Tomcat NIO connector pool) were created at application startup before any request arrived, so their InheritableThreadLocal values are permanently null — the fix does nothing for pre-existing pooled threads — UUID_A on attempt 1, UUID_B on retry; developers who test with SimpleAsyncTaskExecutor never see the failure because that executor creates a new child thread per task, which does inherit the test thread’s context. The third mode: placing @Async and @Retryable on the same billing method causes @Retryable (default order=1, outermost advisor) to see a CompletableFuture return from the inner @Async proxy — no exception at the proxy boundary — retry never fires — developer splits the annotations to separate beans — inner @Retryable service runs on the executor thread with no SecurityContext — UUID fallback on every attempt — @Retryable retries generate UUID_B on the same null code path. Fix for all three: DelegatingSecurityContextAsyncTaskExecutor wrapping the ThreadPoolTaskExecutor — captures SecurityContext at task submission time (not thread creation time) — restores it on the executor thread for every invocation including @Retryable retries. Integration test: ThreadPoolTaskExecutor initialized before test, WireMock 503 on attempt 1 then 200 on attempt 2, assert Idempotency-Key headers from both requests are equal.
Three OAuth2 resource server–specific Stripe billing failure modes absent from plain Spring Boot services: the JWT jti claim is a UUID per token issuance — a developer who builds idempotencyKey = jti + billingRef gets UUID_B after the agent refreshes its access token following a network timeout where Stripe already committed ch_A; ReactiveSecurityContextHolder.getContext().flatMap() is lazy per Reactor subscription — Mono.retryWhen() re-subscribes the upstream chain and UUID.randomUUID() inside the flatMap callback generates UUID_B on every retry — wrapping in Mono.defer() does not help because defer() is lazy-per-subscription, not lazy-with-cache; and CompletableFuture.supplyAsync() submits work to ForkJoinPool.commonPool() threads that do not inherit the calling thread’s ThreadLocalSecurityContext — SecurityContextHolder.getContext().getAuthentication() returns null inside the supplier — a defensive null-check generates UUID.randomUUID() on the null path — Stripe creates ch_B on the retry that hits the null path. The fix for all three: compute the idempotency key synchronously from stable, request-invariant inputs (sub claim, not jti) before any reactive operator, async boundary, or retry scope that could change execution context. Integration tests: WireMock 503 on attempt 1, succeed on attempt 2 — capture all Idempotency-Key headers and assert equality across all attempts.
Spring Cloud Gateway’s RetryGatewayFilterFactory re-executes the complete filter chain — including all GlobalFilter instances — on every retry attempt. A GlobalFilter that calls UUID.randomUUID() in its filter() body generates UUID_B on the first retry after Stripe committed ch_A; the exchange.getAttributes().putIfAbsent() guard that appears to cache the key can fail when RetryGatewayFilterFactory creates a mutated exchange for the retry attempt. A second failure mode: ModifyRequestBodyGatewayFilterFactory’s rewrite function fires each time the request body is consumed — retry re-reads the body through the rewrite function — UUID.randomUUID() inside the lambda generates UUID_B on every retry; developers who test with Retry=0 never exercise the rewrite function on retry. A third mode: contextWrite(ctx -> ctx.put(key, UUID.randomUUID())) executes per Reactor subscription — each SCG retry creates a new subscription to the filter chain Mono — UUID_B on every retry subscription; Mono.defer() around the chain is lazy-per-subscription (not lazy-with-cache) and does not protect against this. Fix for all three: synchronous deterministic content-hash key in IdempotencyKeyGlobalFilter at order -200 — stored in exchange attributes before any reactive operator — read from attributes by every downstream filter and rewrite lambda. Integration test: WireMock fails attempt 1 (503), succeeds attempt 2 — assert Idempotency-Key headers from both requests are equal.
Three @QuarkusTest patterns that give your billing code a clean bill of health before it reaches production: @InjectMock on a @RegisterRestClient interface replaces the SmallRye REST Client CDI proxy entirely — @ClientHeaderParam generator methods never fire, so UUID.randomUUID() inside the generator is structurally invisible to tests; WireMock retry scenarios that verify Idempotency-Key is present and non-empty on both retry attempts with matching(".+") pass even when UUID_A on attempt 1 and UUID_B on attempt 2 are distinct values; and %test.MP_Fault_Tolerance_NonFallback_Enabled=false in the test application properties disables @Retry interceptors so UUID.randomUUID() inside an annotated method body never re-evaluates in tests but fires on every retry in production. The correct test configuration: WireMock as the HTTP backend (not @InjectMock), SmallRye Fault Tolerance enabled in at least one integration test class, and an explicit key-stability assertion that captures the Idempotency-Key header value from each retry request and asserts all values are equal. The correct fix: remove @ClientHeaderParam with a dynamic UUID generator; compute a stable content-hash key before the @Retry boundary; pass it as a @HeaderParam to the interface method so the same value arrives at Stripe on every retry attempt.
HC5’s CloseableHttpAsyncClient introduces three Stripe billing failure modes absent from the sync client post: FutureCallback.failed() fires after HC5’s internal AsyncRetryExec exhausts its built-in retries (all with UUID_A on the same SimpleHttpRequest object) — an external retry in failed() builds a new request with UUID.randomUUID() — UUID_B — ch_B when ch_A committed before the final exhaustion; a concurrent NIO batch billing loop submits execute() for all N customers in parallel — server-side error fires all N failed() callbacks simultaneously on reactor threads — each independently generates UUID.randomUUID() — N distinct UUID_B values — ch_B for any customer whose ch_A was committed before the error; and the H2AsyncClientBuilder HTTP/2 client receives a GOAWAY frame from Stripe’s infrastructure — HC5 calls failed() for all in-flight streams — developer retries all with fresh UUIDs — ch_B for streams processed by Stripe before GOAWAY (HC5 does not expose Last-Stream-Id in failed()). Stable content-hash keys computed before the first execute() call, closed over by all callbacks, and passed as explicit parameters through retry helpers fix all three. Pre-flight ON CONFLICT (customer_id, billing_period) DO NOTHING and vault key spend cap as backstops.
gRPC-Kotlin generates coroutine-native stubs where every unary RPC is a suspend fun and every streaming RPC uses Flow. This creates three Stripe billing failure modes absent from the Java gRPC post: a suspend fun retry loop re-enters the billing function on each attempt and re-evaluates UUID.randomUUID() at function entry — UUID_B on the first retry after Stripe committed ch_A from the initial attempt; a CoroutineContext.Element-based idempotency key is recreated with UUID.randomUUID() on each withContext(StripeKeyElement(...)) retry call — the interceptor reads UUID_B from the coroutine context on the retry attempt; and a Flow-based bidirectional stream uses .retry() on the response flow which re-collects the cold request Flow from scratch — flow { emit(...UUID.randomUUID()...) } re-evaluates the producer on reconnect — UUID_B for every customer already charged before the stream failure. Content-hash stable keys, pre-computed Flow request lists, and gRPC Context propagation fix all three.
Teams using AWS SDK v2 often reuse its bundled HTTP clients — ApacheHttpClient (sync), NettyNioAsyncHttpClient (async), UrlConnectionHttpClient (sync, zero extra dependencies) — to call Stripe rather than add another HTTP library. This introduces three failure modes specific to the SDK v2 API: SdkHttpFullRequest is an immutable value object — developers rebuild it per retry attempt and call UUID.randomUUID() in the builder chain on each iteration — UUID_B on the first retry — ch_B alongside committed ch_A; subtler: request construction extracted to a buildStripeRequest() factory method that generates UUID internally — factory called per retry — UUID_B regardless of the method name. NettyNioAsyncHttpClient.execute() returns CompletableFuture<Void> — onError() fires after full request transmission to Stripe — .exceptionally() retry handler builds a new AsyncExecuteRequest with UUID.randomUUID() — UUID_B — ch_B; subtler: recursive retryAsync(int attemptsLeft) method generates UUID at entry — recursive invocation computes UUID_B. ExecutionInterceptor.modifyHttpRequest() fires once per wire attempt including SDK StandardRetryStrategy retries — UUID.randomUUID() inside produces UUID_B on the first retry; subtler: developer moves UUID to beforeExecution() (correct) but then overwrites it in modifyHttpRequest() with a second UUID.randomUUID() call via putAttribute() instead of getAttribute() — UUID_B on retry. Fix: compute stable content-hash key before retry loop; build SdkHttpFullRequest once; store UUID in ExecutionAttributes in beforeExecution() and read (not regenerate) it in modifyHttpRequest(); pre-flight INSERT ... ON CONFLICT (customer_id, billing_period) DO NOTHING; vault key spend cap at expected_total × 1.10.
http4k’s “Server as a Function” typealias Filter = (HttpHandler) -> HttpHandler model makes idempotency key placement a compile-time-silent ordering bug: retryFilter.then(uuidFilter).then(client) places the UUID-generating filter inside the retry boundary — RetryFilter re-invokes uuidFilter per retry — UUID.randomUUID() generates UUID_B — ch_B alongside committed ch_A; ClientFilters.SetHeader("Idempotency-Key", UUID.randomUUID().toString()) evaluates the UUID expression at Kotlin call time (client construction), producing a static string applied to every request from the client instance — all billing calls share one key — Stripe returns cached ch_A for every customer; Kotlin coroutines withTimeout { } fires TimeoutCancellationException while Stripe committed ch_A — catch block retries by calling chargeCustomer() again, which generates UUID_B internally — ch_B. Content-hash idempotency keys, outer-layer filter composition, and caller-provided stable keys fix all three.
reactor.netty.http.client.HttpClient (Reactor Netty, the HTTP layer that powers Spring WebFlux’s WebClient) introduces three Stripe billing failure modes distinct from those covered in the Spring WebFlux post: HttpClient.headers(Consumer<HttpHeaders>) registers a consumer that is called at request-send time, not at Mono assembly time — when retryWhen(Retry.backoff(3, Duration.ofSeconds(1))) triggers a retry, Reactor Netty builds a new outbound request and invokes the consumer again — UUID.randomUUID() inside the consumer produces UUID_B — Stripe committed ch_A before the I/O error — retry with UUID_B creates ch_B — subtler: extracting the consumer to a method reference does not memoize the result — the method is still called per request-send — UUID re-evaluates per send; HttpClient.doOnRequest(BiConsumer<HttpClientRequest, Connection>) is a lifecycle observer hook that fires for every HTTP request Reactor Netty sends to the wire — developers use it for cross-cutting concerns (request signing, audit logging, correlation IDs) and may call UUID.randomUUID() inside the hook not realising that retryWhen causes the hook to fire again on the retried send — UUID_B written via req.header("Idempotency-Key", UUID_B) replaces UUID_A — ch_B — subtler: developer adds stable key in headers() as fix but doOnRequest() runs after headers() and overwrites the stable key with UUID_B; Mono.timeout(Duration) applied to the response publisher fires TimeoutException at the Reactor level while the original request was fully transmitted and Stripe is still processing it (client-side timeout on response receipt, not on TCP send) — onErrorResume(TimeoutException.class, e -> httpClient.post()...) retry handler generates UUID.randomUUID() in .headers() — UUID_B creates ch_B before ch_A’s response arrives — subtler: switching to HttpClient.responseTimeout(Duration) eliminates the Mono.timeout() operator but the same race exists — wrapping in Mono.defer() with UUID.randomUUID() inside the factory re-evaluates UUID per retryWhen re-subscription. Pre-flight INSERT ... ON CONFLICT (customer_id, billing_period) DO NOTHING as authoritative billing mutex, stable content-hash keys computed before the reactive chain, vault key spend cap at expected_total × 1.10 fix all three failure modes.
org.asynchttpclient (the Async HTTP Client library, built on Netty) introduces three Stripe billing failure modes distinct from Apache HttpClient 5, OkHttp, and Java’s built-in HttpClient: IOExceptionFilter.filter() runs before each retry triggered by setMaxRequestRetry(n) — a UUID.randomUUID() call inside the filter generates a new idempotency key per exception, overwriting UUID_A with UUID_B before the first retry — if ch_A was committed before the I/O exception, the retry with UUID_B creates ch_B — subtler variant: developer uses IOExceptionFilter as the sole source of the key (thinking “I only need idempotency on retries”), so the original request carries no Idempotency-Key header at all — Stripe processes the original as a non-idempotent charge creating ch_A without a cache entry — retry carries UUID_B as an entirely new request — ch_B; new RequestBuilder(originalRequest) in AsyncCompletionHandler.onThrowable() copies all headers from the original request including Idempotency-Key: UUID_A — calling .addHeader("Idempotency-Key", UUID.randomUUID()) then appends UUID_B alongside UUID_A rather than replacing it — retry carries two Idempotency-Key header values — Stripe’s behavior with multiple values is undefined and may process UUID_B as a new request creating ch_B — subtler: even switching to .setHeader() still produces ch_B if the value is UUID.randomUUID() — the correct fix is to not change the key at all on retry; and both AsyncCompletionHandler.onThrowable() and ListenableFuture.toCompletableFuture().exceptionally() fire independently when a request fails — developer implements retry billing in both paths — network exception triggers both simultaneously — two concurrent executeRequest() calls with different UUIDs (UUID_B from onThrowable(), UUID_C from exceptionally()) create ch_B and ch_C for a customer already charged ch_A. Stable content-hash keys computed before the first request, single error-handling path selection, pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period), and vault key spend cap complete all three fixes.
Java 11’s java.net.http.HttpClient — the JDK’s built-in HTTP client — introduces three Stripe billing failure modes distinct from Apache HttpClient 5 and OkHttp: CompletableFuture recursive retry re-evaluates UUID.randomUUID() at billing method entry per recursive invocation — initial sendAsync() creates ch_A before IOException — retry recursive call creates ch_B via new UUID from method entry — subtler variant: retry utility wraps a Supplier<CompletableFuture<T>> that delegates to the billing method — supplier re-invokes the method per attempt — UUID re-evaluated per supplier invocation — compared to a supplier that calls sendAsync(preBuiltRequest, ...) with the same immutable HttpRequest object, which correctly reuses the same key — fix: compute content-hash key before the async chain, thread as explicit parameter through all recursive chargeWithKey(key, ...) calls; HttpRequest.Builder.header() accumulates header values rather than replacing — builder reused across retry loop iterations — second attempt calls builder.header("Idempotency-Key", UUID.randomUUID()) which appends a second Idempotency-Key value alongside the first — request has two Idempotency-Key headers — Stripe’s behavior with multiple values is implementation-defined and may process the last (UUID_B) as a new request — Stripe creates ch_B — subtler: HttpRequest.Builder has no .copy() method, pushing developers toward factory methods that call UUID.randomUUID() per factory invocation — fix: compute stable key before loop, build HttpRequest once (immutable, reusable) or use builder.setHeader() to replace rather than accumulate; and Java 21 StructuredTaskScope.ShutdownOnFailure billing fan-out — one subtask fails — scope cancels remaining tasks but does not revert tasks already succeeded — developer’s catch block retries the entire customer list with fresh UUIDs — customers whose subtasks committed Stripe charges before scope cancellation get ch_B from retry — scope.throwIfFailed() provides cooperative cancellation semantics, not ACID rollback — fix: track per-customer subtask state after scope.join(), retry only failures, compute stable content-hash keys before any scope run, add pre-flight INSERT ... ON CONFLICT DO NOTHING per subtask to skip already-billed customers. Stable content-hash keys, setHeader() vs header() discipline, and pre-flight ON CONFLICT DO NOTHING guard fix all three.
Undertow used directly as a standalone embedded server — not Spring Boot’s embedded Undertow adapter — introduces three Stripe billing failure modes from Undertow’s own APIs: UndertowClientClientCallback<ClientConnection> retry reconnects build a new ClientRequest inside completed() — UUID.randomUUID() inside the callback re-evaluates per callback invocation — initial attempt creates ch_A before IOException from the exchange callback — retry callback creates ch_B via fresh UUID in request-building step — subtler variant: connection reused but buildRequest() helper called per conn.sendRequest() invocation — UUID inside helper evaluates fresh per call — fix: compute stable content-hash key before first connect() call, capture as effectively-final local, pass as parameter through all recursive connectAndCharge() calls and all nested callbacks; HttpServerExchange.dispatch() worker-thread retry loop generates new UUID.randomUUID() per loop iteration — initial iteration creates ch_A before SocketTimeoutException — second iteration generates UUID_B — Stripe creates ch_B — subtler variant: UUID stable per request but attempt-index suffix appended (“uuid + ‘-attempt-’ + attempt”) creates a structurally distinct key per iteration by construction, guaranteeing ch_B on iteration 2 regardless of whether the base UUID is stable — fix: compute stable key before dispatch(), attach to exchange via AttachmentKey<String>, read from attachment on worker thread, all loop iterations read same value; and XnioIoThread.executeAfter() delay-based recursive retry calls UUID.randomUUID() at the top of the recursive method entry — each executeAfter() callback re-enters the recursive method — initial direct call creates ch_A before IOException — first executeAfter() callback creates ch_B via fresh UUID from method entry — subtler variant: AtomicReference.compareAndSet(null, UUID.randomUUID().toString()) evaluates UUID argument before the CAS is called — accidentally correct because UUID is discarded when CAS returns false — a subsequent refactoring to ref.set(UUID.randomUUID()) breaks it by resetting the key on every retry — fix: compute stable key in handleRequest() before first scheduleRetry() call, thread as method parameter through all recursive calls, executeAfter() lambda passes same key value to next recursion. Pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period) as the authoritative cluster-wide billing mutex and vault key spend cap at expected_total × 1.10 complete all three fixes.
SmallRye REST Client introduces three Stripe billing failure modes distinct from RESTEasy and Micronaut HTTP Client: @ClientHeaderParam with a dynamic method reference — SmallRye calls the generator method on every outbound request including each @Retry attempt — UUID.randomUUID() inside the generator produces a new idempotency key per retry — initial attempt creates ch_A before socket timeout — retry creates ch_B via fresh UUID from the generator — subtler: instance method on @ApplicationScoped CDI bean called per request (not per bean instantiation) — UUID still re-evaluates per attempt — fix: remove @ClientHeaderParam, pass stable sha256(customerId:billingPeriod:smallrye-billing)[:32] as explicit @HeaderParam method parameter; @Retry placed on @RegisterRestClient interface method directly — SmallRye CDI proxy retries the REST call at proxy level with no natural stable key computation point before the retry boundary — compared to service-level @Retry where stable key can be computed in method body before the REST call — subtler: @Retry(retryOn={WebApplicationException.class}) at interface level — Stripe 408 Request Timeout throws WebApplicationException in MicroProfile REST Client — Stripe may have committed ch_A before sending 408 — @Retry fires with UUID_B — ch_B — fix: move @Retry to service method wrapper, narrow retryOn to ProcessingException (network-level) only, pass stable key as @HeaderParam; and Quarkus Panache @Transactional + SmallRye @Retry stacked — SmallRye FT interceptors bind at priority 1000 (outer), @Transactional at priority 500 (inner) — each @Retry attempt starts a fresh JTA transaction scope — PersistenceException from INSERT unique constraint violation after Stripe committed ch_A triggers @Transactional rollback (which does not undo the Stripe charge — Stripe is not in the JTA transaction) and then triggers @Retry with UUID_B — Stripe creates ch_B — the developer used the unique constraint as the billing idempotency guard, but instead it became the @Retry trigger — fix: pre-flight INSERT ... ON CONFLICT (customer_id, billing_period) DO NOTHING outside the @Retry boundary to skip already-billed customers, exclude PersistenceException from retryOn, upsert billing record after successful Stripe call. Stable content-hash key, pre-flight ON CONFLICT DO NOTHING guard, and vault key spend cap at expected_total × 1.10 fix all three.
Micronaut HTTP Client introduces three Stripe billing failure modes structurally distinct from the general @Retryable AOP pattern: @ClientFilter injects a new UUID.randomUUID() in doFilter() on every HTTP request including each @Retryable retry attempt — doFilter() is called for each new MutableHttpRequest, and @Retryable on the calling service creates a new request per retry, so the initial attempt creates ch_A before a ReadTimeoutException and the retry creates ch_B via a fresh UUID from the filter — fix: compute sha256(customerId:billingPeriod:micronaut-http-billing)[:32] before the @Retryable method boundary and pass as a @Header parameter; transform the filter into a validation filter that asserts the header is already set rather than a generation filter that creates it; ReactorHttpClient.exchange() wrapped in Mono.fromCallable(() -> buildRequest(UUID.randomUUID().toString())) with .retry(3) — Reactor’s .retry() operator re-subscribes the upstream Mono on each failure, the fromCallable factory re-runs per re-subscription, UUID.randomUUID() inside the callable evaluates again, and the initial subscription creates ch_A before a SocketTimeoutException while the first retry re-subscription creates ch_B — the failure is at the reactive stream level not the AOP level, and extends to Mono.defer(), Single.fromCallable() (RxHttpClient), and any other deferred factory — fix: compute stable key before chain assembly, capture as effectively-final variable in enclosing scope, callable reads from outer scope; and @Retryable on a composite service method that encompasses both the Stripe HTTP call and a post-charge database insert catches a database CannotAcquireLockException (PostgreSQL deadlock) after the Stripe call already succeeded — @Retryable is agnostic about which line threw the exception, re-invokes the entire method body with UUID_B, Stripe creates ch_B alongside committed ch_A even though the Stripe call itself succeeded — subtler variant: @Retryable combined with @Transactional — transaction rollback due to DB constraint violation does not affect the already-committed Stripe charge, @Retryable fires after rollback with UUID_B — fix: scope @Retryable narrowly to a Stripe-only method with a stable key parameter; handle DB insert separately with ON CONFLICT DO NOTHING; pre-flight billing mutex outside both retry boundaries. Pre-flight INSERT ... ON CONFLICT DO NOTHING on (customer_id, billing_period) as the authoritative cluster-wide guard and vault key spend caps at expected_total × 1.10 complete all three fixes.
RESTEasy Client’s MicroProfile Rest Client integration introduces Stripe billing failure modes rooted in how the MicroProfile Fault Tolerance @Retry CDI interceptor wraps service methods. Three RESTEasy Client-specific failure modes: @Retry on a @RegisterRestClient service method — the CDI interceptor re-invokes the method body on each retry attempt via context.proceed() — UUID.randomUUID() computed as a local variable inside the method body evaluates fresh on each invocation — initial attempt creates ch_A before WebApplicationException from a socket timeout — first retry creates ch_B — subtler variant: @Retry + @Timeout stacked on the same method — @Timeout fires at the JVM level after 8 seconds, but Stripe may have already received and committed ch_A before the timeout fired — @Retry re-invokes the method with a new UUID and creates ch_B while ch_A is in Stripe’s ledger — fix: compute sha256(customerId:billingPeriod:resteasy-billing)[:32] in the caller and pass as a parameter to the @Retry method; CDI interceptor passes same parameter values on every context.proceed() call; RESTEasy Reactive Mutiny Uni.createFrom().item() supplier re-called on onFailure().retry() re-subscription — UUID.randomUUID() inside the supplier re-evaluates per subscription — initial subscription sends the request, Stripe commits ch_A before socket timeout — retry re-subscription calls the supplier again with UUID_B — Stripe creates ch_B — subtler variant: UUID computed outside the supplier but inside a .flatMap() or .chain() lambda that is inside the retried section — the outer retry re-subscribes past the flatMap step, the lambda re-executes, UUID re-evaluates — fix: compute key before any Uni assembly, capture as effectively-final variable, lambda closes over stable value; and @Scheduled billing trigger fires on all Kubernetes replicas simultaneously — each pod’s @RestClient proxy generates its own UUID.randomUUID() per customer — Stripe’s 24-hour idempotency cache does not protect against concurrent simultaneous requests arriving before Stripe commits the first response to the cache — all three pods create ch_A, ch_B, and ch_C per customer per billing period — fix: pg_try_advisory_lock(hashtext(‘resteasy-monthly-billing:’ + billingPeriod)) serializes billing loop to one pod, or Quartz clustering with quarkus.quartz.clustered=true; pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period) as the permanent durable billing mutex and vault key spend cap at expected_total × 1.10 complete all three fixes.
Vert.x Web Client’s reactive model introduces Stripe billing failure modes that differ structurally from blocking HTTP client retry loops. Three Vert.x-specific failure modes: Single.defer() + retryWhen() — the RxJava 3 deferred factory re-executes on every retry re-subscription including UUID.randomUUID() inside the lambda — initial subscription sends the request and Stripe commits ch_A before a socket timeout fires; retryWhen() re-subscribes the deferred Single, the factory runs again with a new UUID, and Stripe creates ch_B — subtler variant: attempt counter embedded in the idempotency key string (UUID.randomUUID() + "-attempt-" + attempt) guarantees a structurally distinct key per attempt by construction, producing ch_A on attempt 0 and ch_B on attempt 1 regardless of whether the UUID is stable; Future.recover() retry that calls doCharge() again — a new method invocation generates a new UUID.randomUUID() local variable — initial call creates ch_A before HttpException from socket timeout — recover() callback creates ch_B with new UUID — subtler variant: HttpRequest.copy() inside the retry loop copies the per-invocation UUID if the base request is rebuilt per doCharge() call rather than assembled once outside all retry scope — fix: build base request once with stable content-hash key outside the retry scope and copy it per attempt; and vertx.setPeriodic() is JVM-local and fires on all Kubernetes replicas simultaneously — clustered EventBus publish() compounds the race by delivering billing trigger messages to all consumers on all pods rather than round-robin to one — with no distributed advisory lock, all three pods execute the billing loop concurrently, and Stripe’s idempotency cache does not protect against concurrent simultaneous requests that arrive within the same cache-write window — ch_A, ch_B, ch_C per customer per billing period — fix: pg_try_advisory_lock() at billing-run entry serializes execution to one pod per period; use EventBus send() (point-to-point) not publish() (all consumers) for billing triggers. Content-hash stable key sha256(customerId:billingPeriod:vertx-billing)[:32] captured outside Single.defer() and Future factory scope, pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period), and vault key spend cap at expected_total × 1.10 fix all three.
Axon Framework’s CQRS and event-sourcing architecture introduces Stripe billing failure modes that differ structurally from reactive-stream retry patterns. Three Axon-specific failure modes: RetryScheduler re-dispatches the same command object through CommandBus to the @CommandHandler method — UUID.randomUUID() inside the handler generates a fresh UUID on each method invocation, not at command construction time — initial dispatch creates ch_A before a socket timeout raises StripeException and the retry dispatch creates ch_B — fix: carry a content-hash idempotencyKey field in the command object (sha256(customerId:billingPeriod:axon-billing)[:32]) so the @CommandHandler reads the same pre-computed value on every dispatch, plus aggregate billedPeriods guard in @EventSourcingHandler; Saga @SagaEventHandler on a MonthlyBillingDueEvent — Axon’s Tracking Event Processor commits the event processing token only after the handler returns successfully — if the handler throws after the Stripe call (e.g., local PostgreSQL write timeout) the token is not committed, the Saga’s billingInitiated flag changes are discarded (Axon unit-of-work rollback), the event is replayed on the next pod start, the Saga is reloaded with billingInitiated=false, and a fresh UUID.randomUUID() call creates ch_B while ch_A is already in Stripe’s ledger — fix: store content-hash billingKey as a Saga field, set guard before the Stripe call, add pre-flight ON CONFLICT DO NOTHING for post-24h replay; and Spring @Scheduled billing trigger fires on all Kubernetes replicas simultaneously — each pod dispatches ChargeBillingCommand with a different UUID.randomUUID() per customer — Axon’s CommandBus routes by aggregate identifier, not command content — three commands with three different keys each reach @CommandHandler and Stripe independently — ch_A, ch_B, ch_C per customer per billing period — fix: content-hash key in command object plus ShedLock distributed lock on @Scheduled method, or replace with Axon EventScheduler (handled exactly once by the TEP). Pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period) and vault key spend cap at expected_total × 1.10 complete all three fixes.
Ratpack’s Promise<T> is a lazy description of a computation — each subscription triggers a fresh execution of the producer chain from the factory lambda through every flatMap() and Blocking.get() step — Promise.retry() re-subscribes the upstream Promise on each failure, so UUID.randomUUID() inside a Blocking.get() supplier evaluates fresh per retry attempt and the initial subscription creates ch_A before a socket timeout while the first re-subscription creates ch_B. Three Ratpack-specific Stripe billing failure modes: Blocking.get() supplier re-called on each Promise.retry() re-subscription — UUID inside the blocking supplier evaluates fresh per invocation — fix: compute sha256(customerId:billingPeriod:ratpack-billing)[:32] before constructing the Blocking.get() Promise and capture as effectively-final variable so the factory lambda closes over the pre-computed value; flatMap lambda inside an outer Promise.retry() re-subscribed on outer failure — UUID captured outside the inner Blocking.get() but still inside the outer flatMap lambda body re-evaluates fresh per outer re-subscription even though it appears to be evaluated once per customer, because the outer retry re-invokes the entire flatMap lambda body — fix: use deterministic content-hash key derived from stable fields so the key is the same value whether computed on the first flatMap invocation or the fifth; and ExecController.executor().scheduleAtFixedRate() fires independently on every Ratpack node in a Kubernetes Deployment with replicas:3 — Ratpack’s ScheduledExecutorService has no built-in cluster-wide coordination — each node generates a distinct UUID per customer independently, producing ch_A, ch_B, and ch_C per customer per billing period — fix: pg_try_advisory_lock() at the billing run entry point or a separate replicas:1 billing worker Deployment. Pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period) as the permanent durable billing mutex and vault key spend cap at expected_total × 1.10 complete all three fixes.
Spring Retry’s @Retryable annotation wraps the annotated bean method in a Spring AOP proxy — when the method throws a retryable exception, the proxy re-invokes the underlying method from its first line on each retry attempt — UUID.randomUUID() inside the method body evaluates fresh per invocation, so the initial attempt creates ch_A before a StripeException from a socket timeout and the first retry evaluates a new UUID, causing Stripe to create ch_B. Three Spring Retry-specific Stripe billing failure modes: @Retryable re-invokes the annotated method body with fresh UUID.randomUUID() per retry attempt — fix: compute stable sha256(customerId:billingPeriod:spring-retry-billing)[:32] key in the caller and pass as a parameter to the @Retryable method — the AOP proxy passes the same parameter values on every retry invocation; RetryTemplate.execute() re-calls the RetryCallback.doWithRetry() lambda per attempt — UUID inside the lambda evaluates fresh per invocation — subtler variant: including RetryContext.getRetryCount() in the key construction produces a structurally distinct key per attempt by design (count=0 on attempt 1, count=1 on attempt 2) — guaranteed ch_B on every retry — fix: compute stable key before execute() and capture as effectively-final local in the lambda closure, or store in RetryContext attributes on attempt 0 and retrieve on subsequent attempts; and @Retryable billing method called from @Scheduled(fixedRate=86400000) on Kubernetes Deployment with replicas:3 fires on all three pods simultaneously — Spring Retry cannot coordinate across JVMs — each pod generates distinct UUID.randomUUID() per customer — ch_A, ch_B, ch_C per customer per billing period — fix: pg_try_advisory_lock() at the @Scheduled method entry, or deploy billing scheduler to a separate replicas:1 Deployment. Pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period) as the permanent durable billing mutex and vault key spend cap at expected_total × 1.10 complete all three fixes.
Apache Camel’s onException().maximumRedeliveries() error handler re-invokes the processor on each redelivery attempt — UUID.randomUUID() inside the processor body evaluates fresh per invocation, so the initial delivery creates ch_A before a StripeException from a socket timeout and the first redelivery evaluates a new UUID, causing Stripe to create ch_B. Three Apache Camel-specific Stripe billing failure modes: onException().maximumRedeliveries() re-invokes the processor with a fresh UUID.randomUUID() per redelivery — subtler variant: including exchange.getProperty(Exchange.REDELIVERY_COUNTER) in the idempotency key construction produces a structurally distinct key per delivery attempt by design, guaranteeing ch_B on redelivery 1 and ch_C on redelivery 2; the Camel Kafka component hides Kafka message identity behind exchange.getIn().getMessageId(), which is Camel’s internal UUID regenerated per Exchange object — a crash between Stripe call completion (ch_A committed) and Kafka offset commit causes redelivery on restart with a new Camel-internal message ID, producing ch_B while ch_A is already in Stripe’s ledger — subtler variant: using KafkaConstants.OFFSET combined with partition number is unsafe across Kafka partition rebalances, whereas the Kafka message key (set by the producer from deterministic billing fields) is stable across all redeliveries, reconnections, and rebalances; and from(“timer://billing?period=86400000”) fires independently in every JVM in a Kubernetes deployment with replicas:3 — Camel has no built-in cross-JVM timer coordination — all three pods execute the billing route within milliseconds of each other, each processor generating a distinct UUID.randomUUID() per customer, producing ch_A, ch_B, and ch_C per customer per billing period. Content-hash stable keys set as exchange properties at route entry, Kafka message key used as the idempotency key seed instead of Camel’s internal message ID, pg_try_advisory_lock() at the timer route’s first processor step for cluster-wide serialization, and pre-flight ON CONFLICT DO NOTHING fix all three.
Spring Batch’s FaultTolerantStep retry re-invokes ItemWriter.write() on every retry attempt — UUID.randomUUID() inside write() evaluates fresh per invocation, so the initial write creates ch_A before a StripeException from a socket timeout and the first retry invocation evaluates a new UUID, causing Stripe to create ch_B. Three Spring Integration and Spring Batch-specific Stripe billing failure modes: FaultTolerantStep retry re-invokes ItemWriter.write() with a fresh UUID.randomUUID() per attempt — subtler variant: FaultTolerantChunkProcessor’s scatter-gather scan re-calls write() individually for each item in the chunk to isolate the failing item, so a StripeException on item 23 of a 50-item chunk causes up to 22 scan invocations of write() for already-charged customers, each with a fresh UUID — ch_B for customers 1–22 even though their charges succeeded in the original bulk write(); Spring Integration’s RequestHandlerRetryAdvice re-invokes handleRequestMessage(message) with a fresh UUID.randomUUID() per retry — subtler variant: using MessageHeaders.ID as the idempotency key is safe when the retry advice replays the same Message object (stable ID across re-invocations), but unsafe when a MessagePublishingErrorHandler republishes the payload as a new Message with a new MessageHeaders.ID UUID after retry exhaustion — the downstream retry path on the republished message uses the new ID, creating ch_B for a customer whose ch_A was already committed — subtler still: a MessageHandlerChain with RequestHandlerRetryAdvice wrapping the chain causes a BillingMessageTransformer upstream in the chain that calls UUID.randomUUID() to overwrite the idempotency header on every retry chain re-run; and TaskExecutorPartitionHandler concurrent partitions TOCTOU race plus job-restart re-execution of failed chunks — overlapping partition slices (from a billing_status query without row-level locking) cause two partition threads to compute distinct UUID.randomUUID() for the same customer and both reach Stripe, creating ch_A from thread 1 and ch_B from thread 2 — subtler variant: job restart after StepExecution failure re-executes the failed chunk against items already charged in the prior execution with fresh UUID.randomUUID() per customer — ch_B on restart — and two pods both launching from the same queue-delivered message create separate JobExecutions and both run write() with fresh UUIDs before Spring Batch’s JobRepository uniqueness constraint fires. Content-hash idempotency keys from sha256(customerId:billingPeriod:spring-batch-billing)[:32] derived in the write() body, stable key stored in a user-defined MessageHeaders entry at message origination and preserved via copyHeadersIfAbsent() on republish, pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period) as the authoritative cluster-wide billing mutex, and pg_try_advisory_lock() for cross-pod job serialization fix all three.
Resilience4j’s Retry.decorateCallable(retry, supplier) stores a reference to the supplier lambda and calls supplier.call() on every retry attempt — UUID.randomUUID() inside the lambda body evaluates fresh per invocation, so the initial attempt creates ch_A before a StripeException from a socket timeout and the first retry invocation evaluates a new UUID, causing Stripe to create ch_B. Three Resilience4j-specific Stripe billing failure modes: Retry.decorateCallable() re-invokes the supplier lambda with a fresh UUID.randomUUID() per attempt — subtler variant: Decorators.ofCallable() chain with CircuitBreaker, Retry, and RateLimiter stacked — every decorator layer that triggers re-execution re-invokes the innermost supplier lambda with a fresh UUID; TimeLimiter combined with Retry — TimeLimiter submits the callable to a ScheduledExecutorService thread and fires TimeoutException after canceling the future, but the HTTP bytes are already in Stripe’s network stack and Stripe commits ch_A before the response arrives back — Retry fires with fresh UUID.randomUUID() inside the callable creating ch_B — subtler variant: cancelRunningFuture(false) leaves the original callable running, so attempt 1 and attempt 2 are both in-flight simultaneously and both committed at Stripe with distinct idempotency keys; and per-JVM ScheduledExecutorService billing on Kubernetes replicas:3 — TOCTOU race — all three pods call Retry.executeCallable() with distinct UUID.randomUUID() per customer before any pod writes the billing-started record — ch_A, ch_B, ch_C per customer per billing period — subtler variant: thread pool ThreadPoolBulkhead sized too small for the billing run — BulkheadFullException on pool saturation triggers Retry via the same retryOnException predicate that catches StripeException, conflating pre-request rejection with post-commitment failure and firing a fresh UUID per re-invocation. Content-hash idempotency keys from sha256(customerId:billingPeriod:resilience4j-billing)[:32] captured as a final variable before the lambda, pg_try_advisory_lock() for cross-pod scheduler serialization, pre-flight ON CONFLICT DO NOTHING as the authoritative cluster-wide billing mutex, and vault key spend caps at expected_total × 1.10 fix all three.
Apache CXF’s AbstractPhaseInterceptor.handleMessage() is called once per outbound CXF message — a retry loop that re-calls the CXF proxy method re-runs the full PhaseInterceptorChain with UUID.randomUUID() inside handleMessage() — the initial proxy invocation creates ch_A before a ProcessingException from a stale HTTP conduit connection and the retry’s handleMessage() evaluates a fresh UUID, causing Stripe to create ch_B. Three Apache CXF-specific Stripe billing failure modes: an AbstractPhaseInterceptor<Message> computes UUID.randomUUID() inside handleMessage() — subtler variant: CXF async WebClient with an InvocationCallback.failed() retry that calls webClient.async().post(entity, newCallback) again in failed(), each new post() call is a new CXF message that drives the phase chain fresh with a new UUID per failed() execution; a JAX-WS SOAPHandler<SOAPMessageContext> registered on a CXF-generated dynamic proxy computes UUID.randomUUID() inside handleMessage() — Resilience4j @Retry re-invokes the service bean method, which makes another JAX-WS proxy method call, firing the handler chain fresh per proxy invocation, initial invocation creates ch_A before WebServiceException, first @Retry re-invocation creates ch_B — subtler variant: singleton injected CXF proxy bean — singleton scope does not prevent the handler chain from re-executing per method call, @Retry re-invokes the annotated method 3 times producing up to 3 distinct idempotency keys per billing operation; and a per-JVM ScheduledExecutorService fires the billing loop independently on all Kubernetes replicas — with replicas:3, all three pods pass the concurrent hasCompletedForPeriod() check before any pod commits the billing-started record (TOCTOU race), all three CXF proxies generate distinct UUID.randomUUID() per customer, and all three create ch_A, ch_B, ch_C per customer per billing period (subtler variant: ShedLock @SchedulerLock with lockAtMostFor shorter than billing P99 — lock expires mid-run while pod 1 is still executing, pod 2 acquires the expired lock and reads hasCompletedForPeriod() = false for not-yet-processed customers, producing ch_B for those customers). Content-hash idempotency keys from sha256(customerId:billingPeriod:cxf-billing)[:32] stored in BindingProvider.getRequestContext() and propagated into the CXF Exchange before the interceptor chain runs, pg_try_advisory_lock() for cross-pod scheduler serialization, and pre-flight ON CONFLICT DO NOTHING as the authoritative cluster-wide billing mutex fix all three.
JAX-RS’s ClientRequestFilter.filter() is called once per client.target().request().post() invocation — a retry loop that re-invokes the full JAX-RS target call re-runs the filter pipeline with UUID.randomUUID() inside filter() — the initial POST /v1/charges creates ch_A before a ProcessingException from a stale pooled connection, and the retry’s filter() evaluates a fresh UUID, causing Stripe to create ch_B. Three Jersey and JAX-RS-specific Stripe billing failure modes: a ClientRequestFilter computes UUID.randomUUID() inside filter() — subtler variant: a ClientResponseFilter that implements retry by re-submitting via requestContext.getClient().target(...).request().post(...) fires the ClientRequestFilter chain again on each re-submission, generating a fresh UUID per retry without any change to the idempotency filter itself; a MicroProfile Rest Client @RegisterRestClient interface with MicroProfile Fault Tolerance @Retry on the calling CDI bean method — UUID.randomUUID() at the call site argument position re-evaluates per @Retry re-invocation, initial call creates ch_A before WebApplicationException, first @Retry re-invocation creates ch_B — subtler variant: @Retry placed on the interface method via a ClientRequestFilter registered with @RegisterProvider that calls UUID.randomUUID() inside filter(), so each @Retry re-invocation fires a distinct JAX-RS request and the filter fires with a fresh UUID per @Retry attempt; and Jersey’s async AsyncInvoker with InvocationCallback.failed() retry — failed() calls target.request().header("Idempotency-Key", UUID.randomUUID().toString()).post(entity, callback) — UUID.randomUUID() at the header() argument position evaluates when failed() executes, not at request-build time — initial request creates ch_A before SocketTimeoutException, failed() evaluates a new UUID, Stripe creates ch_B — subtler variant: Jersey’s rx()CompletableFuture chain with .exceptionally() retry evaluates UUID.randomUUID() at lambda invocation time, not at chain assembly time. Content-hash idempotency keys from sha256(customerId:billingPeriod:jaxrs-billing)[:32] stored in ClientRequestContext via the property bag, pre-flight ON CONFLICT DO NOTHING as the authoritative cluster-wide billing mutex, and vault key spend caps as the financial backstop fix all three.
Feign’s SynchronousMethodHandler executes an executeAndDecode() loop that re-invokes all registered RequestInterceptors on every retry attempt triggered by Retryer.Default — UUID.randomUUID() inside RequestInterceptor.apply() produces a different Idempotency-Key on the initial request and on Feign’s transparent retry: the initial POST /v1/charges creates ch_A before a RetryableException from a transient connection failure, and the retry’s apply() evaluates a fresh UUID, causing Stripe to create ch_B. Three Feign and Spring Cloud OpenFeign-specific Stripe billing failure modes: a RequestInterceptor computes UUID.randomUUID() per apply() invocation — subtler variant: Spring Cloud OpenFeign with spring-retry on the classpath enables Feign-level retry automatically, and a Retryer bean in the @FeignClient configuration class silently overrides the default no-retry policy, causing every Feign retry to re-invoke apply() with a fresh UUID without any change to the interceptor code; a service method annotated with Resilience4j @Retry that computes UUID.randomUUID() at the @FeignClient interface call site argument position — Resilience4j re-invokes the method body on each retry, re-evaluating the UUID argument expression, initial call creates ch_A before FeignException.ServiceUnavailable, first @Retry re-invocation creates ch_B — subtler variant: Feign’s own Retryer.Default also configured on the client, so Resilience4j outer retry (3 attempts) × Feign inner retry (up to 3 executeAndDecode() loops per Resilience4j attempt) produces up to 9 distinct idempotency keys from a single service instance per billing operation; and a per-JVM @Scheduled billing job fires the Feign-backed billing service independently on all Kubernetes replicas — with replicas:3, all three pods pass the concurrent hasCompletedForPeriod() check before any pod commits the billing-started record (TOCTOU race), all three compute UUID.randomUUID() per customer independently, and all three create ch_A, ch_B, ch_C per customer per billing period (subtler variant: ShedLock @SchedulerLock with lockAtMostFor shorter than the billing job’s tail latency — lock expires mid-run, a second pod acquires it and reads hasCompletedForPeriod() = false for not-yet-processed customers, producing ch_B for those customers). Content-hash idempotency keys from sha256(customerId:billingPeriod:feign-billing)[:32] passed as @RequestHeader parameter to the Feign client interface, pg_try_advisory_lock() for cross-pod scheduler serialization with lockAtMostFor set to billing-run P99 plus safety margin, and pre-flight ON CONFLICT DO NOTHING as the authoritative cluster-wide billing mutex fix all three.
Apache HttpClient 5’s DefaultHttpRequestRetryStrategy re-runs the full HttpRequestInterceptor chain on every retry attempt — UUID.randomUUID() inside Interceptor.process() produces a different idempotency key per chain execution, so the initial POST /v1/charges creates ch_A before a NoHttpResponseException from a stale pooled connection and HC5’s RetryExec fires the interceptor chain again with a fresh UUID, causing Stripe to create ch_B. Three Apache HttpClient 5-specific Stripe billing failure modes: an HttpRequestInterceptor computes UUID.randomUUID() per process() invocation — subtler variant: HttpComponentsClientHttpRequestFactory in Spring 6.x’s RestClient re-runs the underlying HC5 interceptor chain on every @Retryable re-invocation, compounding Spring and HC5 retries into up to 9 distinct idempotency keys per billing operation; a CloseableHttpAsyncClient retry that rebuilds the AsyncRequestProducer inside FutureCallback.failed() — UUID.randomUUID() at producer-construction time re-evaluates per failed() invocation, creating ch_B on the first async retry — subtler variant: HC5 HTTP/2 GOAWAY-triggered transparent retry rebuilds the producer via a factory lambda with a fresh UUID, invisible to the billing callback with no exception raised; and a per-JVM ScheduledExecutorService fires the billing job independently on all Kubernetes replicas — with replicas:3, all three pods pass the concurrent hasCompletedForPeriod() check before any pod commits the billing-started record (TOCTOU race), all three generate distinct UUID.randomUUID() per customer, and all three create ch_A, ch_B, ch_C per customer per billing period (subtler variant: retry-on-ConnectionRequestTimeoutException calls chargeCustomer() again with fresh UUID per call on pool exhaustion). Content-hash idempotency keys from sha256(customerId:billingPeriod:httpclient5-billing)[:32] stored in HttpClientContext and read by the interceptor, pg_try_advisory_lock() for cross-pod scheduler serialization, and pre-flight ON CONFLICT DO NOTHING as the authoritative cluster-wide billing mutex fix all three.
OkHttp’s application interceptors run on the initial request and on every transparent connection-pool retry — UUID.randomUUID() inside Interceptor.intercept() produces a different idempotency key per attempt, so the initial POST /v1/charges creates ch_A before a pooled-connection SocketException and OkHttp’s transparent retry sends ch_B with a fresh key. Three OkHttp and Retrofit-specific Stripe billing failure modes: an application Interceptor computes UUID.randomUUID() per intercept() invocation — subtler variant: moving UUID generation to a network interceptor via addNetworkInterceptor() fires the UUID on redirects, auth challenges, and physical retries, making the failure worse; a Retrofit async retry handler re-invokes the service interface method to get a fresh Call with a new UUID — initial call creates ch_A before IOException, retry’s re-invocation creates ch_B — subtler variant: Kotlin coroutine retry lambda re-calls the suspend fun, which re-evaluates UUID.randomUUID() at the argument expression position on every coroutine retry iteration; and a per-JVM ScheduledExecutorService (or Kotlin fixedRateTimer()) fires the billing job independently on every Kubernetes replica — with replicas:3, all three pods pass the concurrent hasCompletedForPeriod() database check before any pod commits the billing-started record (TOCTOU race), all three generate distinct UUID.randomUUID() per customer, and all three create ch_A, ch_B, ch_C per customer per billing period. Content-hash idempotency keys from sha256(customerId:billingPeriod:okhttp-billing)[:32], key computed in calling code before OkHttpClient.newCall(request).execute(), Call.clone() for async retry, pg_try_advisory_lock() for cross-pod scheduler serialization, and pre-flight ON CONFLICT DO NOTHING as the authoritative cluster-wide billing mutex fix all three.
Armeria’s RetryingClient decorator re-executes the decorator chain downstream of it on each retry attempt — a SimpleDecoratingHttpClient placed between RetryingClient and the base transport that calls UUID.randomUUID() to generate the Idempotency-Key header fires on the initial request and on every retry: the initial POST to Stripe creates ch_A before a transient ResponseTimeoutException, and the first retry’s decorator invocation produces a fresh UUID so Stripe creates ch_B. Three Armeria-specific Stripe billing failure modes: a SimpleDecoratingHttpClient inside RetryingClient computes UUID.randomUUID() per execute() invocation — subtler variant: WebClient.prepare().header("Idempotency-Key", UUID.randomUUID()) inside a helper called per retry attempt; a recursive CompletableFuture retry callback calls buildBillingRequest() with a fresh UUID.randomUUID() per invocation — initial attempt creates ch_A before ResponseTimeoutException, exceptionally() re-invokes the helper with a new UUID, Stripe creates ch_B (subtler variant: Mono.fromFuture with Mono.defer wrapping both UUID.randomUUID() and the WebClient call — retryWhen() re-subscriptions re-evaluate the defer factory and UUID on every attempt); and Armeria’s per-JVM ScheduledExecutorService from blockingTaskExecutor() fires the billing job independently on every Kubernetes replica — with replicas:3, all three pods pass a concurrent hasCompletedForPeriod() check before any pod commits the billing-started record (TOCTOU race), all three generate distinct UUID.randomUUID() per customer, and all three create ch_A, ch_B, ch_C per customer per billing period. Content-hash idempotency keys from sha256(customerId:billingPeriod:armeria-billing)[:32], pg_try_advisory_lock() for cross-pod scheduler serialization, and pre-flight ON CONFLICT DO NOTHING fix all three.
Play Framework’s Akka-backed concurrency model and Future-based async APIs make retry feel natural to compose, but three failure modes specific to Play each produce a duplicate Stripe charge from a different idempotency key: akka.pattern.retry accepts a () => Future[T] supplier and re-invokes it on every retry attempt — UUID.randomUUID() inside the supplier evaluates fresh on every invocation, so the initial call creates ch_A before a connection timeout and the first retry’s supplier invocation creates ch_B (subtler variant: Play WS client with UUID.randomUUID() in the withHttpHeaders argument evaluates the UUID at call time, inside the supplier, re-evaluating on every retry invocation); Future.recoverWith chains a new billing call with a fresh UUID.randomUUID() when the initial Future fails — the initial call creates ch_A before the WSRequestHolder.post() read timeout fires, recoverWith fires with a new UUID, Stripe creates ch_B (subtler variant: wrapping recoverWith in a Retry.retry() re-applies recoverWith repeatedly, each invocation fires retryBilling with a fresh UUID creating ch_B on attempt 2 and ch_C on attempt 3 if all attempts time out after Stripe commits each charge); and Akka Scheduler’s scheduleAtFixedRate creates one independent timer per JVM with no cross-pod coordination — with replicas:3 on Kubernetes, all three pods fire simultaneously, all three call hasCompletedForPeriod() concurrently before any pod writes the billing-started record (TOCTOU race), all three generate UUID.randomUUID() per customer independently creating ch_A, ch_B, ch_C per customer per billing period (subtler variant: Akka ClusterSingleton ensures billing runs on one cluster node at a time but a rolling deploy terminates the old singleton mid-run while the new pod starts fresh and reads hasCompletedForPeriod() = false, creating ch_B for already-charged customers). Content-hash idempotency keys from sha256(customerId:billingPeriod:play-billing)[:32], pg_try_advisory_lock() for cross-pod Akka Scheduler serialization, ClusterSingleton with pre-flight per-customer completion check, and pre-flight ON CONFLICT DO NOTHING as the authoritative cluster-wide billing mutex fix all three.
Spring Retry’s @Retryable is an AOP proxy that calls MethodInvocation.proceed() on every retry attempt, re-executing the annotated method body from its first line — so UUID.randomUUID() at method entry evaluates fresh on every retry, the initial POST to Stripe creates ch_A before a transient error, and the first retry’s new UUID causes Stripe to create ch_B. Three Spring Boot-specific Stripe billing failure modes: @Retryable AOP re-invocation (subtler variant: @Retryable + @Transactional proxy chain — the transaction commits the billing INSERT on the initial attempt, the retry’s new transaction hits a UNIQUE constraint violation, and DataIntegrityViolationException may itself be retried if retryFor is too broad); RestTemplate’s ClientHttpRequestInterceptor computing a fresh UUID per execute() call, re-triggered by every @Retryable retry (subtler variant: MockRestServiceServer in unit tests does not validate idempotency key uniqueness so tests pass green while production sends a fresh UUID on each retry); and @Scheduled billing jobs running on all three Kubernetes replicas simultaneously with no cross-pod coordination — TOCTOU race on the database check, three independent UUID values per customer, ch_A, ch_B, and ch_C per billing period (subtler variant: Spring Batch job with in-memory MapJobRepository launches independent job instances on all three pods with no JobRepository coordination). Content-hash keys from sha256(customerId:billingPeriod:spring-billing)[:32], ShedLock @SchedulerLock, pg_try_advisory_lock(), and pre-flight ON CONFLICT DO NOTHING fix all three.
Apache HttpClient’s request interceptor chain executes on every send attempt — including every retry from DefaultHttpRequestRetryHandler and ServiceUnavailableRetryStrategy — so a UUID.randomUUID() call inside an HttpRequestInterceptor that sets the Idempotency-Key header produces a new key on each attempt: the initial POST to Stripe creates ch_A before a transient 503, and the first retry’s interceptor produces a fresh UUID so Stripe creates ch_B. Three Dropwizard-specific Stripe billing failure modes: Apache HttpClient’s interceptor chain re-executes on every retry; retry wrappers like Failsafe that re-build the HttpPost inside the retry lambda generate a fresh UUID per attempt (subtler variant: HttpPost built with stable key outside the lambda but setHeader("Idempotency-Key", UUID.randomUUID()) inside the lambda overwrites the stable value for per-attempt tracing, ch_B on the first retry); and Dropwizard’s ManagedScheduledExecutorService fires the billing job independently on every Kubernetes replica with no cross-pod coordination — three pods pass a concurrent database check before any pod commits, generate distinct UUID.randomUUID() values per customer, and create ch_A, ch_B, and ch_C per billing period (same failure with dropwizard-jobs using Quartz RAMJobStore). Fixes: store stable content-hash key in HttpClientContext so interceptors read it instead of regenerating; compute stableKey() before the Failsafe lambda and capture as effectively-final; pg_try_advisory_lock() as cross-pod distributed mutex; Quartz JDBCJobStore with clustering for dropwizard-jobs; and pre-flight ON CONFLICT DO NOTHING on (customer_id, billing_period) as the authoritative cluster-wide billing mutex.
Helidon MP’s MicroProfile Fault Tolerance @Retry annotation and Helidon SE 4.x’s virtual-thread-friendly retry loops both make retry feel natural to add, but where the idempotency key is computed relative to the retry boundary determines whether a transient Stripe error produces a safe retry or a duplicate charge — three failure modes specific to Helidon that each produce a duplicate Stripe charge from a different idempotency key: Helidon MP’s MicroProfile FT interceptor re-invokes the CDI method body in full on each retry attempt via InvocationContext.proceed() — UUID.randomUUID() at method entry fires on every invocation (initial attempt creates ch_A before a transient StripeException, first retry re-invokes the method body with a new UUID and Stripe creates ch_B); a subtler variant arises with @ClientHeaderParam on a Helidon MP REST Client interface — a generateIdempotencyKey() default interface method that calls UUID.randomUUID() is invoked fresh on each call to the interface method, which happens on each @Retry re-invocation of the REST client proxy, so retry 1 sends a different Idempotency-Key header and Stripe creates ch_B even though the key generation appears to be cleanly factored into a separate method; Helidon SE 4.x virtual-thread retry loop with UUID.randomUUID() inside the loop body produces a different idempotency key on each iteration — a common refactoring from Helidon SE 3.x reactive code (where UUID inside a flatMap lambda was equally deferred) accidentally moves the UUID into the loop body, ch_B on the first retry; and Helidon MP @Scheduled (or Helidon SE ScheduledExecutorService) runs the billing method on every Kubernetes replica independently with no cross-pod coordination — with replicas: 3, all three pods call hasCompletedForPeriod() concurrently before any pod commits the billing-started record (TOCTOU race), all three generate distinct UUID.randomUUID() values per customer, and all three create ch_A, ch_B, ch_C per customer per billing period; subtler variant: CDI @Initialized(ApplicationScoped.class) observer fires on every pod start including rolling deploys — new pod starts during an ongoing cron-triggered billing run, observer reads hasCompletedForPeriod() = false (old pod hasn’t finished yet), and starts a concurrent billing run creating an overlap window regardless of the cron schedule. Content-hash idempotency keys derived from sha256(customerId:billingPeriod:helidon-billing)[:32] stable across MicroProfile FT re-invocations, ClientRequestFilter injection for REST client idempotency key supply, pg_try_advisory_lock() for cross-pod scheduler serialization including startup observer paths, pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period) as the authoritative cluster-wide billing mutex, and per-billing-period vault keys capped at expected total × 1.10.
Quarkus’s SmallRye Fault Tolerance and Mutiny both make retry easy to configure, but where the idempotency key is computed relative to the retry boundary determines whether a transient Stripe error produces a safe retry or a duplicate charge — three failure modes specific to Quarkus that each produce a duplicate Stripe charge from a different mechanism: SmallRye Fault Tolerance’s @Retry interceptor calls context.proceed() to re-invoke the annotated CDI method body on each retry attempt — UUID.randomUUID() at method entry fires on every invocation (initial attempt creates ch_A before a transient StripeException, first retry re-invokes the method body with a new UUID and Stripe creates ch_B); a subtler variant involves @Retry combined with @Timeout — the @Timeout interceptor fires a TimeoutException after N milliseconds interrupting the Stripe call before the response arrives but ch_A was already committed on Stripe’s side, @Retry re-invokes the method body and UUID.randomUUID() at method entry produces a new UUID, creating ch_B even though @Timeout was added precisely to limit the blast radius of slow Stripe calls; Mutiny’s Uni.onFailure().retry().atMost(N) re-subscribes the upstream Uni pipeline on each failure — any computation inside a Uni.createFrom().item() supplier lambda, a Uni.createFrom().callable() callable, or a .flatMap() / .chain() lambda is deferred and re-evaluated on each subscription, so UUID.randomUUID() placed inside such a lambda produces a different idempotency key on each retry’s re-subscription (first subscription creates ch_A before a 503, first retry re-subscribes, new UUID in the supplier creates ch_B); a subtler Mutiny variant: UUID generated in a .map() or .chain() stage appears to be “upstream” of the retry but is still deferred — the entire Uni pipeline re-executes on each re-subscription regardless of stage position; and Quarkus @Scheduled (without quarkus-quartz clustering) runs the annotated method on every JVM in a Kubernetes Deployment independently — with replicas: 3, all three pods execute the billing loop at the cron second, a TOCTOU race on the alreadyRan database check means all three pass before any pod commits the billing-started record, all three generate distinct UUID.randomUUID() values per customer, and all three charge each customer creating ch_A, ch_B, and ch_C per customer per billing period across 500 customers. Content-hash idempotency keys derived from sha256(customerId:billingPeriod:quarkus-billing)[:32], quarkus.quartz.clustered=true for native @Scheduled serialization via Quartz database trigger locks, pg_try_advisory_xact_lock() as a backstop for cases the Quartz lock does not cover, pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period) as the authoritative cluster-wide billing mutex, and per-billing-period vault keys capped at expected total × 1.10.
Ktor’s HttpRequestRetry plugin and Kotlin coroutine-based retry loops both make retry feel natural in Ktor applications, but where the idempotency key is computed relative to each retry attempt determines whether a transient Stripe error produces a safe retry or a duplicate charge — three failure modes specific to Ktor that each produce a duplicate Stripe charge from a different idempotency key: the modifyRequest callback in HttpRequestRetry runs only before retry attempts, not the initial request — a developer who puts UUID.randomUUID() inside modifyRequest as the sole source of the Idempotency-Key header sends the initial request with no idempotency key at all (Stripe processes it, creates ch_A, returns a transient 503) and each retry with a fresh UUID so Stripe sees a new key on every attempt and creates ch_B on retry 1 and ch_C on retry 2; a subtler variant occurs when modifyRequest incidentally calls headers { set("Idempotency-Key", UUID.randomUUID().toString()) } alongside other modifications, overwriting the stable hash key that was correctly set in the initial post { } builder so retry 1 carries a different key than the initial request and Stripe creates ch_B; and a manual retry loop that calls the billing suspend fun on each attempt executes UUID.randomUUID() at function entry afresh on every invocation so the first call creates ch_A before HttpRequestTimeoutException fires and the second call creates ch_B; and Application.launch { } starts one independent billing coroutine per Kubernetes replica with no cross-pod coordination — with replicas: 3, all three pods pass a concurrent hasCompletedForPeriod database check before any pod has committed the billing-started record (TOCTOU race), all three run the full billing loop, and each generates distinct UUID.randomUUID() values per customer creating ch_A from pod 1, ch_B from pod 2, and ch_C from pod 3 per customer per billing period. Content-hash idempotency keys derived from sha256(customerId:billingPeriod:ktor-billing)[:32] stable across modifyRequest overwrites, function re-invocations, and multi-pod concurrent billing loops, pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period) as the cluster-wide billing mutex, pg_try_advisory_lock() and pg_try_advisory_xact_lock() for Application.launch coroutine scheduler serialization, and per-billing-period vault keys capped at expected total × 1.10.
Spring WebFlux’s reactive model makes retry deceptively easy to wire up and deceptively easy to get wrong — three failure modes specific to Spring WebFlux that each produce a duplicate Stripe charge from a different idempotency key: UUID.randomUUID() inside a Mono.defer() factory before retryWhen(Retry.backoff()) re-evaluates on every re-subscription so the first attempt creates ch_A before a 503 arrives and the retry’s fresh UUID causes Stripe to create ch_B; Flux.retryWhen() applied at the outermost pipeline level wrapping a flatMap over customer IDs re-subscribes the entire stream from the beginning when any single customer fails, so all 450 customers already successfully charged on the first subscription are re-billed on the retry with new UUIDs creating ch_1B through ch_450B alongside ch_1A through ch_450A already committed in Stripe; and a @Scheduled billing method that returns Mono<Void> is silently discarded by Spring’s ThreadPoolTaskScheduler without subscribing so no billing executes, and the common fix of changing the return type to void and adding .subscribe() causes all three Kubernetes replicas to fire the billing loop simultaneously at cron time, each generating distinct UUID.randomUUID() values per customer and each creating separate charge IDs until the pre-flight database guard stops the race. Content-hash idempotency keys derived from sha256(customerId:billingPeriod:spring-webflux-billing)[:32] stable across defer re-evaluations and method re-invocations, retryWhen scoped inside flatMap to one customer’s Mono, ShedLock @SchedulerLock and pg_try_advisory_lock() for distributed cron coordination, pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period) as the authoritative cluster-wide billing mutex, and per-billing-period vault keys capped at expected total × 1.10.
Micronaut’s AOP-first compile-time design means @Retryable re-invokes the entire annotated method body on each retry attempt — three failure modes specific to Micronaut that each produce a duplicate Stripe charge from a different idempotency key: a service method annotated with @Retryable calls UUID.randomUUID() inside the method body to generate the Stripe-Idempotency-Key header — Micronaut’s DefaultRetryInterceptor calls context.proceed() again on each retry exception, the method body re-executes, UUID.randomUUID() produces a new value, and if the first attempt already created ch_A before the exception was thrown (socket timeout while Stripe was processing the charge), the retry’s new UUID causes Stripe to create ch_B for a customer already charged; a service method annotated with @Retryable returns a Mono<ChargeResponse> and any UUID.randomUUID() call sits inside a Mono.defer{} lambda or at the top of the method body before the reactive chain is assembled — Micronaut’s ReactiveRetryInterceptor re-invokes context.proceed() on each retry to get a fresh Publisher, which re-runs the method body and re-evaluates the defer factory, generating a fresh UUID per retry; and a @Scheduled(fixedDelay = “30d”) billing job runs in a Micronaut application deployed with replicas: 3 on Kubernetes — Micronaut’s built-in scheduler is per-JVM with no cross-pod coordination unlike Quartz with a JDBC JobStore — all three pods fire the scheduled task within seconds of each other, each generates UUID.randomUUID() independently, and all three send a Stripe charge request with a different key creating ch_A, ch_B, and ch_C for the same customer in the same billing period. Content-hash idempotency keys stable across @Retryable re-invocations and Publisher re-subscriptions, pg_try_advisory_lock() and ShedLock for scheduler serialization, pre-flight PostgreSQL ON CONFLICT DO NOTHING, and per-billing-period vault keys capped at expected total × 1.10.
gRPC’s interceptor model, bidirectional streaming, and built-in hedging policy create three Stripe billing failure modes that each produce a duplicate charge from a different idempotency key: a ClientInterceptor that constructs fresh Metadata with a new UUID.randomUUID() inside start() on each retry attempt — if the original RPC reached the billing server and Stripe created ch_A before the server returned UNAVAILABLE due to a Kubernetes rolling restart, the retry generates a new UUID and Stripe creates ch_B for a customer already charged; a bidirectional streaming billing RPC that maintains an unACK’d command queue and replays it on stream reconnect — if any per-stream attribute (stream ID, System.identityHashCode() of the StreamObserver, connection timestamp, HTTP/2 stream sequence number, server-assigned x-grpc-stream-id) is embedded in the idempotency key, each reconnect produces different keys for the same billing commands and Stripe creates ch_B for customers already charged on the prior stream before the ACK arrived; and a ServiceConfighedgingPolicy with hedgingDelay: 3s that sends concurrent copies of the billing RPC to different backend pods when the first attempt does not respond within the delay — each pod’s billing handler calls UUID.randomUUID() on RPC entry, both hedged copies complete and Stripe sees two different idempotency keys, creating ch_A from pod A and ch_B from pod B simultaneously before either is committed to Stripe’s idempotency cache. Three gRPC-specific Stripe billing failure modes with gRPC Java 1.x code, content-hash keys stable across interceptor retries, stream reconnects, and concurrent hedged copies (must not include UUID.randomUUID() inside the interceptor’s start(), System.identityHashCode() of the StreamObserver, System.currentTimeMillis() at stream establishment, HTTP/2 stream ID, or Metadata.get(GRPC_CALL_ID_KEY) from per-call gRPC context), pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period) as a cluster-wide mutex so only the first pod claiming the billing row calls Stripe, caller-supplied idempotency keys extracted from Metadata by a server-side ServerInterceptor and passed to Stripe unchanged, and per-billing-period vault keys capped at expected total × 1.10.
Akka HTTP’s Future-based retry model and reactive streaming pipeline create three Stripe billing failure modes: a recursive Future retry function calls UUID.randomUUID() on each invocation — the original Stripe call creates ch_A before the network timeout, the retry generates a new UUID, and Stripe creates ch_B for a customer already charged; Http().cachedHostConnectionPool() wrapped in RetryFlow.withBackoff() calls the request-builder function inside the retry callback, generating a fresh idempotency key on each retry — when a stale TCP connection causes a pool-level StreamTcpException, the retry re-builds the request with a new UUID and Stripe creates ch_B while ch_A was created from the original pooled connection before the RST arrived; and withRequestTimeout(15.seconds) fires a 503 before Stripe’s API call completes — an upstream ALB with proxy_next_upstream timeout retries the identical request to a second Akka HTTP pod, that pod generates its own UUID.randomUUID() in its billing route, and Stripe creates ch_B on the second pod while ch_A is still processing on the first. Three Akka HTTP-specific Stripe billing failure modes with Scala Akka HTTP 10.x code, content-hash idempotency keys stable across recursive retries, RetryFlow re-materializations, and cross-pod upstream retries (must not include UUID.randomUUID() inside the billing function, System.currentTimeMillis() at request-build time, attempt counter in the key, per-ActorSystem startup UUID, or pod hostname from HOSTNAME env var), pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period), caller-supplied idempotency keys passed through to Stripe to survive load-balancer cross-pod retries, and per-billing-period vault keys capped at expected total × 1.10.
Netty’s ChannelPipeline introduces three Stripe billing failure modes: an exceptionCaught() handler retries by re-firing the BillingRequest through the pipeline head via ctx.pipeline().fireChannelRead(msg), but the BillingHandler at the tail calls UUID.randomUUID() inside channelRead0() on every invocation — after the original Stripe call creates ch_A and a downstream AuditHandler throws a SQLTimeoutException, the retry calls channelRead0() again, generates a new UUID, and Stripe creates ch_B for a customer already charged; a billing service triggers a Stripe charge in channelActive() when a metered API connection is established and guards against double-billing with a channel.attr(BILLING_INITIATED) flag, but Netty creates a fresh DefaultAttributeMap for each new Channel — when channelInactive() fires and a ChannelFutureListener reconnects via Bootstrap.connect(), the new channel has no inherited attributes, channelActive() fires and finds alreadyBilled = null, and System.currentTimeMillis() at the new connection time produces a different idempotency key and ch_B; and a handler schedules periodic billing via channel.eventLoop().scheduleAtFixedRate() inside channelActive() — in a multi-replica Kubernetes Deployment each pod’s NioEventLoopGroup creates an independent billing timer per channel, and any per-channel value in the key (Channel.id().asShortText(), Thread.currentThread().getId(), Instant.now() at schedule-registration time) produces distinct keys per channel and replica and Stripe creates ch_A through ch_N per customer per billing period. Three Netty-specific Stripe billing failure modes with Java Netty 4.1.x code, content-hash idempotency keys stable across exceptionCaught() retries and channelActive() reconnects (must not include UUID.randomUUID() inside channelRead0(), System.currentTimeMillis() at connection time, Channel.id().asShortText(), Thread.currentThread().getId(), or channel.remoteAddress().toString()), pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period) as a durable guard that survives channel lifecycle events, PostgreSQL advisory locks and Kubernetes leader election for scheduleAtFixedRate() serialization across channels and replicas, and per-billing-period vault keys capped at expected total × 1.10.
Akka Typed’s EventSourcedBehavior recovery finds BillingInProgress state from a replayed BillingInitiated event and resumes billing via a RecoveryCompleted self-message, but the ResumePendingBilling handler computes the Stripe idempotency key using System.currentTimeMillis() at recovery time rather than the initiatedAt timestamp stored in the persisted event — a different millisecond on every recovery produces a different key and Stripe creates ch_B for a customer whose original charge (ch_A) completed before the actor crashed; a billing scheduler actor sends BillCustomer commands to sharded billing entities via context.ask() and retries on timeout, but each context.ask() call allocates a unique ephemeral replyTo: ActorRef[BillingReply] whose path is captured in the billing entity’s idempotency key — the second ask carries a different replyTo path, producing a different key, and Stripe creates ch_B for a customer already charged by the first ask; and a ClusterSharding billing entity uses Behaviors.withTimers to schedule a periodic billing tick at entity activation time, so a rolling deploy that triggers shard rebalancing deactivates the entity on node A and reactivates it on node B with a new activation timestamp — when the timer fires on node B at a wall-clock time outside Stripe’s 24-hour idempotency cache window relative to node A’s original fire, the entity activation timestamp in the key produces ch_B. Three Akka Typed-specific Stripe billing failure modes with Scala Akka Typed 2.9.x code, content-hash keys stable across EventSourcedBehavior recovery cycles and shard migrations, billingInProgress and issuedPeriods guard conditions in the event-sourced commandHandler, pre-flight PostgreSQL ON CONFLICT DO NOTHING, and per-billing-period vault keys capped at expected total × 1.10.
Akka Streams’ RestartSource.withBackoff() calls its source factory on each restart — a UUID.randomUUID() generated inside the factory lambda to correlate billing run logs produces a new value on every restart, billing already-charged customers again with a different idempotency key and Stripe creates ch_B for every customer processed before the stream failed; Alpakka Kafka Consumer.committableSource with mapAsync(N) parallelism batches and commits offsets after all N in-flight Stripe calls complete, so an ActorSystem crash after 3 of 4 parallel Stripe calls return (ch_A, ch_B, ch_C created) but before the offset batch commits causes Kafka to replay all 4 messages on restart — any per-ActorSystem startup UUID or System.currentTimeMillis() at processing time in the idempotency key produces different values on replay and Stripe creates ch_A′ through ch_C′; and Source.tick(initialDelay, interval, tick) is per-ActorSystem with no cluster-wide coordination — in a Kubernetes Deployment with replicas: 3 three ActorSystems fire Source.tick() simultaneously, each running the complete billing loop for all customers, and a pod hostname from the HOSTNAME environment variable or a per-ActorSystem name in the idempotency key produces three distinct keys per customer per billing period creating ch_A, ch_B, and ch_C. Three Akka Streams-specific Stripe billing failure modes with Scala Akka Streams 2.9.x code, content-hash idempotency keys stable across RestartSource re-materializations and Alpakka Kafka crash-recovery replays (must not include per-factory UUID, currentTimeMillis at materialization time, per-ActorSystem startup UUID, consumer group generation ID, or pod hostname), an Akka Cluster Singleton and pg_try_advisory_lock() pattern for Source.tick() billing serialization across pods, pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period), write-before-call ordering, and per-billing-period vault keys capped at expected total × 1.10.
Eclipse Vert.x’s eventBus.send() reply timeout triggers retry logic — the consumer already processed the billing message and called stripe.charges.create() (creating ch_A), but the reply never arrived before the timeout fired; the retry delivers a fresh Message object with a new per-send UUID or System.currentTimeMillis() in the idempotency key, and Stripe creates ch_B for a customer already charged; vertx.setPeriodic() is per-instance and not cluster-aware — in a 3-node clustered deployment three independent timer events fire simultaneously, each running the complete billing loop for all customers; any node-local value in the idempotency key (Vert.x node ID, context deployment ID, Instant.now() at callback time) produces three distinct keys and Stripe creates ch_A, ch_B, ch_C per customer per billing period; and Vert.x Web Client’s rxSend().retry() (or Single.defer().retry()) re-subscribes the RxJava chain on each retry, re-evaluating any System.currentTimeMillis() or UUID.randomUUID() captured inside Single.defer() — if attempt 1 reached Stripe and created ch_A but the HTTP response was lost in transit, attempt 2 with a new millisecond-derived key creates ch_B. Three Eclipse Vert.x-specific Stripe billing failure modes with Java Vert.x 4.x code, content-hash idempotency keys stable across EventBus retries and Web Client re-subscriptions (must not include per-send UUID, send timestamp, currentTimeMillis() inside defer(), Vert.x node ID, or context deployment ID), a Vert.x SharedData cluster-wide lock pattern to serialize setPeriodic() billing across cluster nodes, pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period), write-before-call ordering, and per-billing-period vault keys capped at expected total × 1.10.
Quartz Scheduler’s getFireTime() is the actual JVM wall-clock instant the job started — different on every fire including misfire re-fires — and getScheduledFireTime() changes under the default CronTrigger misfire instruction MISFIRE_INSTRUCTION_FIRE_ONCE_NOW, so a billing job that derives its Stripe idempotency key from either timestamp generates a different key when the scheduler restarts and fires the missed trigger, and Stripe creates ch_B for every customer charged before the crash; in a clustered Quartz setup, a JVM GC pause (stop-the-world full GC) causes the billing node’s database connection to time out and the QRTZ_LOCKS row lock to be released by the database while the job thread is still running, a second Quartz node acquires the lock, sees the trigger still in WAITING state (the first node’s transaction rolled back on connection drop), fires the same trigger, and both nodes call stripe.charges.create() simultaneously — if either includes a hostname, SchedulerInstanceId (especially under instanceId=AUTO which generates a unique UUID per JVM startup), or thread-local random seed in the key, the two keys differ and Stripe creates ch_B; and SimpleTrigger with MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_COUNT creates a new trigger instance with a new startTime, getFireTime(), and getScheduledFireTime() when the scheduler finds the trigger in ACQUIRED state past the misfireThreshold on restart, meaning any billing key derived from the trigger’s scheduled time or JobDataMap counter re-generated by the management job produces a different hash for what is operationally a retry of the same billing intent. Three Quartz-specific Stripe billing failure modes with Java Quartz 2.3 Job.execute() code, content-hash idempotency keys stable across misfire re-fires and cluster node assignments (must not include getFireTime(), getScheduledFireTime(), getSchedulerInstanceId(), timesTriggered, JobDataMap counters re-generated at trigger creation, or hostname), pre-flight PostgreSQL ON CONFLICT DO NOTHING on (customer_id, billing_period), write-before-call ordering, and per-billing-period vault keys capped at expected total × 1.10.
Spring Cloud Stream’s default max-attempts=3 retry re-invokes the entire Consumer<T> lambda when a downstream exception fires after stripe.charges.create() returns ch_A — a UUID.randomUUID() or KafkaHeaders.DELIVERY_ATTEMPT-based idempotency key generated inside the lambda body produces a new value on attempt 2 and Stripe creates ch_B; the Kafka Binder’s default BATCHAckMode defers Kafka offset commits until all records in a poll batch are processed, so a billing pod that crashes between processing record 4 of 10 (after Charge.create() returns for records 1–4 but before the batch commit) causes all 10 records to be redelivered to the new partition owner, creating ch_B₁–ch_B₄ for already-charged customers — a failure mode invisible when testing the Consumer<T> directly because BATCH semantics only surface through the SCS Kafka Binder runtime; and the RabbitMQ Binder’s republish-to-dlq: true configuration parks failed Consumer exceptions in a DLQ with x-exception-stacktrace headers, creating an operational pattern where engineers move messages showing StripeException: Read timed out back to the main queue for reprocessing — a Stripe SDK timeout does not mean the charge was not created server-side, and DLQ reprocessing without the original idempotency key creates ch_B for customers already charged before the network dropped, especially dangerous when the DLQ message is more than 24 hours old and Stripe’s idempotency window has expired. Three Spring Cloud Stream-specific Stripe billing failure modes with Java SCS 3.x functional-model Consumer<T> code, content-hash idempotency keys stable across SCS retry attempts and Kafka BATCH redeliveries (must not include UUID.randomUUID(), KafkaHeaders.DELIVERY_ATTEMPT, System.currentTimeMillis(), or Spring Batch stepExecution.id), pre-flight PostgreSQL ON CONFLICT DO NOTHING, write-before-call ordering, and per-billing-period vault keys capped at expected total × 1.10.
Solace PubSub+ Guaranteed Messaging redelivers a billing message when the consumer’s session disconnects after stripe.charges.create() completes but before msg.ack() is sent — the flow goes down, the broker marks the message unacknowledged, and the consumer receives the same billing event on flow rebind and creates ch_B; a publisher accidentally using Direct delivery mode (the default in many Solace SDK examples) fans the billing event out to every billing pod’s DirectMessageReceiver simultaneously rather than load-balancing it across one PersistentMessageReceiver, so three billing pods create three Stripe charges for the same customer in the same billing period; and an operator triggers a Solace Message Replay during incident recovery, replaying all billing events from a 30-minute window including events for customers whose charges already completed before the crash — the restored billing service calls stripe.charges.create() for each replayed event and creates ch_B per customer. Three Solace-specific failure modes with Python solace-pubsubplus SDK code, content-hash idempotency keys stable across flow rebinds and ReplayStartLocationFactory-triggered replays (must not include Solace message ID, flow ID, session ID, redelivery count, or msg.get_receive_time() — all change across redeliveries), pre-flight PostgreSQL ON CONFLICT DO NOTHING, write-before-call ordering, and per-billing-period vault keys capped at expected total × 1.10.
Apache Pulsar’s batchReceive() API returns a Messages<T> object and consumer.acknowledge(messages) acknowledges the entire batch atomically — a crash between processing message 47 of 50 and the batch ack call redelivers all 50 messages including the 47 already charged; consumer.negativeAcknowledge(messages) called on the batch object on a per-message Stripe exception (caught in the outer loop handler) redelivers all 50 messages including the messages processed successfully before the exception — the correct call is consumer.negative_acknowledge(msg) on the individual failing message, not the batch object, a distinction absent from most batch receiver examples; and a per-batch UUID or batch-start timestamp generated inside the batchReceive() call loop to “trace which billing run a charge came from” and included in the Stripe idempotency key changes on every batchReceive() call including ack-timeout-triggered redeliveries of the same messages, producing different keys for customers already charged in the previous batch. Three Pulsar batch receiver-specific failure modes with Java and Python code, content-hash idempotency keys excluding all per-batch generation metadata (batch UUID, batch-start timestamp, batch sequence counter), per-message acknowledgment within the batch loop, pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, and per-billing-period vault keys capped at expected total × 1.10.
Hazelcast’s IMap.executeOnKey() sends an EntryProcessor to the primary partition owner — if the primary member fails after stripe.charges.create() returns ch_A but before process() returns, Hazelcast promotes the backup to primary and re-executes the EntryProcessor; a UUID.randomUUID() generated inside process() produces a new value on re-execution and Stripe creates ch_B; a Reliable Topic subscriber that checkpoints its RingBuffer sequence position in an IMap entry with MaxIdle eviction will lose the checkpoint if a maintenance window exceeds the TTL — on restart the subscriber reads null from IMap, falls back to OLDEST initial position, and replays the entire 7-day RingBuffer retention window including billing events already processed and charged in previous runs; and CP Subsystem FencedLock session expiry during a JVM GC pause releases the lock and allows a second member to acquire it with a new fencing token — if the developer included the fencing token in the Stripe idempotency key to “version the billing run,” the second member’s different fence value produces a different key and Stripe creates ch_B for a customer already charged by the first lock holder. Three Hazelcast-specific failure modes with Java code (IMap.executeOnKey(), Reliable Topic ReliableMessageListener, CP Subsystem FencedLock), content-hash idempotency keys excluding all Hazelcast execution and session metadata (EntryProcessor execution UUID, RingBuffer sequence, fencing token, Hazelcast member UUID, CP session ID), pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, write-before-call ordering, and per-billing-period vault keys capped at expected total × 1.10.
NSQ’s per-message Attempts counter (uint16) starts at 1 on first delivery and increments on every requeue — including the requeue triggered when a consumer crashes after stripe.charges.create() returns ch_A but before message.Finish() is called; a developer who includes msg.Attempts in the Stripe idempotency key gets a different key on every redelivery and Stripe creates ch_B through ch_N with each retry, up to the consumer’s max-attempts limit (go-nsq default: unlimited); NSQ consumers maintain a separate TCP connection per nsqd instance discovered via nsqlookupd, and developers who generate a UUID at connection establishment time and include it in Stripe idempotency keys to “scope billing keys to this consumer session” get a different UUID when the nsqd connection drops and the consumer reconnects — a billing message redelivered from nsqd’s disk persistence after the consumer reconnects is processed with a new connection UUID and creates ch_B for a customer already charged before the crash; and pynsq’s Reader processes messages in a Tornado IOLoop — a synchronous stripe.Charge.create() call in a message handler blocks the single-threaded IOLoop for the duration of the Stripe request, preventing TOUCH keepalive commands from being sent to nsqd, causing nsqd to requeue the message after msg-timeout (60s default) expires while Stripe is still processing ch_A, and the requeued message is delivered to a second consumer that creates ch_B. Three NSQ-specific failure modes with Go (go-nsq) and Python (pynsq) code, content-hash idempotency keys excluding all NSQ delivery metadata (Attempts counter, connection UUID, nsqd TCP address, per-handler invocation timestamp), pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, the IOLoop-safe async pattern for pynsq billing handlers using loop.run_in_executor(), and per-billing-period vault keys capped at expected total × 1.10.
ZeroMQ’s lazy-pirate REQ/REP timeout recovery destroys and recreates the socket — a timestamp captured at socket construction (int(time.time() * 1000)) or object ID (id(socket)) used as a Stripe idempotency key component becomes a different value on the recreated socket, and stripe.charges.create() on the retry creates ch_B because the REP billing worker called Stripe successfully on the first request and crashed before sending the reply; a PUSH/PULL pipeline with application-level ACK acknowledgments maintains a monotonic sequence counter in the PULL worker process — when the worker is killed after calling stripe.charges.create() but before sending the ACK, the producer resends the billing message and the restarted worker’s counter has reset to zero, so the same billing message gets ack_sequence=0 instead of the original ack_sequence=41 and stripe.charges.create() produces ch_B from a different key; and ZeroMQ ROUTER/DEALER routing assigns a random 5-byte identity to every DEALER socket at construction time — socket.setsockopt(zmq.IDENTITY, ...) is not called by default — when the DEALER disconnects and reconnects the new socket receives a new random identity, and a billing service that extracts the DEALER identity from the routing frame (frames[0] of recv_multipart()) and includes dealer_identity.hex() in the Stripe idempotency key builds a completely different key for the same billing message on the DEALER’s retry and Stripe creates ch_B. Three ZeroMQ-specific failure modes with Python pyzmq code, content-hash idempotency keys excluding all ZeroMQ socket metadata (socket creation timestamp, id(socket), ACK sequence counter, DEALER identity bytes, socket.getsockopt(zmq.LAST_ENDPOINT)), pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, write-before-send ordering, and per-billing-period vault keys capped at expected total × 1.10.
IBM MQ’s MQMD.BackoutCount field records the number of times a message has been rolled back (via MQBACK) — a developer who includes BackoutCount in the Stripe idempotency key to “version” delivery attempts gets a different key on every rollback and Stripe creates a new charge for each one; with BOTHRESH=5 the same billing message can produce five Stripe charges (one per BackoutCount value from 0 to 4) before the queue manager moves the message to the backout queue; IBM MQ Dead Letter Queue retry via rfhutil, IBM MQ Explorer, or custom MQPUT scripts calls MQPUT to re-enqueue the message and IBM MQ generates a new 24-byte MQMD.MsgId for the re-enqueued message — the original MsgId is not preserved by default — a developer whose Stripe idempotency key is derived from bytesToHex(message.messageId) gets a completely different key on DLQ retry and Stripe creates ch_B for a customer charged during any of the original delivery attempts, especially dangerous when DLQ messages are retried days later beyond Stripe’s 24-hour idempotency window; and IBM MQ automatic client reconnect (MQCNO_RECONNECT) rolls back the in-progress unit of work on disconnection and re-delivers billing messages obtained under MQGMO_SYNCPT but not yet committed via MQCMIT — developers who captured the MQHCONN connection handle value or a connection timestamp at MQCONN time and included it in the idempotency key get a different key after reconnect and stripe.charges.create() creates ch_B for a customer whose ch_A was created before the connection dropped. Three IBM MQ-specific failure modes with Java code using the native IBM MQ API (com.ibm.mq.MQQueueManager, MQMessage, MQGetMessageOptions), content-hash idempotency keys that exclude all IBM MQ delivery and session metadata (BackoutCount, MsgId, CorrelId, MQHCONN handle, connection timestamp, reconnect counter), pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, write-before-commit ordering, and per-billing-period vault keys capped at expected total × 1.10.
Artemis’s _AMQ_DUPL_ID duplicate detection fires at message enqueue time from the producer — not at consumer delivery time — so a consumer crash between stripe.charges.create() returning and the consumer ACK causes redelivery that bypasses duplicate detection entirely and calls Stripe a second time; _AMQ_GROUP_ID message groups pin delivery to a single consumer, which leads developers to maintain in-memory HashSet dedup state on that consumer, but when Artemis kills the pinned consumer (OOM, SIGKILL, max-delivery-attempts triggered by an unrelated error) and re-pins the group to a new consumer, the new consumer’s dedup set is empty and all redelivered billing messages call stripe.charges.create() for already-charged customers; and the Artemis dead-letter address retryMessage() management operation (console, JMX, Artemis CLI) re-enqueues the dead-lettered billing message as a new message with a new broker-assigned JMSMessageID — the original JMSMessageID is moved to _AMQ_ORIG_MESSAGE_ID but is no longer the active message identifier — a developer whose Stripe idempotency key is derived from message.getJMSMessageID() gets a completely different key for the DL-retried message and Stripe creates ch_B for a customer charged during one of the 10 original delivery attempts where Stripe succeeded but the consumer could not acknowledge. Three Artemis-specific failure modes with Java JMS API code, content-hash idempotency keys excluding all Artemis routing and delivery metadata (_AMQ_DUPL_ID, JMSXGroupID, JMSMessageID, _AMQ_SCHED_DELIVERY, _AMQ_ORIG_MESSAGE_ID), pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, write-before-acknowledge ordering, and per-billing-period vault keys capped at expected total × 1.10.
ActiveMQ increments JMSXDeliveryCount on every redelivery — a developer who includes it in the Stripe idempotency key to “version” retry attempts produces a different key on each redelivery and Stripe creates ch_B, ch_C, up to maximumRedeliveries; a JMS transacted session rollback() re-delivers all messages in the batch — including those whose stripe.charges.create() already completed successfully — and a batch-start timestamp in the idempotency key changes on the rollback-replay pass producing ch_B for every customer charged in the first pass; and the ActiveMQ Dead Letter Queue admin retry (via web console, JMX, or activemq-admin CLI) re-enqueues the failed message with a new broker-assigned JMSMessageID — a developer whose idempotency key was derived from the original JMSMessageID now has a completely different key for the DLQ-retried message and Stripe creates ch_B for a customer who was charged during one of the six original delivery attempts. Three ActiveMQ-specific failure modes with Java JMS API code, content-hash idempotency keys that exclude all JMS session metadata (JMSXDeliveryCount, JMSMessageID, batch-start timestamp, session.hashCode(), JMSTimestamp, JMSExpiration), pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, write-before-acknowledge ordering, and per-billing-period vault keys capped at expected total × 1.10.
Apache Storm’s topology.message.timeout.secs (default 30 seconds) fires before stripe.charges.create() completes when Stripe P99 latency with retries exceeds the window — the root spout calls fail(messageId) and re-emits the original record; Storm assigns a new tuple ID to the replayed tuple; the billing bolt processes the replayed tuple and calls stripe.charges.create() a second time; an idempotency key derived from the Storm-assigned tuple ID (a new random 64-bit long per emission) produces a different key on replay and Stripe creates ch_B for the same customer and billing period; a JVM GC pause stops the worker heartbeat thread — the supervisor kills the worker process; all tuples in-flight on the killed worker are orphaned; the root spout re-emits them after topology.message.timeout.secs; if the billing bolt used a bolt-local set or dict as its dedup guard, the replacement worker starts with an empty in-memory store, the pre-flight check passes, and stripe.charges.create() is called again for customers already charged by the killed worker — within Stripe’s 24-hour idempotency window the content-hash key returns ch_A silently, but after 24 hours (topology offline for maintenance, Nimbus failover longer than one day, or a large replay queue that drains slowly), Stripe creates ch_B; and Storm Trident’s exactly-once processing guarantee is scoped specifically to TridentState updates (opaque-transactional or transactional commits managed by Trident’s batch coordinator) — it does not cover BaseFunction.execute() side effects like stripe.charges.create(), which runs on every attempt of the batch including replays; a developer who derives the Stripe idempotency key from Trident’s txid plus partition.index creates ch_B when partition assignment changes after a worker failure, because partition.index differs between the original attempt and the replay even though txid is stable. Three Storm-specific failure modes with Python (streamparse) and Java (Trident) code, content-hash idempotency keys that exclude all Storm metadata (tuple ID, task ID, Trident partition index, Trident offset-within-partition), pre-flight PostgreSQL checks with ON CONFLICT DO NOTHING, write-before-ack ordering, and per-billing-period vault keys capped at expected total × 1.10.
Firehose S3 delivery retry creates a second S3 object with the same billing records when the PutObject succeeds but the TCP acknowledgment is lost in transit — an S3 event notification fires for both objects, triggering two independent billing Lambda invocations that each call stripe.charges.create() for the same customers; Firehose-to-Redshift COPY command completes (rows inserted) but the Data API connection drops before Firehose receives the FINISHED status — a retry COPY inserts duplicate rows into the billing_events table and a billing job querying SELECT * FROM billing_events WHERE processed_at IS NULL processes both rows independently calling stripe.charges.create() twice; and the Firehose HTTP endpoint request body timestamp field is the server-side delivery-attempt timestamp that changes on every retry — a billing endpoint that constructs its Stripe idempotency key as sha256(timestamp + customer_id + billing_period) produces a different key on the Firehose retry and Stripe creates ch_B for the same customer and billing period (the requestId stays stable across retries and is designed as an idempotency token for batch-level dedup, but must not be used as a per-record Stripe idempotency key component because one requestId covers all records in the batch). Three Firehose-specific failure modes with Python code, content-hash idempotency keys derived exclusively from stable record-body fields (must not include S3 object key, UUID suffix, delivery timestamp, Firehose requestId, or Redshift row serial/bigserial), per-billing-period vault keys capped at expected total × 1.10, and a pre-flight PostgreSQL database check with ON CONFLICT DO NOTHING on (customer_id, billing_period) that closes all three failure modes regardless of Firehose destination type (S3, Redshift, HTTP endpoint), retry duration configuration, or buffer flush settings.
Kinesis KCL shard lease expiry during stripe.charges.create() — DynamoDB lease renewal returns ProvisionedThroughputExceededException under concurrent checkpoint writes from all shard workers during a billing run; the expired lease is acquired by a competing KCL worker that reads from the last checkpointed sequence number and independently calls stripe.charges.create() for every billing event processed since the last checkpoint, creating ch_B alongside ch_A that the original worker’s in-flight Stripe call completed; Enhanced Fan-Out SubscribeToShard consumer crash between stripe.charges.create() returning and the DynamoDB checkpoint write — re-subscription on restart begins from the last checkpointed sequence number, re-delivering records since the checkpoint to the billing callback which calls stripe.charges.create() again; and PutRecords partial-failure full-batch retry — a producer that retries all 500 records when FailedRecordCount > 0 re-writes successfully-accepted records to the stream; Kinesis assigns new sequence numbers to the retry copies; a billing consumer whose Stripe idempotency key includes the Kinesis SequenceNumber, ShardId, or ApproximateArrivalTimestamp produces a different key for each copy and Stripe creates ch_A and ch_B for the same customer and billing period. Three Kinesis-specific failure modes with boto3 Python code, content-hash idempotency keys derived from stable business fields only (must not include SequenceNumber, ShardId, or ApproximateArrivalTimestamp — all change on PutRecords retry copies), safe partial-batch retry producer, per-billing-period vault keys capped at expected total × 1.10, and a write-before-checkpoint pre-flight PostgreSQL database check that closes all three failure modes regardless of consumer type (polling vs. Enhanced Fan-Out), checkpoint frequency, or shard topology.
Kafka Streams’ at_least_once commit cycle flushes state store changelog records to Kafka before committing consumer group offsets — an OOM kill or max.poll.interval.ms session timeout between the changelog flush and the commitSync() causes task migration starting from the last committed consumer offset, re-delivering every billing event processed since the last offset commit to the new task’s process() method and calling stripe.charges.create() again for each one; processing.guarantee=exactly_once_v2 atomically commits output Kafka records and consumer offsets using Kafka producer transactions, but stripe.charges.create() executes inside process() before the transaction commits — a ProducerFencedException from a broker epoch bump or transaction coordinator failover aborts the transaction, re-delivers the input event, and calls stripe.charges.create() a second time for the same billing period; and DSL operations including groupBy() and selectKey() create internal repartition topics that act as intermediate Kafka topics with at_least_once delivery semantics — a crash in the upstream stage after writing a billing event to the internal repartition topic but before committing the source partition’s consumer offset causes the source event to be re-delivered and a second copy written to the repartition topic, which the downstream billing processor processes independently calling stripe.charges.create() for each copy with no cross-copy deduplication. Three Kafka Streams-specific failure modes with Java Processor API code, content-hash idempotency keys stable across task migrations and repartition topic duplicates (must not include Kafka partition, offset, task ID, or repartition topic offset — all change between duplicate copies), per-billing-period vault keys capped at expected total × 1.10, and a pre-flight PostgreSQL database check that closes all three failure modes regardless of processing.guarantee setting, DSL operation type, or Punctuator firing schedule.
Azure Blob Storage checkpoint write fails after stripe.charges.create() returns ch_A — StorageServiceError under Blob Storage throttling or OOM kill between Stripe returning and update_checkpoint() executing leaves the partition offset at the pre-charge position; the next EventProcessorClient to own the partition re-reads from the uncommitted offset, calls stripe.charges.create() for the same customer, and successfully checkpoints — ch_A and ch_B both appear on the customer’s Stripe account with no indication in Event Hubs metrics that the event was processed twice; the EventProcessorClient’s partition lease renewal thread calls BlobLeaseClient.lease.renew() on a background schedule, and if Blob Storage returns 429 TooManyRequests under storage account IOPS pressure (concurrent checkpoint writes from all partitions saturate the shared storage account), the blob lease expires and a competing EventProcessorClient instance acquires the expired lease mid-stripe.charges.create() — the competing instance starts from the last committed checkpoint, calls stripe.charges.create() for the same customer creating ch_B, and the first instance’s update_checkpoint() fails with ResourceModifiedError that many SDK defaults swallow; and EventHubProducerClient retries on network timeout deliver duplicate events with different sequence_numbers — the producer publishes a billing event, the server accepts it at sequence_number=1042 but the acknowledgment is lost in transit, the SDK retries and delivers a second copy at sequence_number=1043, and a processor that derives its Stripe idempotency key from sequence_number, offset, or enqueued_time from event.system_properties produces different keys for each copy and calls stripe.charges.create() twice — creating ch_A and ch_B for the same customer and billing period. Three Azure Event Hubs-specific failure modes with azure-eventhub Python code, content-hash idempotency keys stable across partition re-assignments, checkpoint recoveries, and producer retries (must not include sequence_number, offset, enqueued_time, or partition_id from event.system_properties — all change between producer retry copies), write-before-checkpoint patterns, per-billing-period vault keys capped at expected total × 1.10, and a pre-flight database check against PostgreSQL — which survives Blob Storage checkpoint failures — that closes all three failure modes regardless of Event Hubs tier, partition count, or checkpoint frequency.
XAUTOCLAIM reclaims a billing message from the Pending Entries List when its idle time exceeds min-idle-time — Redis Streams has no lock-renewal mechanism like SQS’s change_message_visibility or Azure Service Bus’s AutoLockRenewer, so idle time counts continuously from XREADGROUP delivery with no way for the processing consumer to extend its window; a monitoring job calling XAUTOCLAIM at T=60 claims a billing message that consumer-A has been processing since T=0, consumer-B calls stripe.charges.create() for the same customer, and XPENDING shows 0 after both consumers process the message — from Redis’s perspective the message was delivered once, from Stripe’s perspective the customer was charged twice; XREADGROUP with COUNT > 1 adds all batch messages to the PEL atomically — a consumer that reads 10 billing events, successfully calls stripe.charges.create() for customers 1–6, and receives stripe.error.Timeout on customer 7 leaves all 10 PEL entries unacknowledged (the exception interrupts the processing loop before per-event XACK is called), and when XAUTOCLAIM reclaims the idle batch, the new consumer re-bills customers 1–6 alongside 7–10; and appendfsync everysec — the default AOF persistence mode in managed Redis offerings including AWS ElastiCache, Google Cloud Memorystore, and Azure Cache for Redis — batches write commands and flushes to disk once per second, creating a 1-second window where an XACK written to the primary’s in-memory buffer is not flushed before primary crash; after failover, the replica that becomes primary has the PEL entry as still-pending, the next consumer receives the re-delivered message, calls stripe.charges.create() without a stable idempotency key, and creates ch_B alongside ch_A that Stripe processed before the Redis primary crashed. Three Redis Streams-specific failure modes with redis-py Python code, content-hash idempotency keys stable across XAUTOCLAIM redeliveries and AOF replay (must not include Redis message ID, delivery count from XPENDING, or consumer name from the PEL), COUNT=1 to eliminate batch partial-failure blast radius, per-billing-period vault keys capped at expected total × 1.10, and a write-before-ack pre-flight database check against PostgreSQL — which survives Redis failover — that closes the AOF data-loss mode regardless of appendfsync setting.
Azure Service Bus lock duration (default 60 seconds, max 5 minutes) expires while stripe.charges.create() is in-flight and AutoLockRenewer’s background renew_message_lock() call fails independently with ServerBusyError under namespace throttling — Service Bus releases the lock token and delivers the message to a competing consumer that calls stripe.charges.create() for the same customer, creating ch_B alongside ch_A that the first consumer’s Stripe call is still waiting to complete; dead-letter resubmission via Service Bus Explorer, Azure Portal, or SDK tooling constructs new ServiceBusMessage objects from dead-letter bodies and Azure Service Bus assigns new message_id UUIDs to each — a billing callback that derives its Stripe idempotency key from msg.message_id treats the resubmitted message as a new billing request and creates a new charge alongside the one that succeeded before MaxDeliveryCount was exhausted; and calling receiver.abandon_message(msg) on OperationTimeoutError or stripe.error.Timeout makes the message immediately available for redelivery with no configurable backoff at the abandon layer (unlike SQS’s visibility timeout or Pub/Sub’s retry policy) — the abandoned message is picked up within milliseconds by the next competing consumer, which calls stripe.charges.create() again and creates ch_B if Stripe created ch_A before the timeout response was lost. Three Service Bus-specific failure modes with azure-servicebus Python code, content-hash idempotency keys stable across lock expirations and DLQ resubmissions (must not include Service Bus message_id, lock_token, delivery_count, enqueued_time_utc, or sequence_number), per-billing-period vault keys capped at expected total × 1.10, and a write-before-complete pre-flight database check that closes the gap for DLQ resubmissions where Stripe’s 24-hour idempotency window has expired.
Pub/Sub’s ack deadline (default 10 seconds) starts counting the moment a subscriber receives a message — the Python client’s StreamingPullFuture extends it via a background lease manager thread, but if that thread’s ModifyAckDeadline RPC fails independently while the billing callback is blocked inside stripe.charges.create() (GCP rate limit, pod eviction, zone partition), Pub/Sub redelivers the message to a second subscriber that calls stripe.charges.create() for the same customer; exactly_once_delivery is a subscription-level attribute that prevents Pub/Sub from redelivering an acknowledged message, but a subscriber that crashes after Stripe creates ch_A and before message.ack() is called has never sent the acknowledgment — Pub/Sub redelivers regardless of the subscription setting; and calling message.nack() on stripe.error.Timeout makes the message immediately available for redelivery — if Stripe created ch_A server-side before the timeout response was lost, the redelivery calls stripe.charges.create() again with a new idempotency key, creating ch_B alongside ch_A. Three Pub/Sub-specific failure modes with google-cloud-pubsub Python code, content-hash idempotency keys stable across redeliveries (must not include Pub/Sub message_id, ack_id, delivery_attempt, or publish_time), per-billing-period vault keys capped at expected total × 1.10, and a write-before-ack pre-flight database check that closes the crash-between-stripe-and-ack gap regardless of how many times the message is redelivered.
SQS Standard Queue visibility timeout (default 30 seconds) starts counting the moment receive_message returns — if stripe.charges.create() takes longer than the timeout (Stripe P99 can exceed 30 seconds under load, plus stripe-python’s two retries push worst-case to 90+ seconds), SQS makes the message visible again and a second billing worker picks it up, calling stripe.charges.create() for the same customer before the first charge returns; a Lambda SQS event source mapping with BatchSize=10 and no ReportBatchItemFailures requeues the entire batch when any exception is raised — customers billed in positions 1–7 of the batch are re-charged when the Lambda retries all 10 messages due to a failure at position 8; and the Dead Letter Queue redrive via start_message_move_task re-fires stripe.charges.create() for messages where the charge succeeded server-side before a timeout response was lost — the worker received stripe.error.Timeout, did not delete the SQS message, and after MaxReceiveCount retries the message landed in the DLQ where an operator redrived it days later, outside Stripe’s 24-hour idempotency window. Three SQS-specific failure modes with boto3 Python code, a visibility timeout extension thread that calls change_message_visibility every 20 seconds to prevent expiry during long Stripe calls, content-hash idempotency keys stable across redeliveries (must not include SQS MessageId, ReceiptHandle, ApproximateReceiveCount, or ApproximateFirstReceiveTimestamp), per-billing-period vault keys capped at expected total × 1.10, and a pre-flight database check that closes the gap for DLQ redrives where Stripe’s 24-hour deduplication window has expired.
Faust commits Kafka offsets on a configurable timer (commit_interval, default 3 seconds) rather than per-message — a Kafka partition rebalance triggered by a Kubernetes rolling deploy during the stripe.charges.create() HTTP call starts the new partition owner from the last committed offset, which predates the in-flight Stripe call, re-delivering the same billing event and creating ch_B alongside ch_A; Faust Tables are rebuilt from their Kafka changelog topic on agent restart, but if the worker crashes after calling Stripe and before writing “billed” to the Table, the changelog does not have the update — after restart the Table shows the customer as pending and the next processing pass re-charges them; and Faust @app.cron and @app.timer tasks fire on every running worker replica simultaneously, with no built-in leader election — a three-replica Kubernetes deployment creates three charges per customer per billing period. Three Faust-specific failure modes with Python agent code, content-hash idempotency keys stable across Kafka rebalances and offset resets (must not include Kafka offset, consumer group ID, or Faust worker node name), a Redis distributed lock for multi-replica cron coordination, per-billing-period vault keys capped at expected total × 1.10, and a pre-flight database check against billing_records (not the Faust Table) as the authoritative billing guard — the two-layer governance pattern that makes Faust billing agents safe to redeploy, rebalance, and scale horizontally.
NATS JetStream’s AckWait timer expires while the stripe.charges.create() HTTP call is still in-flight — NATS makes the message available for the next Fetch() call, a second billing worker processes the same usage event and creates ch_B before the first Stripe call returns ch_A; a NATS Core direct subscription without a queue group delivers every usage event to all connected billing worker pods simultaneously (not just one), each pod calls stripe.charges.create() independently — three workers create three charges for the same customer in the same billing period with nothing in NATS’s monitoring output distinguishing two-event fan-out from duplicate-event fan-out; and deleting a durable JetStream consumer loses its delivery sequence — recreating it with the default DeliverPolicy.ALL starts from stream sequence 1, reprocessing every billing event retained in the stream’s window and re-charging customers billed in prior periods. Three NATS-specific failure modes with nats-py Python consumer code, content-hash idempotency keys stable across AckWait redeliveries (must not include NATS sequence number, delivery timestamp, or AckWait retry count), per-billing-period vault keys capped at expected total × 1.10, a unique-constraint-backed pre-flight DB check that closes the concurrent subscriber race condition, and the safe consumer configuration that combines 120s AckWait with DeliverPolicy.NEW to prevent consumer recreation replays.
RabbitMQ’s prefetch window (basic_qos prefetch_count) means the consumer holds up to N messages in its unacknowledged buffer — a channel or TCP connection close after stripe.charges.create() calls but before basic_ack requeues all N messages, and the next consumer re-charges those customers without any signal from RabbitMQ that they were already billed; the dead letter exchange retry loop re-fires billing when the billing function sends basic_nack(requeue=False) on stripe.error.Timeout, routing the message to the DLX which re-publishes it to the billing exchange after a TTL delay — the retry calls stripe.charges.create() again for a charge that was created server-side before the timeout response was lost; and AMQP connection-level multiplexing means a single TCP connection drop requeues in-flight messages across all channels on that connection simultaneously — a billing consumer running four channels with prefetch_count=25 per channel has up to 100 messages requeued when the TCP connection drops, not 25. Three RabbitMQ-specific failure modes with pika Python code, content-hash idempotency keys stable across redeliveries (must not include delivery tag, x-death count, or DLX routing key suffix), per-billing-period vault keys capped at expected total × 1.10, and a pre-flight database check that closes the gap for redeliveries beyond Stripe’s 24-hour idempotency window — the two-layer governance pattern that makes RabbitMQ billing consumers safe to restart, evict, and DLX-retry.
Apache Pulsar’s cumulative acknowledgment mode means the billing consumer sends a single ack for the entire batch after all stripe.charges.create() calls complete — a crash or eviction between the last Stripe call and the cumulative ack leaves all 500 messages unacknowledged, and the broker redelivers the entire batch to the restarted consumer which charges all 500 customers again; a Shared subscription with multiple billing consumer pods redistributes unacknowledged in-flight messages to surviving pods when a consumer disconnects mid-batch (Kubernetes eviction, spot interruption, rolling deploy), causing surviving pods to re-receive and re-charge messages already processed by the disconnected pod; and Pulsar’s dead letter topic retry mechanism re-fires the billing Lambda when the billing function nacks a message after catching a stripe.error.Timeout, even though the charge was created server-side before the timeout response was lost — the retry creates ch_B alongside ch_A for the same customer. Three Pulsar-specific failure modes with consumer configuration, content-hash idempotency keys stable across Pulsar redeliveries (must not include Pulsar message sequence ID, publish timestamp, or redelivery count), per-billing-period vault keys capped at expected total × 1.10, and a pre-flight database check that closes the gap for redeliveries beyond Stripe’s 24-hour idempotency window — the two-layer governance pattern that makes Pulsar billing consumers safe to restart, rebalance, and retry.
Monte Carlo’s mid-run circuit breaker polls for open incidents and exits cleanly when one is detected — but does not commit partial-progress state before exiting, so the billing retry re-starts from the beginning of the customer list and re-charges customers already billed in the partial first run; a Monte Carlo incident-resolution webhook wired directly to the billing Lambda trigger fires concurrently with the daily scheduled billing trigger when the two events coincide within the same billing period, launching two independent billing runs that both charge all customers without coordination; and a dbt model JOIN fan-out that multiplies usage rows via an unintentional subscription_tiers join inflates the billing dataset in a way that falls within Monte Carlo’s row-count anomaly detection band — letting duplicate rows through to the billing function which creates two Stripe charges per customer in a single run. Three Monte Carlo-specific failure modes with Monte Carlo API usage, a custom SQL rule monitor for fan-out detection, content-hash idempotency keys, per-billing-period vault keys capped at expected total × 1.10, a PostgreSQL advisory lock that blocks concurrent webhook + schedule invocations, and a pre-flight database check.
Meltano’s Singer state is written to the state backend only after a pipeline run exits cleanly with code 0 — a failed run that already emitted 900 records to the target discards its state, so the next meltano run starts from the last saved bookmark and re-emits all 900 records, which the downstream billing function treats as new usage events and charges again; concurrent meltano run invocations (a scheduled run overlapping a manual replay, or Meltano’s --concurrent mode) both read the same initial Singer state bookmark, both start the tap at the same watermark, both emit the same unbilled usage records to independent target instances, and both call stripe.charges.create() independently — every customer billed twice, with nothing in Meltano’s job log indicating a problem; and meltano run --no-state-update disables Singer state read and write entirely, causing the tap to start from its historical origin on every invocation — a pattern intended for CI that, when run with production .env credentials, charges every historical customer from the beginning of the source dataset. Three Meltano-specific failure modes with Meltano YAML, Singer tap Python code, a content-hash idempotency key stable across Singer retries (must not include _sdc_received_at, _sdc_sequence, or _sdc_batched_at), per-billing-period vault keys capped at expected total × 1.10, a PostgreSQL advisory lock that blocks concurrent billing runs from starting simultaneously, and a pre-flight database check that closes the gap for replays older than Stripe’s 24-hour idempotency window — the two-layer governance pattern that makes Meltano billing pipelines safe to retry, run concurrently, and redeploy.
Airbyte’s incremental sync saves cursor state only on job success — a normalization failure after raw data is written causes the next sync to retry from the last saved cursor, re-emitting all 1,200 rows with new _airbyte_raw_id values that bypass dedup logic based on that field; a connection reset via the UI, API, or automatic schema-change propagation clears Airbyte’s cursor state and triggers a full historical replay, re-delivering every usage record ever emitted to the billing webhook, which fires Stripe charges for all historical customers in minutes; and PyAirbyte’s get_pandas_dataset() rebuilds from the connector’s full history on every cache miss (new container, cleared /tmp, serverless cold start), returning all historical rows to an agent pipeline that calls stripe.charges.create() for each one. Three Airbyte-specific failure modes with pyairbyte and Python code, a content-hash idempotency key stable across Airbyte retries (must not include _airbyte_raw_id or _airbyte_extracted_at), per-billing-period vault keys capped at expected total × 1.10, and a pre-flight database check that closes the gap for replays older than Stripe’s 24-hour idempotency window — the two-layer governance pattern that makes Airbyte billing pipelines safe to retry, reset, and redeploy.
Kafka’s at-least-once delivery means a consumer crash between stripe.charges.create() and the offset commit causes the same billing event to be reprocessed on restart — ch_A succeeded before the crash but the offset was never committed, so the new consumer creates ch_B for the same customer and billing period; max_poll_interval_ms exceeded during sequential Stripe calls over a large billing batch evicts the consumer from the group mid-charge, reassigns the partition to a new consumer that starts from the last committed offset — charging all the customers the evicted consumer had already processed; and a consumer group offset reset via kafka-consumer-groups.sh --reset-offsets --to-earliest, or a new group.id with auto_offset_reset=earliest, replays all billing events in the retention window, charging every customer in that window a second time. Three Kafka-specific failure modes with kafka-python code, a content-hash idempotency key that is stable across partition reassignments and consumer restarts, per-billing-period vault keys capped at expected total × 1.10, a pre-flight database check that closes the gap beyond Stripe’s 24-hour idempotency window, and manual offset commit after the database write — the two-layer governance pattern that makes Kafka billing consumers safe to restart, redeploy, and rebalance.
Pachyderm’s datum_tries re-runs the billing container from line 1 after stripe.charges.create() succeeded and the downstream write failed — the retry container re-executes the billing script from scratch, calling Stripe again without an idempotency key; parallelism_spec.constant: N spawns N concurrent worker pods that all receive the same unrestricted STRIPE_SECRET_KEY from a Kubernetes Secret with no per-pod spend cap — a data error in amount_cents propagates to all N customer datums simultaneously before any result is written to the output repository; and pachctl run pipeline triggered by an operator alongside a cron-scheduled job creates two concurrent jobs over the same input commit, where Pachyderm’s datum cache only protects completed datums — in-progress datums from the first job are re-scheduled by the second, charging those customers twice. Three Pachyderm-specific failure modes with pipeline specification JSON, Python transform container code, a per-job vault key issued by an upstream pipeline step, and a per-datum advisory lock in PostgreSQL — the two-layer governance pattern that closes each one.
Prefect’s @task(retries=N) re-executes the entire billing callable from line 1 after any downstream exception — if stripe.charges.create() succeeded but the database write raised a psycopg2.OperationalError, the retry fires a second Stripe charge with no idempotency key; .map() with ConcurrentTaskRunner dispatches N billing task instances simultaneously in the same Python process, all sharing one module-level stripe.api_key with no per-task spend cap — a data error in amount_cents propagates to all N customer batches before any result returns; and Prefect Cloud Automations trigger flow runs independently of manual deployment runs with no at-most-one-per-billing-period gate — an on-call engineer who triggers a manual run after a partial failure creates two concurrent flow runs that both execute the full billing task map. Three Prefect-specific failure modes with Prefect Python code, a Redis distributed billing-period lock, the per-flow vault key spend cap, and content-hash idempotency keys — the two-layer governance pattern that closes each one.
Snowflake Tasks with ALLOW_OVERLAPPING_EXECUTION = TRUE let two billing stored procedure instances run simultaneously — both read the same unbilled customer set from billing_queue and both call stripe.charges.create() independently, charging every customer twice; Snowflake serverless compute auto-suspension terminates a mid-run billing stored procedure after the cluster's idle window elapses, and the next scheduled task run restarts the procedure from row 1 because the terminated run never committed per-row billing results; and in a Task DAG where a root task builds the billing queue and triggers child billing tasks, re-triggering the root task after a child failure re-triggers all children — including the ones that previously succeeded and already called Stripe for their customer subset. Three Snowflake-specific failure modes with Snowpark Python stored procedure code, advisory locking with billing_run_locks, MERGE instead of INSERT OVERWRITE, and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
dbt Python model re-runs re-execute every stripe.charges.create() call in the model because dbt has no partial-checkpoint mechanism — on failure the entire Python model function reruns from the first customer row, re-charging everyone processed in the first run; dbt post-hooks configured to trigger a billing Lambda fire again on model re-runs, giving the Lambda two invocations for the same billing period with no built-in deduplication between them; and dbt run --full-refresh rebuilds incremental billing-queue models from scratch, causing customers already billed in the current period to reappear in the queue as if they were unbilled. Three dbt-specific failure modes with dbt Python code and YAML, the DynamoDB billing-period lock and per-customer pre-flight check, and the two-layer governance pattern — content-hash idempotency keys plus per-period vault keys via a spend-cap proxy — that closes each one.
Dask worker death on spot or preemptible instances causes the scheduler to retry billing tasks that already called stripe.charges.create() before the worker stopped responding — the retry worker calls Stripe again without any knowledge of the first charge; a dask.delayed billing graph has no checkpoint mechanism, so calling .compute() again after a downstream database write failure re-executes every Stripe charge from scratch; and Client.submit(retries=N) resubmits billing task functions that called Stripe successfully but raised a downstream exception — the resubmitted function calls Stripe again on the new execution. Three Dask-specific failure modes with Dask Distributed Python code, the DynamoDB pre-flight check and per-customer write pattern, and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
AWS Glue's MaxRetries re-runs the entire billing script from line 1 after stripe.charges.create() succeeded and a downstream DynamoDB or Redshift write failed — the retry creates new charges for every customer without idempotency keys; a Glue Scheduled Trigger backed by EventBridge Scheduler has at-least-once delivery semantics and can fire two concurrent job runs for the same billing period, both executing the billing loop independently and both succeeding with no duplication flag in Glue's run history; and Glue Job Bookmark state is committed only on clean job success — if the job fails after charging Stripe but before the bookmark advances, the next run re-reads the same S3 files and re-charges every customer from the failed run. Three Glue-specific failure modes with job Python code, the DynamoDB conditional-write billing lock, and the two-layer governance pattern — content-hash idempotency keys plus per-job vault keys via a spend-cap proxy — that closes each one.
NiFi's failure-relationship retry loop re-invokes InvokeHTTP against the Stripe charges endpoint after a successful charge when only the downstream PutDatabaseRecord failed — the retried FlowFile has no idempotency key and creates a second charge; Primary Node failover during ZooKeeper session loss causes the newly elected Primary Node to re-trigger the billing GenerateFlowFile cron for the same period while the old node's in-flight FlowFiles are still in the connection queue; and two connections drawn from the same upstream relationship to two InvokeHTTP processors load-balance billing FlowFiles across both Stripe calls, charging half of each customer cohort twice. Three NiFi-specific failure modes with flow XML configuration and ExecuteScript Python billing code, and the two-layer governance pattern — content-hash idempotency keys plus per-FlowFile vault keys via a spend-cap proxy — that closes each one.
Databricks notebook task max_retries re-runs the entire notebook from cell 1 after stripe.charges.create() succeeded in cell N and a downstream Delta Lake write raised a schema mismatch or PermissionError — each retry creates a new charge without an idempotency key; for_each_task with a duplicate in the input list dispatches two concurrent billing task instances for the same customer, both independently calling stripe.charges.create() with no cross-instance coordination; and max_concurrent_runs set above 1 allows a second scheduled billing run to start while the first is still executing when the run exceeds the cron interval, sending two runs through the same customer cohort simultaneously. Three Databricks-specific failure modes with Jobs YAML and Python billing code, and the two-layer governance pattern — content-hash idempotency keys plus per-task vault keys via a spend-cap proxy — that closes each one.
AWS Batch's retryStrategy.attempts re-runs the billing container from line 1 after stripe.charges.create() succeeded and a downstream DynamoDB write failed — each retry attempt creates a new charge without an idempotency key; Array Job fan-out dispatches N concurrent child containers that all receive the same unrestricted STRIPE_SECRET_KEY from Secrets Manager with no per-child spend cap, so a data error in amount_cents charges all N customers simultaneously; and EventBridge Scheduler at-least-once delivery can submit two concurrent SubmitJob API calls for the same billing period — both Batch jobs execute, both call stripe.charges.create() independently, and Batch shows two SUCCEEDED statuses with no sign of duplication. Three AWS Batch-specific failure modes with job definition YAML, Lambda dispatcher code, and Python billing container code, and the two-layer governance pattern — content-hash idempotency keys plus per-child vault keys via a spend-cap proxy — that closes each one.
Celery Beat's singleton requirement is violated during Kubernetes rolling updates — two Beat pods run with separate PersistentScheduler SQLite files and both dispatch the billing task to the broker simultaneously; PersistentScheduler fires one catch-up dispatch per missed billing interval on Beat restart after downtime, so a two-hour Beat outage with an hourly schedule fires two concurrent billing executions immediately on restart; and a manual billing_task.apply_async() call alongside the Beat-scheduled task creates two task messages with different IDs that workers dequeue and execute independently — both charging all customers for the same period. Three Celery Beat-specific failure modes with Beat configuration, Kubernetes deployment YAML, and Python task code, and the two-layer governance pattern — content-hash idempotency keys plus per-task vault keys via a spend-cap proxy — that closes each one.
Kubernetes CronJob's default concurrencyPolicy: Allow fires two overlapping billing Job instances for the same period when a run exceeds the cron interval — both Jobs call stripe.charges.create() independently with no dedup; Job backoffLimit creates a new billing Pod from line 1 after a downstream write failure that follows a successful Stripe charge, re-charging the same customers up to backoffLimit times; and an unset startingDeadlineSeconds causes the CronJob controller to fire one catch-up Job per missed schedule when it recovers from a control-plane outage, generating three simultaneous billing runs after a three-hour outage. Three Kubernetes-specific failure modes with CronJob YAML and Python billing container code, and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys issued by an init container — that closes each one.
GCP Workflows' try/retry block re-invokes the Cloud Function step from the beginning after any exception — a Firestore write failure after a successful stripe.charges.create() triggers a second charge with no idempotency protection; a parallel for loop dispatches N concurrent iterations that all resolve the same unrestricted Secret Manager secret with no per-iteration spend cap, so a data error in amount_cents charges all customers the wrong amount before any result returns; and Cloud Scheduler at-least-once delivery can fire two concurrent Workflows executions for the same billing period, each proceeding independently through the fan-out and Stripe charge steps. Three Google Cloud Workflows-specific failure modes with Workflows YAML and Python Cloud Functions code, and the two-layer governance pattern — content-hash idempotency keys plus per-iteration vault keys via a spend-cap proxy — that closes each one.
SageMaker Pipelines RetryPolicy re-invokes the Processing Job container from line 1 after stripe.charges.create() has already succeeded — a DynamoDB write failure triggers a second charge with no idempotency protection; ParallelStep fan-out dispatches N concurrent Processing Job instances that all pull the same unrestricted STRIPE_SECRET_KEY from SSM with no per-cohort spend cap, so a data error in amount_cents propagates to the full customer cohort simultaneously; and EventBridge at-least-once delivery can trigger two concurrent StartPipelineExecution calls with identical billing inputs — both executions run to completion and both bill the same customers, with SageMaker showing two Succeeded statuses and no sign of the duplicate. Three SageMaker-specific failure modes with Python SageMaker SDK code, and the two-layer governance pattern — content-hash idempotency keys plus per-execution vault keys via a spend-cap proxy — that closes each one.
Huey's retries=N decorator re-executes the billing callable from line 1 after stripe.charges.create() succeeded — a database write failure triggers a second charge with no idempotency protection; a @huey.periodic_task running alongside a manually-dispatched task.schedule() for the same billing period creates two independent task instances that both reach Stripe with no built-in dedup; and immediate=True (Huey's eager mode) accidentally set in production executes billing synchronously in the request thread, where ALB or nginx upstream retry middleware fires a second charge. Three Huey-specific failure modes with Python task code, and the two-layer governance pattern — content-hash idempotency keys plus per-task vault keys via a spend-cap proxy — that closes each one.
Taskiq's asyncio-native retry re-executes the full billing task callable from line 1 after stripe.charges.create() has already succeeded — when a downstream database write raises, Taskiq re-enqueues the task with the same arguments and a second charge fires without an idempotency key; concurrent asyncio workers share one unrestricted STRIPE_SECRET_KEY from os.environ across all parallel billing task executions with no per-task spend cap, so a data error in amount_cents propagates to the full cohort before any worker surfaces a failure; and TaskiqScheduler cron combined with a manual task.kiq() dispatch for the same billing period creates two independent task messages with different task IDs that workers dequeue and execute independently with no built-in deduplication. Three Taskiq-specific failure modes with async Python task code, and the two-layer governance pattern — content-hash idempotency keys plus per-task vault keys via a spend-cap proxy — that closes each one.
APScheduler's BackgroundScheduler embedded in a multi-worker WSGI app fires one billing execution per worker process — with 4 Gunicorn workers every customer is charged 4 times, because each worker creates its own scheduler instance with no cross-process coordination; MemoryJobStore state loss on process restart causes APScheduler to fire a catch-up billing execution against customers already charged in the previous run, because the scheduler has no record that the job succeeded; and coalesce=False on a billing job fires one billing execution per missed scheduled interval on recovery — 3 hours of scheduler downtime with an hourly cron fires 3 concurrent billing runs for the same customer cohort. Three APScheduler-specific failure modes with Python scheduler code, and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Dramatiq's Retries middleware re-invokes the actor callable from line 1 after stripe.charges.create() has already succeeded — when a downstream database write raises, the middleware re-enqueues the original message with the same arguments and a second charge fires without an idempotency key; group().run() fan-out dispatches N concurrent actor instances that all share one module-level stripe.api_key = os.environ["STRIPE_SECRET_KEY"] with no per-actor spend cap, so a unit error in amount_cents charges all N customers simultaneously before any worker surfaces a failure; and APScheduler's actor.send() cron combined with a manual actor.send() call for the same billing period creates two messages with different UUIDs that Dramatiq dequeues independently — no built-in dedup between scheduled and manual dispatch paths. Three Dramatiq-specific failure modes with Python actor code, and the two-layer governance pattern — content-hash idempotency keys plus per-batch vault keys via a spend-cap proxy — that closes each one.
RQ's Retry class re-queues the job function from line 1 after stripe.charges.create() has already succeeded — when a downstream database write raises, RQ re-enqueues the billing job and a second charge fires without an idempotency key; multiple rq worker processes share one unrestricted os.environ['STRIPE_SECRET_KEY'] across all concurrently dequeued jobs with no per-job spend isolation, so a unit error in amount_cents propagates to the full concurrent wave simultaneously; and rq-scheduler's recurring cron combined with a manual q.enqueue() call for the same billing period creates two independent job instances with no built-in deduplication between the two invocation paths. Three RQ-specific failure modes with Python job code, and the two-layer governance pattern — content-hash idempotency keys plus per-batch vault keys via a spend-cap proxy — that closes each one.
BullMQ job retry re-executes the entire processor function from line 1 after stripe.charges.create() has already succeeded — when a downstream database write throws, BullMQ re-invokes the billing processor and a second charge fires without an idempotency key; worker concurrency: N dispatches N simultaneous job executions that all share one unrestricted process.env.STRIPE_SECRET_KEY with no per-job spend isolation, so a data error in amountCents propagates across the full concurrent wave simultaneously; and BullMQ's repeat cron combined with a manual queue.add() call creates two independent job instances for the same billing period, both executing billing logic with no built-in deduplication between invocation paths. Three BullMQ-specific failure modes with TypeScript processor code, and the two-layer governance pattern — content-hash idempotency keys plus per-batch vault keys via a spend-cap proxy — that closes each one.
Luigi task retries re-invoke run() from line 1 after stripe.charges.create() has already succeeded — when a downstream database write fails, Luigi re-executes the entire billing task and a second charge fires; yield-based dynamic fan-out dispatches N parallel BillingTask instances that all share one unrestricted STRIPE_SECRET_KEY with no per-task spend cap; and output target deletion — from S3 lifecycle rules, DBA table truncation, or cleanup scripts — causes Luigi to re-run billing tasks that already created Stripe charges, even months later. Three Luigi-specific failure modes with Python task code, and the three-layer governance pattern — content-hash idempotency keys, pre-flight database checks, and per-pipeline vault keys via a spend-cap proxy — that closes each one.
Mage block retries re-execute the billing transformer from line 1 after stripe.charges.create() has already succeeded — when a downstream export block fails, Mage re-runs the entire billing block and a second charge fires; scheduled and API-triggered pipeline runs execute independently for the same billing period with no cross-run dedup, so a concurrent manual trigger creates duplicate charges; and backfills launch N parallel pipeline runs that simultaneously charge customers with no cross-run idempotency. Three Mage AI failure modes with Python block code and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Conductor task retries re-invoke workers from line 1 after stripe.charges.create() has already succeeded — when a worker dies between the Stripe call and the COMPLETED acknowledgment, Conductor re-queues the task and a second charge fires; FORK_JOIN parallel tasks dispatch N billing workers that all share one unrestricted STRIPE_SECRET_KEY with no per-task spend cap; and POST /workflow/{id}/restart re-executes every task including COMPLETED billing tasks, bypassing Conductor's own retry safety. Three Orkes Conductor failure modes with Python SDK code and the two-layer governance pattern that closes each one.
Spark's fault-tolerance model re-schedules failed tasks on other executors — any stripe.charges.create() call inside a partition function re-executes from record zero on the new executor without idempotency keys; foreachPartition() broadcasts STRIPE_SECRET_KEY to all concurrent executors with no per-partition dollar cap, so a data error in amount_cents propagates to every partition simultaneously before any errors surface; and Structured Streaming's checkpoint-based recovery replays foreachBatch() for the same micro-batch records when a query recovers from failure. Three PySpark failure modes with code examples — batch jobs, speculative execution, and streaming — and the two-layer governance pattern that closes each one.
Restate's ctx.run() retries the side-effect lambda when it raises an exception — if stripe.charges.create() fires but a network glitch prevents the response from arriving, the Python SDK raises APIConnectionError, Restate retries the ctx.run() block, and a second charge is created without an idempotency key; fan-out via parallel service calls dispatches N concurrent billing handlers that all share one unrestricted STRIPE_SECRET_KEY with no per-handler dollar cap; and Stripe calls made directly in the handler body outside a ctx.run() block re-execute on every journal replay because Restate only skips journaled side effects. Three Restate failure modes with Python SDK code, and the two-layer governance pattern — content-hash idempotency keys plus per-handler vault keys via a spend-cap proxy — that closes each one.
Flink's checkpoint-based recovery replays all records between the last checkpoint and the point of failure through every downstream operator — including billing ProcessFunction operators that call stripe.charges.create() — without any idempotency protection by default; AsyncDataStream.unorderedWait() dispatches N concurrent Stripe calls from parallel subtask instances that all share one unrestricted STRIPE_SECRET_KEY with no per-subtask dollar cap; and a savepoint restore triggered by an upgrade or rescaling re-executes all billing events processed since the savepoint was taken. Three Apache Flink failure modes with Python DataStream API code, and the two-layer governance pattern — content-hash idempotency keys plus per-pipeline vault keys — that closes each one.
Durable Functions' RetryOptions re-invokes an activity function from line 1 when any exception is raised — including after stripe.charges.create() already succeeded and a downstream Cosmos DB write failed; Task.WhenAll fan-out distributes billing across N concurrent activity executions that all share one unrestricted STRIPE_SECRET_KEY with no per-item dollar cap; and ContinueAsNew-based eternal orchestrations or overlapping TimerTrigger cron functions can launch two billing runs for the same period during deployment slot swaps. Three Azure Durable Functions failure modes with Python activity code, and the two-layer governance pattern — content-hash idempotency keys plus per-fan-out vault keys — that closes each one.
Letta's agent step retry re-executes billing tools when a downstream exception removes in-context evidence of a completed charge — the LLM sees no proof that ch_A was created and calls charge_customer again; multi-agent network fan-out dispatches N worker agents that all share one unrestricted STRIPE_SECRET_KEY with no per-agent spend cap; and long-running sessions evict charge completion records from recall memory, causing the agent to replay billing actions it already executed. Three Letta-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-agent vault keys — that closes each one.
Step Functions' Retry configuration re-invokes the Lambda function from line 1 when any error is thrown — including after stripe.charges.create() already succeeded and a downstream DynamoDB write failed; the Map state distributes items across concurrent Lambda invocations that all share one unrestricted STRIPE_SECRET_KEY from the environment with no per-item dollar cap; and EventBridge at-least-once delivery or concurrent StartExecution calls can launch two Standard Workflow executions for the same billing period. Three AWS Step Functions failure modes with Python Lambda code and state machine JSON, and the two-layer governance pattern — content-hash idempotency keys plus per-item vault keys via a spend-cap proxy — that closes each one.
Trigger.dev's maxAttempts setting retries the entire task run from line 1 on any unhandled exception — including exceptions thrown by downstream steps after stripe.paymentIntents.create() already succeeded; batch.triggerAndWait() fans out to N parallel task runs all sharing one unrestricted STRIPE_SECRET_KEY with no per-task spend cap; and a schedules.task() without concurrencyLimit: 1 fires overlapping billing runs for the same period when a manual trigger fires during an active scheduled run. Three Trigger.dev-specific failure modes with TypeScript code, and the two-layer governance pattern — content-hash idempotency keys plus per-task vault keys via a spend-cap proxy — that closes each one.
Dagger's layer cache is bypassed when CI uses --no-cache or a cold runner starts without a warm cache — silently re-executing billing steps developers assumed were deduplicated by Dagger's input-addressed caching; a parallel container fan-out via asyncio.gather() shares one unrestricted dag.set_secret() value across all concurrent instances with no per-invocation spend cap; and a CI job retry after a downstream database write failure re-executes the billing function from line 1 on a fresh runner. Three Dagger-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-invocation vault keys via a spend-cap proxy — that closes each one.
Kedro's SequentialRunner pipeline re-run after a downstream node failure re-executes the billing node from the start when any node raises after stripe.charges.create() already succeeded; ParallelRunner concurrent namespaced pipeline instances share one process-level stripe.api_key global with no per-cohort spend cap; and an on_node_error Hook implementing automatic node-level retry re-invokes the billing node's callable directly without a stable idempotency key after a transient timeout. Three Kedro-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-cohort vault keys via a spend-cap proxy — that closes each one.
Hamilton's driver retry loop re-executes every node in the computation graph — including the billing function — from the start when any downstream node fails after stripe.charges.create() already succeeded; Parallelizable[T] and Collect[T] fan-out dispatches N concurrent billing tasks across a thread pool all sharing one unrestricted Stripe key with no per-task spend cap; and a NodeExecutionHook lifecycle adapter that retries transient errors fires a fresh stripe.charges.create() call without an idempotency key each time the hook decides to retry. Three Hamilton-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-execution vault keys — that closes each one.
GitHub Actions "re-run failed jobs" re-executes the billing job from step 1 when any downstream step fails after stripe.charges.create() already succeeded; strategy.matrix launches N parallel runner processes all sharing one unrestricted STRIPE_SECRET_KEY secret with no per-matrix-entry spend cap; and an on.schedule workflow without a concurrency group allows workflow_dispatch to fire a second billing run while the first is still executing. Three GitHub Actions-specific failure modes with Python and YAML code, and the two-layer governance pattern — content-hash idempotency keys plus per-job vault keys via a spend-cap proxy — that closes each one.
Kubeflow Pipelines retries a failed component pod from line 1 — including after stripe.charges.create() already returned a charge ID and only the downstream database write failed; dsl.ParallelFor dispatches N concurrent billing pods all inheriting the same unrestricted Stripe key from a Kubernetes Secret with no per-customer spend cap; and a recurring run without max_concurrency=1 launches a second billing pipeline while the first is still executing, charging the same customers twice. Three Kubeflow-specific failure modes with Python and KFP SDK v2 code, and the two-layer governance pattern — content-hash idempotency keys plus per-item vault keys via a spend-cap proxy — that closes each one.
Argo Workflows' retryStrategy re-executes the entire billing container from line 1 when any step exits non-zero — including after stripe.charges.create() already succeeded and only the downstream database write failed; withItems and withParam fan-out runs N concurrent container instances all sharing the same STRIPE_SECRET_KEY with no per-item spend cap; and CronWorkflow without concurrencyPolicy: Forbid allows two overlapping billing runs to charge the same customers simultaneously. Three Argo-specific failure modes with Python and YAML code, and the two-layer governance pattern — content-hash idempotency keys plus per-item vault keys via a spend-cap proxy — that closes each one.
Apache Beam's DoFn retry mechanism re-executes the entire process() method from line 1 when any exception is raised — including after stripe.charges.create() already returned a charge ID; a bundle-level worker failure on Dataflow or Spark replays every element in the bundle on a new worker, re-firing Stripe calls for customers that were already charged; and streaming pipelines with PubSub sources redeliver unacked messages after a checkpoint failure, causing the same billing trigger to execute twice. Three Beam-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-pipeline-run vault keys via a spend-cap proxy — that closes each one.
BentoML's async task retry re-executes the entire handler function from line 1 when any exception is raised — including after stripe.charges.create() already returned a charge ID; multiple worker processes spawned by the same Service share one unrestricted Stripe key with no per-request deduplication, so concurrent billing calls can each create an independent charge for the same customer; and a Service restart or auto-scaling event re-routes in-flight billing requests to a fresh worker that calls stripe.charges.create() again without knowledge of the previous successful charge. Three BentoML-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-request vault keys via a spend-cap proxy — that closes each one.
Metaflow's @retry decorator re-executes the entire step function from line 1 when any exception is raised — including after stripe.charges.create() already returned a charge ID; foreach parallel branches all share one unrestricted Stripe key with no per-branch spend cap, so a billing bug hits the entire customer cohort simultaneously; and the resume command re-runs a billing step that already successfully charged customers, creating a duplicate charge with no warning in Metaflow's dashboard. Three Metaflow-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-branch vault keys via a spend-cap proxy — that closes each one.
ZenML's StepRetryConfig re-executes the entire step function from line 1 when any exception is raised — including after stripe.charges.create() already returned a charge ID; parallel pipeline steps running concurrently share one unrestricted Stripe key with no per-step spend cap, so a billing bug hits the entire customer cohort simultaneously; and step cache invalidation silently re-runs billing steps that have already charged customers, creating duplicate charges with no warning. Three ZenML-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-step vault keys via a spend-cap proxy — that closes each one.
Flyte's @task(retries=N) decorator re-executes the entire task function from line 1 when any exception is raised — including after stripe.charges.create() already returned a charge ID; map_task() fanout dispatches N concurrent instances all sharing one unrestricted Stripe key with no per-customer spend cap; and workflow re-execution from a failed downstream node re-fires an already-completed billing task, creating a duplicate charge. Three Flyte-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-task vault keys via a spend-cap proxy — that closes each one.
Kestra's task retry block re-executes the entire Python Script from line 1 when any exception is raised — including after stripe.charges.create() already returned a charge ID; EachParallel tasks share one unrestricted Stripe key across all concurrent executions with no per-item spend cap, amplifying any billing bug to the entire customer cohort simultaneously; and a Schedule trigger without concurrencyLimit: 1 fires a second billing run before the first finishes, charging the same customers twice. Three Kestra-specific failure modes with YAML and Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-execution vault keys via a spend-cap proxy — that closes each one.
Hatchet's @hatchet.step(retries=N) decorator re-executes the entire step function from line 1 when any exception is raised — including after stripe.charges.create() already returned a charge ID; concurrent child workflows spawned via context.spawn_workflow share one unrestricted Stripe key across all parallel executions with no per-child spend cap; and a cron billing workflow without a concurrency guard spawns a second overlapping instance before the first finishes, billing the same customers twice. Three Hatchet-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-step vault keys via a spend-cap proxy — that closes each one.
Ray's @ray.remote(max_retries=N) decorator retries the entire task from line 1 when any exception is raised — including stripe.charges.create() — on any downstream failure; Ray Actors share one Stripe key across all concurrent callers with no per-call spend cap, amplifying any billing bug across the entire cohort; and Ray Serve replicas can deliver the same charge request twice after a health-check-triggered re-route. Three Ray-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-call vault keys via a spend-cap proxy — that closes each one.
Modal's @app.function(retries=N) decorator retries the entire Python callable from line 1 when any exception is raised — including stripe.charges.create() — on any downstream failure; Secret.from_name() injects the same unrestricted Stripe key into every concurrent invocation spawned by .map() or .starmap() with no per-invocation spend cap; and Modal web endpoints receive the same charge request twice when a client retries on a cold-start delay. Three Modal-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-invocation vault keys via a spend-cap proxy — that closes each one.
Dagster's RetryPolicy re-runs the entire op callable when a downstream step fails after the Stripe charge already succeeded; a partitioned asset backfill fires concurrent independent materializations per partition with no cross-partition deduplication; and re-executing a run from failure re-runs the billing op that charged before it raised an exception. Three Dagster-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-op vault keys via a spend-cap proxy — that closes each one.
Celery's autoretry_for parameter re-runs the entire task callable on any downstream exception, firing a duplicate Stripe charge if the original charge succeeded before the exception was raised; a module-level stripe.api_key shared across all worker processes in the pool means a single runaway billing task can exhaust rate limits and spend caps for every other agent simultaneously; and enabling acks_late=True for reliability re-delivers a task to the broker after a worker crash, re-firing a charge that already completed. Three Celery-specific failure modes with Python code, and the two-layer governance pattern — content-hash idempotency keys plus per-task vault keys via a spend-cap proxy — that closes each one.
Airflow's task retry re-fires completed Stripe charges when a downstream step fails after the charge succeeds; DAG backfill and task clearing replay billing for already-charged customers; and dynamic task mapping with expand() creates concurrent billing fan-out with no cross-task deduplication. Three Airflow-specific failure modes with Python TaskFlow API code, and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Inngest's step.run() callback re-executes from the top on any throw — so if stripe.charges.create() succeeds but the database write throws, the charge fires again on retry; step.sendEvent() fan-out triggers concurrent billing function runs with no cross-run deduplication when the parent is triggered twice; and a module-level Stripe key shared across all fan-out instances has no per-customer spend cap. Three Inngest-specific failure modes with TypeScript SDK code, and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Composio's STRIPE_CREATE_CHARGE action does not inject idempotency keys, so LangChain, CrewAI, and AutoGen framework retries create duplicate charges on any tool execution failure; the default entity_id="default" shares one full-access Stripe connection across every agent in your system with no per-agent scope or spend cap; and Composio's trigger system can fire two concurrent billing agent runs when an incoming webhook is delivered twice. Three Composio-specific failure modes and the governance pattern — content-hash idempotency keys plus per-entity vault keys via a spend-cap proxy — that closes each one.
Windmill's per-script retry re-fires completed Stripe charges when a downstream step raises an exception after the charge succeeds; flow step retry restarts billing steps without checkpoint protection; and For loop parallelism fires concurrent charges across all list items with no cross-iteration deduplication. Three Windmill-specific failure modes with TypeScript and Python SDK code, and the two-layer governance pattern — content-hash idempotency keys plus per-script vault keys via a spend-cap proxy — that closes each one.
Prefect's task retry parameter re-fires completed Stripe charges when a downstream task raises an exception after the charge succeeds; flow-level retries re-run all non-cached tasks including billing unless a cache_key_fn is configured; and concurrent scheduled flow runs — triggered when a cron interval is shorter than execution time or when a manual run overlaps with a scheduled one — create duplicate charges with no cross-run deduplication. Three Prefect-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Temporal's durable execution model introduces three Stripe billing risks unique to its architecture: the default activity RetryPolicy retries failed activities including ones where the Stripe charge already completed, firing a second charge with no idempotency key; Stripe API calls placed directly in workflow code (outside activity boundaries) re-execute on every Temporal replay during worker recovery; and child workflow fan-out re-bills already-charged customers when the parent workflow is retried under a new execution. Three Temporal-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
LangGraph's conditional retry edge routes back to the billing node on any tool error, re-firing a Stripe charge that already succeeded; the MemorySaver thread checkpoint persists the full message history including prior billing results, causing the LLM to replay charges on ambiguous continuations of the same thread_id; and the Send API fan-out re-dispatches all customers — including already-billed ones — when the orchestrating node is retried. Three LangGraph-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Mastra's agent retry loop re-fires a completed Stripe charge when a tool returns an error after the charge succeeds; parallel Workflow steps fire concurrent Stripe calls with no cross-call deduplication, so two steps billing the same customer can create two charges before either result is registered; and Mastra's persistent agent memory stores prior tool call results in conversation context, causing the LLM to re-execute the billing tool on ambiguous follow-up prompts in resumed sessions. Three Mastra-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Relevance AI's Tool step retry re-executes completed HTTP steps on any downstream failure, re-firing a Stripe charge that already succeeded; bulk agent trigger runs execute concurrently with no cross-run deduplication, so triggering the same agent twice creates two parallel billing runs that both reach Stripe before either completes; and workspace Tool definitions store one Stripe key shared across every agent and every run in the project. Three Relevance AI-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Gumloop, the AI-native visual workflow builder, introduces three Stripe billing risks unique to its execution model: flow retry re-executes every block from the first one, re-firing any Stripe HTTP Request node that already completed before the downstream error; the Loop block iterates over customer arrays without checkpointing completed items, so batch retry re-bills every customer from index 0 when the original run crashed mid-array; and the AI node can emit multiple billing tool calls in a single LLM turn when models with parallel function calling process an ambiguous billing instruction, sending two simultaneous Stripe requests with no idempotency key. Three Gumloop-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-flow vault keys via a spend-cap proxy — that closes each one.
Activepieces, the open-source automation platform, introduces three Stripe billing risks unique to its execution model: retrying a failed flow run from the dashboard creates a new flow run from step 1 with the original trigger data — the Stripe action step re-fires even if the charge already completed; Activepieces processes each received webhook as an independent flow run with no cross-run deduplication, so an upstream service retrying a webhook delivery creates two concurrent runs that both execute the billing step simultaneously; and a Code piece that initializes the Stripe client at module scope shares that client across all concurrent flow executions in the same worker process. Three Activepieces-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-flow vault keys via a spend-cap proxy — that closes each one.
Zapier's automation platform introduces three Stripe billing risks unique to its execution model: replaying a failed task from Zap History re-runs all Zap steps including the Stripe action — no step is skipped even if the charge already completed; Zapier's built-in auto-retry for failed tasks re-executes the Stripe action on transient errors, creating a duplicate charge with no trace in the task log; and the AI by Zapier step uses tool-use to decide whether to call downstream Zap actions — ambiguous billing instructions or parallel tool calls can trigger the Stripe action multiple times in one AI step run. Three Zapier-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Make.com's visual automation builder introduces three Stripe billing risks unique to its execution model: when a scenario errors after the Stripe HTTP module has already created a charge, Make.com stores the execution as Incomplete — re-running it from the Operations panel restarts the entire scenario and re-fires the Stripe call on an already-billed customer; Make.com's error handler Retry directive re-executes the failed module, so a transient network error after a successful charge fires an identical request without an idempotency key; and Make.com's instant webhook trigger can receive the same payload twice — from the upstream system retrying and from Make.com's own delivery queue — creating two concurrent scenario runs that both reach the billing HTTP module with identical inputs. Three Make.com-specific failure modes and the two-layer governance pattern — content-hash idempotency keys via Make.com's built-in sha256() expression plus per-scenario vault keys via a spend-cap proxy — that closes each one.
Flowise's visual low-code builder introduces three Stripe billing risks unique to its execution model: Custom Tool functions run server-side in the Flowise Node.js process — process.env.STRIPE_KEY is a single shared environment variable with no per-session isolation, so all concurrent billing chatflow sessions draw from the same key with no per-customer or per-run scoping; the Flowise chatflow REST API has no input-hash deduplication, so a network-level caller retry re-runs the full agent conversation and re-fires the billing tool on an already-billed customer; and in Flowise Agentflow (the multi-agent canvas), a Worker agent that exhausts its max iterations limit without producing a valid Final Answer can be retried by the Orchestrator with the same task — re-calling the billing tool that already completed a charge on an earlier iteration. Three Flowise-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-session vault keys via a spend-cap proxy — that closes each one.
n8n's visual workflow builder introduces three Stripe billing risks unique to its execution model: Retry Failed Execution re-runs the entire workflow from the beginning when any downstream node fails — if the billing Code node already completed a charge before the database write timed out, retrying bills the customer again; window buffer memory persists completed tool calls in the agent's conversation window, and when the same workflow fires again the LLM may replay the charge on ambiguous follow-up instructions; and in queue mode (horizontal scaling with Redis workers), duplicate webhook deliveries or concurrent trigger events cause two workers to each run the AI agent simultaneously, both calling the billing tool before either result is registered. Three n8n-specific failure modes and the two-layer governance pattern — content-hash idempotency keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Cohere Command R+ introduces three Stripe billing risks: parallel tool_calls in one co.chat() response fire two charge_stripe invocations simultaneously before any tool result is registered; the Cohere SDK's RequestOptions(max_retries=N) compounds a multi-step while-tool-calls loop — a retry with no memory of the charge that already completed causes the model to call charge_stripe again; and Cohere's stateless chat API means sessions reconstructed from a stored chat_history replay completed billing operations when ambiguous follow-up prompts reference prior context. Three Cohere-specific failure modes and the two-layer governance pattern — restricted keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Mistral Agents API introduces three Stripe billing risks: Mistral large models emit parallel tool_calls in a single completion, firing two charges simultaneously before any tool result is registered; agent-level tool definitions bind the Stripe key at agents.create() time, sharing one key across all concurrent agents.complete() calls with no per-run isolation; and retry wrappers on agents.complete() retrigger billing after transient HTTP errors, firing a second charge on a customer already billed. Three Mistral Agents-specific failure modes and the two-layer governance pattern — restricted keys plus per-run vault keys via a spend-cap proxy — that closes each one.
OpenAI Swarm introduces three Stripe billing risks: context_variables propagates the bare Stripe API key to every agent in a handoff chain, giving read-only support agents the same full billing key as the billing specialist; tool exceptions trigger an LLM retry cycle that re-calls Stripe without idempotency keys on transient errors (a charge that completed before the network error fires again on retry); and max_turns permits multiple billing iterations per run with no spend-cap enforcement between calls. Three Swarm-specific failure modes and the two-layer governance pattern — restricted keys plus per-role vault keys via a spend-cap proxy — that closes each one.
Amazon Bedrock Agents introduces three Stripe billing risks: the agent runtime retries a failed Lambda invocation with identical action arguments (duplicating the Stripe charge if it completed before the timeout); session context accumulation causes the agent to replay billing operations on ambiguous follow-up turns in the same sessionId; and multi-agent supervisor–collaborator setups can produce double charges when the supervisor doesn't receive confirmation from the billing sub-agent. Three Bedrock Agents-specific failure modes and the two-layer governance pattern — restricted keys plus per-role vault keys via a spend-cap proxy — that closes each one.
Microsoft's Azure AI Agent Service introduces three Stripe billing risks: azure-core's built-in retry policy re-executes tool call handlers on transient errors (producing duplicate charges with no idempotency key); the Agent Service marks failed tool runs as retriable and re-submits identical tool call arguments to a new run (which fires the Stripe charge again); and persistent Thread history causes the agent to replay completed billing operations on ambiguous follow-up messages. Three Azure AI Agent Service-specific failure modes and the two-layer governance pattern — restricted keys plus per-role vault keys via a spend-cap proxy — that closes each one.
Google's Vertex AI / Gemini SDK introduces three Stripe billing risks: generate_content() retry logic re-executes FunctionCall responses, creating duplicate charges on API retries; Gemini's parallel function calling emits multiple charge_stripe calls in a single response, firing them simultaneously; and ChatSession history accumulates completed billing operations, which the model may replay on ambiguous follow-up prompts. Three Vertex AI-specific failure modes and the two-layer governance pattern — restricted keys plus per-role vault keys via a spend-cap proxy — that closes each one.
HuggingFace smolagents introduces three Stripe billing risks: CodeAgent generates arbitrary Python that can call stripe.Charge.create() directly in the sandboxed interpreter, completely bypassing your tool governance layer; ToolCallingAgent catches tool exceptions and lets the LLM retry, re-calling Stripe without idempotency keys on transient network errors; and ManagedAgent delegation passes billing context in the task string, which a sub-agent may act on independently and charge again. Three smolagents-specific failure modes and the two-layer governance pattern — restricted keys plus per-role vault keys via a spend-cap proxy — that closes each one.
Google Agent Development Kit (ADK) introduces three Stripe billing risks: ParallelAgent executes sub-agents concurrently so two branches can each fire a charge without coordination; LoopAgent retries the sub-agent on every iteration, re-calling Stripe without idempotency keys when a transient error returns; and ADK's session service persists billing context across resumed runs, enabling re-execution of charge tools from prior conversation history. Three ADK-specific failure modes and the two-layer governance pattern — restricted keys plus per-run vault keys via a spend-cap proxy — that closes each one.
Agno (formerly Phidata) retries failed tool calls by returning the error to the LLM, which re-invokes the tool — without idempotency keys, each retry fires a new Stripe charge. In a multi-agent Team, all member agents can reach Stripe tools, and a bare key gives every member full permissions. Session history replay via SqlAgentStorage puts prior tool call results back in LLM context, inviting re-execution on ambiguous prompts. Three Agno-specific failure modes and the governance pattern that closes each one: content-hash idempotency keys, per-role vault keys scoped by proxy endpoint policy, and a pre-charge existence check tool that guards against session replay.
Haystack 2.x pipeline retries re-call every component from the beginning — including the Stripe component that already charged. Concurrent pipeline branches fire Stripe independently within a single pipeline.run() call. Component instances initialized at build time share one Stripe key across all runs. Three Haystack-specific failure modes and the governance pattern that closes each one: idempotency keys injected as pipeline inputs, atomic billing components that own all Stripe calls, and per-run vault keys via a proxy with daily spend caps and audit.
PydanticAI's ModelRetry re-invokes your Stripe tool without idempotency keys — if the charge already fired before the retry, you bill the customer twice. A shared stripe_client in RunContext.deps means all concurrent agent.run() calls share one API key. Structured result validation loops (Pydantic schema mismatch → LLM retry) re-execute the entire tool call sequence. Three PydanticAI-specific failure modes and the governance pattern that closes each one: run_id-keyed idempotency in deps, per-agent-type vault keys, and structured tool return values that reduce validation retries.
LlamaIndex's ReActAgent retries tool calls on observation errors — without idempotency keys, each retry is a new Stripe charge. SubQuestionQueryEngine parallelizes them. Multi-agent AgentRunner pipelines share a single Stripe key across all child agents by default. Three LlamaIndex-specific failure modes and the governance pattern that closes each one: idempotency keys bound at run time, SubQuestion isolation for billing tools, and per-role vault keys enforced at the proxy layer.
DSPy's teleprompter optimizers (MIPRO, BootstrapFewShot) run real Stripe tool calls during prompt tuning by default. Three failure modes unique to DSPy — optimizer trial explosion, assertion-triggered retry without idempotency keys, and process-global stripe.api_key contamination across loaded modules — and the governance pattern that closes all three: mock-based optimization, per-run idempotency key closures, and per-module vault keys via a spend-cap proxy.
Semantic Kernel's plugin system makes it easy to register a Stripe function and let FunctionChoiceBehavior.Auto invoke it. Three failure modes unique to SK's architecture — all-or-nothing plugin scope, the uncapped planner auto-invoke loop, and ChatHistory persisting live Stripe data across turns — and the governance pattern that closes all three: restricted keys as a first layer, per-invocation vault keys as a second.
AutoGen's conversation-driven architecture makes it easy to register a Stripe function tool and let AssistantAgent call it across multi-turn exchanges. Three failure modes unique to AutoGen — the conversation retry storm, the shared GroupChat function registry, and the code execution bypass — and the governance pattern that closes all three: restricted Stripe keys, per-conversation vault keys, and a proxy layer that enforces spend caps before calls reach Stripe.
LangSmith traces your LLM calls perfectly — tokens, latency, tool arguments, reasoning chains. But when your agent charges Stripe, LangSmith goes blind: it sees the tool was called, not whether the charge succeeded, what the charge ID was, or how much money moved. This post covers the observability gap, a two-line proxy fix, per-agent vault keys with spend caps, LangSmith + Keybrake correlation by run ID, and a full comparison table of what each system traces.
Giving a LangChain agent access to Stripe is three lines of code. Giving it access safely — with spend caps, endpoint allowlists, and an audit trail the agent cannot alter — requires a few more. This post covers scoped Stripe keys, in-process spend caps, proxy routing via api_base, and a full StripeChargeTool with Pydantic validation, idempotency keys, and pytest coverage. Comparison table: bare key vs restricted key vs vault key.
When a human clicks "Pay" twice, your UI blocks the second click. When an agent retries a failed POST /v1/charges, nothing blocks it — and Stripe will happily create two charges. This post covers the three agent failure modes (network timeout, mid-transaction restart, parallel instances), three patterns for agent-safe idempotency (per-run UUID, content hash, external sequence), and how proxy-layer idempotency adds a safety net for agents that forget to set the header.
CrewAI makes it easy to wire a Stripe tool into a multi-agent crew — but the framework hands the agent a raw Stripe key with no spend cap, no kill switch mid-run, and no per-call audit trail. This post walks through three real failure modes, then shows a governance pattern (restricted key + proxy layer + vault key per run) that closes each gap with two lines of config change.
How to build a production-grade governance layer for your AI agent's API calls in Python — Pydantic policy models, a thread-safe pre-call spend enforcer, SQLite audit logging, pytest enforcement tests, and a clear-eyed look at where agent-side code falls short (multi-instance deployments, restarts, sub-second revoke) and why a proxy layer closes the gap.
How to set up, use, and test Stripe restricted API keys in Python-based AI agents — with stripe-python code examples for five agent archetypes (refund agent, analytics agent, subscription manager, payment capture agent, dispute handler), Python-specific gotchas (async safety, per-request key passing, 403 handling), and the two gaps that restricted keys can't close regardless of permission configuration.
A full breakdown of Stripe's ~60 permission toggles — what Read vs. Write unlocks for each resource category, minimum permission sets for five agent archetypes (refund agent, billing agent, subscription manager, analytics agent, dispute handler), the principle of least privilege in practice, and the three gaps permissions alone can't close (spend volume, customer scope, parameter allowlists).
When ten users each trigger an AI agent in your Next.js app at the same time, process.env.STRIPE_SECRET_KEY becomes a shared liability — no per-session spend cap, no per-agent revocation, no attribution in the audit log. Vault keys fix all three. Covers the Route Handler pattern, the Vercel AI SDK streamText streaming edge case, Server Actions, and why a proxy layer is required for spend cap enforcement.
Five concrete Stripe Restricted Key examples with exact permission sets — refund agent, billing agent, subscription manager, payment capturer, and read-only analytics agent. For each: the exact resources to enable, a one-line CLI command, and the specific gap this configuration still leaves open (spend cap, customer scope, parameter allowlist, revoke latency).
When one agent makes one API call, a .env file is fine. When fifty agents call Stripe, Twilio, and Resend, you need a control plane — and neither LLM gateways (LiteLLM, Portkey) nor traditional gateways (Kong, AWS API Gateway) provide it. Four properties a vendor API gateway for agents must have, why cost parsing from response bodies is the hard part, and when to build vs use a managed proxy.
The @function_tool decorator makes Stripe tools trivial to add — and the spend-cap gap just as trivial to miss. Three gaps the decorator can't fill (no per-run budget, no sub-second revoke, no per-call audit with agent context), the two-line proxy override, how to issue per-run vault keys, and what the audit log shows from a stuck billing agent that the Stripe dashboard never will.
CrewAI, LangGraph, and AutoGen all encourage shared API keys by default. In production, that pattern produces five distinct failure modes: attribution collapse (can't tell which agent spent what), rate-limit contention (one agent's burst kills another's quota), blast radius on compromise (rotating one key kills all agents), scope mismatch (least-privileged agents get the most-permissive key), and audit log collapse (one event stream, no per-agent reconstruction). Here's how the per-agent vault key pattern closes all five.
There are four ways to add spend monitoring to an AI agent. Three of them tell you about the damage after it's done — cloud billing alarms fire 8–48 hours late, vendor threshold emails arrive 15–60 minutes post-threshold, and agent-side counters reset on restart and don't aggregate across instances. One pattern fires before the spend happens. Here's how each works, what it catches, and how to layer them.
Handing an AI agent your Twilio key is a four-figure SMS bill waiting to happen. Retry storms send every message 4–6×, international routing bleed turns an $82 batch into $400, and an unsubscribed-list broadcast sends 50,000 messages before anyone checks the console. Four controls — per-day USD cap, destination prefix allowlist, deduplication window, sub-second revoke — prevent all three failure modes at the proxy layer, before calls reach Twilio.
Wiring Stripe into a LangChain agent takes ten lines. Limiting what that agent can spend takes zero lines — because there's nothing to configure. Three concrete failure modes (stuck refund loop, unbounded charges, customer scope bleed) and the two-line fix that closes all three without touching your agent code.
Stripe Agent Toolkit, Stripe Projects, and proxy-layer governance all shipped in 2026 Q1-Q2. Here's the three-layer model — identity, authorization, enforcement — what each release covers, the three gaps that still have no clean answer, and a concrete build-on-today stack for engineers running agents against production money.
Stripe Agent Toolkit lets Claude issue refunds and charges through MCP in under 30 seconds of config. The off-switch — spend cap, kill switch, per-call audit log — takes two minutes to add by routing through a governance proxy. Walkthrough: the two failure modes, the before/after config, and what you get.
Stripe Restricted Keys are the right primitive for about sixty percent of AI agent use cases. The four gaps — no per-day spend cap, no parameter-level scope, no sub-second mid-run revoke, no per-call audit with parsed cost — are where the real money leaks. The native Stripe workarounds for each, and when they stop being enough.
Your agent is burning Stripe charges and you have ten minutes. The two moves people conflate — rotating the upstream key vs revoking a scoped one — have a 2-3 order-of-magnitude latency gap. Minute-by-minute playbook for both, with the call to make first.
The sixteen columns that earn their keep in an AI agent audit log, the SQL with indexes, five operational queries you'll run more than you expect, and a synthetic stuck-refund incident traced from log rows alone.
Agent governance isn't a product — it's a four-layer stack (LLM traffic, LLM observability, SaaS API governance, agent identity). Which proxy covers which risk, which players live at which layer, and the single header that lets you join an incident across all of them.
Five controls every team needs before handing an autonomous agent a production Stripe key — what Stripe gives you out of the box, what it doesn't, and how to assemble the rest with either a wrapper or a proxy.