Spring Cloud Circuit Breaker and Stripe Integration: How CircuitBreakerFactory TimeLimiter Fires After Stripe Commits ch_A and the Fallback Returns FAILED Letting the Caller Retry with UUID_B, the Resilience4j AOP Proxy Is Outer to @Transactional So the Fallback Sees an Empty Database After Rollback While ch_A Stays on Stripe, and the CLOSED-to-OPEN Transition Mid-Batch Means the Threshold-Crossing Request Committed ch_A While All Subsequent CallNotPermittedException Items Are Re-queued with New UUIDs

The prior Resilience4j post in this series covered direct Resilience4j annotation usage with Stripe: @CircuitBreaker, @Retry, and @TimeLimiter stacked on the same method, the fallback execution order when multiple annotations compose, and the HALF_OPEN probe deduplication invariant. This post moves to the Spring Cloud Circuit Breaker abstraction layer — spring-cloud-starter-circuitbreaker-resilience4j — and examines three failure modes that arise specifically from how the abstraction layer integrates with Spring’s AOP infrastructure and how the factory’s retry contract is presented to calling code. In all three modes the outcome is the same: Stripe records two Charge or PaymentIntent objects for the same logical billing event, each with a distinct Idempotency-Key header value, and the customer is billed twice.

The common thread is a gap between what the circuit breaker signals to calling code and what Stripe has actually committed. A circuit breaker communicates resilience outcomes — CLOSED, OPEN, HALF_OPEN, the fallback return value — to the code that initiated the call. Stripe communicates charge outcomes — HTTP 200, Charge.status = "succeeded" — to the Java SDK before any circuit-breaker state is updated. These two communication paths are asynchronous with respect to each other in ways that are easy to misread as synchronous.

The three failure modes in this post are distinguishable by mechanism: the first arises from the TimeLimiter’s relationship to Stripe’s HTTP commit point; the second arises from the Resilience4j AOP aspect’s position in the Spring proxy chain relative to @Transactional; and the third arises from a subtle property of the CLOSED-to-OPEN state transition that is not obvious from the Resilience4j documentation — namely that the call which triggers the transition is not blocked or aborted, it completes normally, and if it committed a Stripe charge that charge is on Stripe’s books regardless of what the circuit does next.

For the tests in this post, the stack is: Spring Boot 3.x, spring-cloud-starter-circuitbreaker-resilience4j 3.x (which brings resilience4j-spring-boot3 and resilience4j-reactor), Spring Data JPA on PostgreSQL 15 via Testcontainers, WireMock for Stripe API stubs, and JUnit 5.

Background: Spring Cloud Circuit Breaker vs. direct Resilience4j

Resilience4j can be used in two ways from a Spring Boot application. The first is direct: add resilience4j-spring-boot3 to the classpath, use the @CircuitBreaker, @Retry, and @TimeLimiter annotations from the io.github.resilience4j.spring6.annotation package, and configure each via application.yml. The second is through the Spring Cloud Circuit Breaker abstraction: add spring-cloud-starter-circuitbreaker-resilience4j, inject a CircuitBreakerFactory bean, and call factory.create("name").run(supplier, fallback). Both approaches ultimately invoke the same underlying Resilience4j CircuitBreaker state machine, but they differ significantly in how they interact with the Spring AOP proxy chain, how they expose state to calling code, and how the TimeLimiter is configured and enforced.

The Spring Cloud Circuit Breaker factory approach is recommended in the official Spring Cloud documentation for new applications because it decouples the calling code from the specific circuit-breaker implementation (Resilience4j, Hystrix, Sentinel). The CircuitBreakerFactory.create(String id) method returns a CircuitBreaker with two methods: run(Supplier<T> toRun) and run(Supplier<T> toRun, Function<Throwable, T> fallback). The factory is configured via a Customizer<Resilience4JCircuitBreakerFactory> bean that sets the CircuitBreakerConfig and TimeLimiterConfig for each named circuit breaker.

The TimeLimiter is a detail that developers sometimes overlook. Unlike a simple connection timeout on the HTTP client, the TimeLimiter wraps the entire Supplier — including both the outbound HTTP call and any processing after it returns. The default TimeLimiterConfig.ofDefaults() sets a 1-second timeout. If the Stripe SDK call takes longer than 1 second end-to-end, the TimeLimiter fires java.util.concurrent.TimeoutException. This happens on the calling thread, regardless of what state Stripe’s server is in at that moment. The Stripe server may have already committed the charge before the calling thread decided the operation timed out.

Mode 1: CircuitBreakerFactory.create("stripe").run(supplier, fallback) — TimeLimiter fires after Stripe commits ch_A — fallback returns ChargeStatus.FAILED — caller retries with new UUID.randomUUID() — ch_B committed alongside committed ch_A

The developer builds a billing service using the Spring Cloud Circuit Breaker factory API. The circuit breaker wraps every call to the Stripe SDK to protect against Stripe outages. The TimeLimiter is left at its default (1 second) to prevent a slow Stripe API from blocking the billing thread pool indefinitely. The Stripe idempotency key is generated with UUID.randomUUID() at the top of the method. If the circuit breaker’s fallback fires (for any reason), the method returns ChargeStatus.FAILED. The calling code re-queues failed charges for retry, which calls the billing method again from scratch.

// application.yml — default TimeLimiterConfig (1 second)
spring:
  cloud:
    circuitbreaker:
      resilience4j:
        enabled: true

// No explicit TimeLimiterConfig customizer — defaults apply:
// TimeLimiterConfig.ofDefaults() → timeoutDuration = 1 second

// BillingService.java — unsafe: UUID at method entry, fallback returns FAILED
@Service
public class BillingService {

    @Autowired
    private CircuitBreakerFactory circuitBreakerFactory;

    @Autowired
    private StripeClient stripeClient;

    public ChargeResult chargeCustomer(
            String customerId, int amountCents,
            String subscriptionId, String billingPeriod) {

        // Generated fresh on every call — also fresh on every retry from the caller
        String idempotencyKey = UUID.randomUUID().toString();

        CircuitBreaker cb = circuitBreakerFactory.create("stripe");

        return cb.run(
            () -> {
                ChargeCreateParams params = ChargeCreateParams.builder()
                    .setCustomer(customerId)
                    .setAmount((long) amountCents)
                    .setCurrency("usd")
                    .putMetadata("subscription_id", subscriptionId)
                    .putMetadata("billing_period", billingPeriod)
                    .build();

                RequestOptions options = RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .setConnectTimeout(2000)    // Stripe SDK connect timeout: 2s
                    .setReadTimeout(10000)      // Stripe SDK read timeout: 10s
                    .build();

                // Stripe receives this request, creates charge, sends HTTP 200.
                // If the HTTP 200 response takes > 1s to traverse the network back
                // to this JVM, TimeLimiter fires TimeoutException on THIS thread
                // before this line returns — even though ch_A is committed on Stripe.
                Charge charge = stripeClient.charges().create(params, options);
                return new ChargeResult(charge.getId(), ChargeStatus.SUCCEEDED);
            },
            throwable -> {
                // Fires on TimeoutException, CallNotPermittedException,
                // or any exception thrown inside the supplier.
                // Developer logs the error and returns FAILED.
                log.warn("Stripe charge failed for customer {} subscription {}: {}",
                    customerId, subscriptionId, throwable.getClass().getSimpleName());
                return new ChargeResult(null, ChargeStatus.FAILED);
            }
        );
    }
}

// SubscriptionBillingJob.java — retries FAILED charges
@Component
public class SubscriptionBillingJob {

    @Autowired
    private BillingService billingService;

    @Autowired
    private RetryQueue retryQueue;

    public void runBillingCycle(List<Subscription> subscriptions) {
        for (Subscription sub : subscriptions) {
            ChargeResult result = billingService.chargeCustomer(
                sub.getCustomerId(),
                sub.getAmountCents(),
                sub.getId(),
                sub.getCurrentBillingPeriod()
            );

            if (result.getStatus() == ChargeStatus.FAILED) {
                // Developer's reasoning: "chargeCustomer returned FAILED —
                // the circuit breaker signaled that the call did not complete
                // successfully — it is safe to retry this subscription."
                retryQueue.add(sub);
            }
        }
        // Retry queue processing: calls chargeCustomer() again for each sub.
        // chargeCustomer() generates new UUID.randomUUID() — UUID_B for the sub
        // whose TimeLimiter fired after ch_A was already committed on Stripe.
    }
}

The sequence of events for the timeout failure case:

1. The calling thread invokes cb.run(supplier, fallback).
2. The Spring Cloud Circuit Breaker implementation (Resilience4j under the hood) submits the supplier to a ScheduledExecutorService and schedules a cancellation task to fire after timeoutDuration (1 second by default).
3. The supplier begins execution: the Stripe SDK sends an HTTP POST to https://api.stripe.com/v1/charges.
4. Stripe receives the request, validates the card, commits ch_A, and begins sending the HTTP 200 response.
5. The HTTP response is in transit. The network is slow (Stripe’s edge is congested, or there is a high-latency path through a VPC NAT). After 1 second, the cancellation task fires.
6. TimeoutException is thrown on the calling thread. The supplier’s future is cancelled.
7. The fallback function receives the TimeoutException and returns new ChargeResult(null, ChargeStatus.FAILED).
8. The calling code sees FAILED and adds this subscription to retryQueue.
9. The retry runs: chargeCustomer() generates key_B = UUID.randomUUID(). key_B ≠ key_A. Stripe has no record of key_B in its 24-hour idempotency cache. Stripe creates ch_B. The customer’s card is debited a second time.

The developer’s reasoning: “The TimeLimiter fired because Stripe did not respond within 1 second. If a charge had been committed, the HTTP 200 would have carried the charge object and the supplier would have returned before the timeout. A timeout means Stripe never sent a response, which means Stripe never processed the request.”

The gap in this reasoning: the TimeLimiter measures latency on the calling thread, not on Stripe’s server. Stripe’s API is synchronous with respect to the charge commitment: Stripe commits the charge and then sends the HTTP 200 response. The HTTP response’s transit time from Stripe’s server to the calling JVM is not instantaneous. A timeout on the calling thread means the response had not arrived within the timeout window. It does not mean Stripe had not committed the charge. Stripe committed the charge at the moment it sent the response. The response was in transit — perhaps 80% of the way through a congested network path — when the calling thread decided it was done waiting. Stripe’s server has no knowledge of the calling thread’s timeout decision.

There is a second important detail: the Stripe SDK’s own read timeout is configured separately from the TimeLimiter. In the code above, setReadTimeout(10000) gives the Stripe SDK 10 seconds to read the response. But the TimeLimiter’s default is 1 second, which fires first. This means the Stripe SDK is happily waiting for a response (within its own 10-second budget) while the TimeLimiter has already fired and returned to the caller. The supplier is submitted to a thread pool and cancellation is attempted, but whether the Stripe SDK actually aborts the TCP connection at this point depends on thread interrupt handling inside the SDK. In practice, the Stripe SDK does not abort the underlying HTTP connection immediately on thread interrupt. The TCP connection remains open, and the response eventually arrives — after the calling thread has moved on.

The fix requires two changes. First, the idempotency key must be stable across retries for the same logical billing event, so that if the retry fires for a charge that was already committed on Stripe, Stripe returns ch_A from its idempotency cache instead of creating ch_B:

// ContentHashKey.java — stable key from business inputs
public final class ContentHashKey {

    public static String forCharge(
            String customerId, String subscriptionId,
            String billingPeriod, int amountCents) {
        // SHA-256 of stable inputs, hex-encoded, truncated to 64 chars
        // (Stripe's Idempotency-Key limit is 255 bytes; 64 hex chars is safe)
        String input = customerId + ":" + subscriptionId + ":"
            + billingPeriod + ":" + amountCents;
        byte[] digest = MessageDigest.getInstance("SHA-256")
            .digest(input.getBytes(StandardCharsets.UTF_8));
        return HexFormat.of().formatHex(digest).substring(0, 64);
    }
}

// BillingService.java — fixed: content-hash key
public ChargeResult chargeCustomer(
        String customerId, int amountCents,
        String subscriptionId, String billingPeriod) {

    // Same key for every retry of the same logical charge
    String idempotencyKey = ContentHashKey.forCharge(
        customerId, subscriptionId, billingPeriod, amountCents);

    CircuitBreaker cb = circuitBreakerFactory.create("stripe");

    return cb.run(
        () -> {
            Charge charge = stripeClient.charges().create(
                ChargeCreateParams.builder()
                    .setCustomer(customerId)
                    .setAmount((long) amountCents)
                    .setCurrency("usd")
                    .build(),
                RequestOptions.builder()
                    .setIdempotencyKey(idempotencyKey)
                    .build()
            );
            return new ChargeResult(charge.getId(), ChargeStatus.SUCCEEDED);
        },
        throwable -> new ChargeResult(null, ChargeStatus.FAILED)
    );
    // Now: if the retry fires for a TimeLimiter-timeout attempt that committed
    // ch_A, the retry sends idempotencyKey = same hash → Stripe returns ch_A
    // from its 24h cache — no ch_B created.
}

Second, the TimeLimiterConfig timeout should be set higher than the Stripe SDK’s read timeout to ensure the TimeLimiter only fires when the Stripe SDK itself has given up, not before:

// CircuitBreakerConfig.java — set TimeLimiter above Stripe SDK read timeout
@Bean
public Customizer<Resilience4JCircuitBreakerFactory> stripeCircuitBreakerCustomizer() {
    return factory -> factory.configure(builder -> builder
        .timeLimiterConfig(TimeLimiterConfig.custom()
            // Stripe SDK read timeout is 10s — give the TimeLimiter 15s
            // so it fires only after the SDK has given up and thrown
            .timeoutDuration(Duration.ofSeconds(15))
            .build())
        .circuitBreakerConfig(CircuitBreakerConfig.custom()
            .failureRateThreshold(50)
            .slidingWindowSize(10)
            .waitDurationInOpenState(Duration.ofSeconds(30))
            .permittedNumberOfCallsInHalfOpenState(3)
            .build()),
        "stripe"
    );
}

With both fixes in place: the idempotency key is stable across retries (so a TimeLimiter timeout on a committed charge de-duplicates on the Stripe side), and the TimeLimiter timeout is above the Stripe SDK timeout (so the TimeLimiter fires only for genuine SDK-level failures, not for slow-but-successful network responses).

Integration test for Mode 1

@SpringBootTest
@WireMockTest(httpPort = 8089)
@Testcontainers
class BillingServiceTimeLimiterTest {

    @Container
    static PostgreSQLContainer<?> pg = new PostgreSQLContainer<>("postgres:15");

    @DynamicPropertySource
    static void pgProps(DynamicPropertyRegistry r) {
        r.add("spring.datasource.url", pg::getJdbcUrl);
        r.add("spring.datasource.username", pg::getUsername);
        r.add("spring.datasource.password", pg::getPassword);
    }

    @Autowired
    BillingService billingService;

    @Test
    void timelimiterFiringAfterStripeCommitsDoesNotCreateDuplicateCharge(
            WireMockRuntimeInfo wm) throws Exception {

        // Stripe stubs: first call delays 1.2s (TimeLimiter fires after 1s)
        // then returns HTTP 200 with ch_A.
        // The second call (retry) returns ch_A immediately from idempotency cache.
        stubFor(post(urlEqualTo("/v1/charges"))
            .inScenario("timeout-then-idempotent")
            .whenScenarioStateIs(STARTED)
            .willReturn(aResponse()
                .withFixedDelay(1200)  // 1.2s — TimeLimiter (1s default) fires first
                .withStatus(200)
                .withBody(chargeJson("ch_A", "succeeded")))
            .willSetStateTo("first-call-done"));

        stubFor(post(urlEqualTo("/v1/charges"))
            .inScenario("timeout-then-idempotent")
            .whenScenarioStateIs("first-call-done")
            .willReturn(aResponse()
                .withStatus(200)
                .withBody(chargeJson("ch_A", "succeeded"))  // idempotency cache returns ch_A
                .withHeader("Idempotency-Replayed", "true")));

        // Attempt 1: TimeLimiter fires → FAILED
        ChargeResult r1 = billingService.chargeCustomer(
            "cus_X", 1000, "sub_Y", "2026-10");
        assertThat(r1.getStatus()).isEqualTo(ChargeStatus.FAILED);

        // Attempt 2 (retry with fixed code): same content-hash key
        ChargeResult r2 = billingService.chargeCustomer(
            "cus_X", 1000, "sub_Y", "2026-10");
        assertThat(r2.getStatus()).isEqualTo(ChargeStatus.SUCCEEDED);
        assertThat(r2.getChargeId()).isEqualTo("ch_A");

        // Stripe received exactly ONE unique Idempotency-Key across both calls
        List<LoggedRequest> calls = findAll(postRequestedFor(urlEqualTo("/v1/charges")));
        assertThat(calls).hasSize(2);
        String key1 = calls.get(0).getHeader("Idempotency-Key");
        String key2 = calls.get(1).getHeader("Idempotency-Key");
        assertThat(key1).isEqualTo(key2);  // same content-hash key — no UUID_B
    }
}

Mode 2: @CircuitBreaker AOP proxy is outer to @Transactional inner proxy — Stripe commits ch_A inside the transaction — paymentRepository.save() throws — @Transactional rolls back — @CircuitBreaker fallback fires and queries the empty database — fallback concludes no charge committed — retry with UUID_B — ch_B alongside ch_A

The developer uses the Spring Cloud Circuit Breaker @CircuitBreaker annotation (not the direct Resilience4j annotation, but the one from org.springframework.cloud.client.circuitbreaker.CircuitBreaker — or in practice the Resilience4j @CircuitBreaker annotation with name and fallbackMethod configured via the Spring Cloud starter). The method is also annotated with @Transactional. The developer assumes that if the circuit breaker’s fallback fires, the database is in a known state (either committed or rolled back), and the fallback can inspect the database to determine whether a Stripe charge was committed.

The critical detail is AOP proxy ordering. Resilience4j’s CircuitBreakerAspect is registered with @Order(Ordered.LOWEST_PRECEDENCE - 2), which evaluates to Integer.MAX_VALUE - 2 = 2147483645. Spring’s @Transactional advice is registered at Ordered.LOWEST_PRECEDENCE = Integer.MAX_VALUE = 2147483647. In Spring AOP, a lower order number means the advice is applied first (outer proxy). CircuitBreakerAspect at 2147483645 is therefore the outer proxy. TransactionInterceptor at 2147483647 is the inner proxy.

The practical consequence of this ordering:

1. An external call arrives at the bean proxy.
2. CircuitBreakerAspect (outer) intercepts it first. Circuit is CLOSED. CircuitBreakerAspect calls through to the next proxy in the chain.
3. TransactionInterceptor (inner) intercepts it. @Transactional opens a JDBC connection and begins a transaction.
4. The method body executes. Stripe call succeeds: ch_A committed on Stripe. paymentRepository.save() throws DataAccessException.
5. TransactionInterceptor catches the exception, marks the transaction for rollback, commits the rollback to the database (all JDBC writes undone), and re-throws the exception.
6. CircuitBreakerAspect catches the re-thrown exception. The circuit records a failure. If the failure rate has not yet crossed the threshold, the circuit remains CLOSED but the exception is forwarded to the fallback method.
7. The fallback method executes. The @Transactional rollback has already been committed to the database at this point. The payment_records row does not exist.

// PaymentService.java — unsafe: @CircuitBreaker (outer) + @Transactional (inner)
@Service
public class PaymentService {

    @Autowired
    private PaymentRecordRepository paymentRecordRepository;

    @Autowired
    private StripeClient stripeClient;

    @Autowired
    private RetryQueue retryQueue;

    // Resilience4j CircuitBreakerAspect: order = Integer.MAX_VALUE - 2 (outer)
    @CircuitBreaker(name = "stripe", fallbackMethod = "chargeFallback")
    // Spring TransactionInterceptor: order = Integer.MAX_VALUE (inner)
    @Transactional
    public ChargeResult charge(
            String customerId, int amountCents, String orderId) {

        // UUID generated at method entry — new value on every call
        String idempotencyKey = UUID.randomUUID().toString();

        // Step 1: Stripe call — ch_A committed on Stripe's side
        Charge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setCustomer(customerId)
                .setAmount((long) amountCents)
                .setCurrency("usd")
                .build(),
            RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build()
        );

        // Step 2: DB write — may throw DataAccessException
        PaymentRecord record = new PaymentRecord();
        record.setIdempotencyKey(idempotencyKey);   // key_A
        record.setCustomerId(customerId);
        record.setOrderId(orderId);
        record.setAmountCents(amountCents);
        record.setStripeChargeId(charge.getId());    // ch_A
        record.setStatus("COMPLETED");
        // Throws DataAccessException:
        //   - UNIQUE constraint violation (concurrent write on same orderId)
        //   - DeadlockLoserDataAccessException
        //   - DataAccessResourceFailureException (failover replica)
        paymentRecordRepository.save(record);

        return new ChargeResult(charge.getId(), ChargeStatus.SUCCEEDED);
    }

    // Fallback — executes AFTER @Transactional has rolled back the JDBC transaction
    public ChargeResult chargeFallback(
            String customerId, int amountCents, String orderId, Throwable t) {

        // The @Transactional rollback has committed — payment_records has NO row.
        // Developer queries to check prior state:
        Optional<PaymentRecord> existing =
            paymentRecordRepository.findByCustomerIdAndOrderId(customerId, orderId);

        if (existing.isPresent()) {
            // Developer's intent: "if a record exists, a charge was committed —
            // return the existing charge ID — do not retry."
            // Reality: this branch is NEVER reached after a @Transactional rollback
            // because the INSERT was rolled back.
            log.info("Found existing charge {} for order {}",
                existing.get().getStripeChargeId(), orderId);
            return new ChargeResult(existing.get().getStripeChargeId(), ChargeStatus.SUCCEEDED);
        }

        // Path taken: DB is empty → "no charge was committed — safe to retry"
        log.warn("No charge record found for order {} — enqueuing retry", orderId);
        retryQueue.add(new PendingCharge(customerId, amountCents, orderId));
        return new ChargeResult(null, ChargeStatus.PENDING_RETRY);
    }
}

The developer’s reasoning: “The @CircuitBreaker fallback fires when something goes wrong. Inside the fallback I query the database. If a PaymentRecord exists for this orderId, I use the existing charge and return without retrying. If no record exists, the charge was not committed and I can safely retry. This pattern correctly handles both the case where the Stripe call succeeded (record exists) and the case where it failed before reaching Stripe (record does not exist).”

This reasoning would be correct if the fallback ran inside the same @Transactional context that was open during the method body. In that hypothetical, if the Stripe call succeeded and paymentRepository.save() threw, the transaction would still be open and the rolled-back INSERT would be visible to the fallback as absent (correctly reflecting the rollback). But the fallback does not run inside the original transaction. It runs after TransactionInterceptor has fully committed the rollback. By the time the fallback executes, the JDBC transaction is closed. The payment_records row is not there because it was rolled back, not because the Stripe call failed.

There is also a subtler path: if the @Transactional propagation is REQUIRES_NEW or if the fallback method itself has its own @Transactional, the timing is the same. The fallback always runs in a new execution context from the perspective of the CircuitBreakerAspect, and by the time it runs, the inner transaction has already rolled back. The database state visible to the fallback reflects committed state only, which means the rolled-back INSERT is absent.

The fix has two parts. First, move the idempotency key to a content-hash, as in Mode 1. Second, persist the charge attempt in a PENDING row in its own REQUIRES_NEW transaction before calling Stripe, so that the PENDING row survives the outer transaction’s rollback and the fallback can detect prior Stripe calls:

// PaymentService.java — fixed: PENDING row in REQUIRES_NEW + content-hash key
@Service
public class PaymentService {

    @Autowired
    private PaymentRecordRepository paymentRecordRepository;

    @Autowired
    private StripeClient stripeClient;

    @Autowired
    private PaymentService self;  // self-injection to call @Transactional(REQUIRES_NEW)

    @CircuitBreaker(name = "stripe", fallbackMethod = "chargeFallback")
    @Transactional
    public ChargeResult charge(
            String customerId, int amountCents, String orderId) {

        // Stable content-hash key
        String idempotencyKey = ContentHashKey.forCharge(customerId, orderId, amountCents);

        // Write PENDING row in a separate REQUIRES_NEW transaction that will
        // commit (and survive) regardless of what happens to the outer transaction.
        // If this throws a UNIQUE constraint violation, a prior PENDING row exists
        // for the same idempotencyKey — we check it before calling Stripe.
        self.insertPendingRow(idempotencyKey, customerId, orderId, amountCents);

        // Call Stripe — ch_A committed
        Charge charge = stripeClient.charges().create(
            ChargeCreateParams.builder()
                .setCustomer(customerId)
                .setAmount((long) amountCents)
                .setCurrency("usd")
                .build(),
            RequestOptions.builder()
                .setIdempotencyKey(idempotencyKey)
                .build()
        );

        // Update the PENDING row to COMPLETED in the outer transaction
        paymentRecordRepository.updateStatusAndChargeId(
            idempotencyKey, "COMPLETED", charge.getId());

        return new ChargeResult(charge.getId(), ChargeStatus.SUCCEEDED);
    }

    // Fallback — PENDING row is durable (committed in REQUIRES_NEW above)
    public ChargeResult chargeFallback(
            String customerId, int amountCents, String orderId, Throwable t) {

        String idempotencyKey = ContentHashKey.forCharge(customerId, orderId, amountCents);

        // Query PENDING row — it exists (survived outer rollback)
        Optional<PaymentRecord> pending =
            paymentRecordRepository.findByIdempotencyKey(idempotencyKey);

        if (pending.isPresent() && "COMPLETED".equals(pending.get().getStatus())) {
            // Outer transaction committed before fallback — charge is done
            return new ChargeResult(pending.get().getStripeChargeId(), ChargeStatus.SUCCEEDED);
        }

        // PENDING status: outer transaction rolled back after Stripe call.
        // Retry will send the same idempotencyKey — Stripe returns ch_A from cache.
        // No ch_B will be created.
        retryQueue.add(new PendingCharge(customerId, amountCents, orderId));
        return new ChargeResult(null, ChargeStatus.PENDING_RETRY);
    }

    @Transactional(propagation = Propagation.REQUIRES_NEW)
    public void insertPendingRow(
            String idempotencyKey, String customerId, String orderId, int amountCents) {
        PaymentRecord record = new PaymentRecord();
        record.setIdempotencyKey(idempotencyKey);
        record.setCustomerId(customerId);
        record.setOrderId(orderId);
        record.setAmountCents(amountCents);
        record.setStatus("PENDING");
        // Upsert — if the row already exists, leave it as-is.
        paymentRecordRepository.insertIfAbsent(record);
    }
}

With this design: the PENDING row is durable because it commits in its own REQUIRES_NEW transaction. The outer transaction’s rollback (from paymentRepository.save() throwing) does not affect the PENDING row. The fallback always finds the PENDING row, and can distinguish between “outer transaction committed” (COMPLETED status, return existing charge ID) and “outer transaction rolled back after Stripe call” (PENDING status, retry with same idempotency key). The content-hash key ensures that the retry sends the same key and Stripe returns ch_A from its 24-hour idempotency cache.

Verifying proxy ordering with a diagnostic bean

The AOP proxy order is not surfaced in most debuggers or stacktraces. A useful diagnostic is to register a custom MethodInterceptor at each order value and log the entry and exit. The output will show which interceptor wraps which:

// Diagnostic — confirms CircuitBreakerAspect is outer, TransactionInterceptor is inner
// Call stack on entry to chargeCustomer():
//   CircuitBreakerAspect.around() [order = Integer.MAX_VALUE - 2, outer]
//     └─ TransactionInterceptor.invoke() [order = Integer.MAX_VALUE, inner]
//          └─ PaymentService.charge() [method body]

// @Transactional annotation honors the order:
// @EnableTransactionManagement(order = Ordered.LOWEST_PRECEDENCE)  // Integer.MAX_VALUE
// Resilience4j CircuitBreakerAspect:
// @Order(Ordered.LOWEST_PRECEDENCE - 2)                            // Integer.MAX_VALUE - 2

// To make @Transactional the outer proxy (wrapping @CircuitBreaker):
// @EnableTransactionManagement(order = Ordered.LOWEST_PRECEDENCE - 5)
// This changes the failure mode: @Transactional is now outer and stays open
// when the @CircuitBreaker fallback fires — but this introduces a different
// problem (transaction staying open across fallback logic).
// The correct fix is the PENDING row approach, not proxy reordering.

Integration test for Mode 2

@SpringBootTest
@WireMockTest(httpPort = 8089)
@Testcontainers
class PaymentServiceProxyOrderTest {

    @Container
    static PostgreSQLContainer<?> pg = new PostgreSQLContainer<>("postgres:15");

    @DynamicPropertySource
    static void pgProps(DynamicPropertyRegistry r) {
        r.add("spring.datasource.url", pg::getJdbcUrl);
        r.add("spring.datasource.username", pg::getUsername);
        r.add("spring.datasource.password", pg::getPassword);
    }

    @Autowired
    PaymentService paymentService;

    @Autowired
    PaymentRecordRepository repo;

    @Test
    void fallbackSeesEmptyDbAfterTransactionalRollback_retryUsesSameKey() {
        // Stripe stub: first call succeeds (ch_A), second call returns from
        // idempotency cache (ch_A again — same key).
        stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody(chargeJson("ch_A", "succeeded"))));

        // Simulate: paymentRepository.save() throws UNIQUE constraint violation.
        // The UNIQUE constraint is on (customerId, orderId) in payment_records.
        // Pre-insert a conflicting row directly (bypassing service layer):
        repo.save(conflictingRecord("cus_X", "ord_1"));

        // First call: Stripe ch_A committed, then save() throws → rollback → fallback
        ChargeResult r1 = paymentService.charge("cus_X", 1000, "ord_1");
        assertThat(r1.getStatus()).isEqualTo(ChargeStatus.PENDING_RETRY);

        // Remove the conflicting row to allow the retry to commit
        repo.deleteConflicting("cus_X", "ord_1");

        // Retry: same content-hash idempotency key
        // Stripe returns ch_A from idempotency cache (Idempotency-Replayed: true)
        stubFor(post(urlEqualTo("/v1/charges"))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody(chargeJson("ch_A", "succeeded"))
                .withHeader("Idempotency-Replayed", "true")));

        ChargeResult r2 = paymentService.charge("cus_X", 1000, "ord_1");
        assertThat(r2.getStatus()).isEqualTo(ChargeStatus.SUCCEEDED);
        assertThat(r2.getChargeId()).isEqualTo("ch_A");

        // Only one unique key was sent to Stripe across both calls
        List<LoggedRequest> calls = findAll(postRequestedFor(urlEqualTo("/v1/charges")));
        Set<String> distinctKeys = calls.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(distinctKeys).hasSize(1);
    }
}

Mode 3: Batch billing job — circuit transitions from CLOSED to OPEN mid-batch — the request that crosses the failure threshold committed ch_A on Stripe but threw from the DB layer — all failed batch items (including the threshold-crosser) are re-queued with new UUID.randomUUID() — ch_B committed for the threshold-crosser

Resilience4j transitions from CLOSED to OPEN synchronously inside the completion of the call that crosses the failure rate threshold. The transition is not applied in advance to that call. The threshold-crossing call executes the supplier fully (including the Stripe HTTP call), records the outcome as a failure (because a downstream DB write threw), and then Resilience4j evaluates whether the failure rate has now reached the configured threshold. If it has, the circuit transitions to OPEN at that moment. The threshold-crossing call itself was not blocked; it ran to completion (from the supplier’s perspective). Subsequent calls after the state transition receive CallNotPermittedException immediately, without running the supplier.

A batch billing job processes hundreds of subscriptions in a loop, calling circuitBreakerFactory.create("stripe").run(supplier, fallback) for each one. With the default sliding window size of 10 and failure rate threshold of 50%, the circuit opens when 5 out of any 10 consecutive calls are recorded as failures. Consider this execution sequence:

• Calls 1–7: 4 successes, 3 failures (failure rate = 3/7 = 43% — below 50% threshold)
• Call 8: supplier executes — Stripe call succeeds (ch_A committed) — paymentRepository.save() throws DataAccessException — recorded as a failure — failure rate = 4/8 = 50% — circuit transitions to OPEN
• Calls 9–50: CallNotPermittedException immediately — supplier never invoked — no Stripe calls made

The batch error handler catches all exceptions uniformly and re-queues every failed item:

// SubscriptionBatchJob.java — unsafe: uniform exception handling re-queues
// the threshold-crosser alongside CallNotPermittedException items
@Component
public class SubscriptionBatchJob {

    @Autowired
    private CircuitBreakerFactory circuitBreakerFactory;

    @Autowired
    private StripeClient stripeClient;

    @Autowired
    private RetryQueue retryQueue;

    public void processBatch(List<Subscription> subscriptions) {
        List<Subscription> failedItems = new ArrayList<>();

        for (Subscription sub : subscriptions) {
            CircuitBreaker cb = circuitBreakerFactory.create("stripe");
            ChargeResult result = cb.run(
                () -> {
                    // New UUID per call — also new per retry
                    String key = UUID.randomUUID().toString();
                    Charge charge = stripeClient.charges().create(
                        buildParams(sub),
                        RequestOptions.builder().setIdempotencyKey(key).build()
                    );
                    paymentRepository.save(buildRecord(sub, charge.getId(), key));
                    return new ChargeResult(charge.getId(), ChargeStatus.SUCCEEDED);
                },
                throwable -> {
                    // Developer's reasoning:
                    // "throwable is either:
                    //   (a) a Stripe SDK exception — nothing was charged, safe to retry
                    //   (b) a DataAccessException from paymentRepository.save() —
                    //       @Transactional rolled back, nothing is in the DB
                    //   (c) CallNotPermittedException — circuit is OPEN, no Stripe call
                    //       was made at all, definitely safe to retry
                    // In all three cases: nothing was committed. Safe to re-queue."
                    log.warn("Charge failed for sub {}: {}",
                        sub.getId(), throwable.getClass().getSimpleName());
                    return new ChargeResult(null, ChargeStatus.FAILED);
                }
            );

            if (result.getStatus() == ChargeStatus.FAILED) {
                failedItems.add(sub);
            }
        }

        // Re-process failed items — each will call UUID.randomUUID() fresh
        // Sub #8 (the threshold-crosser) is in failedItems.
        // Its retry generates UUID_B. Stripe creates ch_B alongside ch_A.
        for (Subscription sub : failedItems) {
            retryQueue.scheduleRetry(sub);  // retry calls processBatch with [sub]
        }
    }
}

The developer’s reasoning is almost entirely correct. For items receiving CallNotPermittedException (calls 9–50 in the example above), the reasoning is correct: the circuit was OPEN when those calls were initiated, the supplier was never invoked, no Stripe call was made, and retrying is safe. The reasoning fails only for the threshold-crossing call (call 8): that call did invoke the supplier, the supplier did call Stripe, Stripe did commit ch_A. The fact that the call was eventually recorded as a failure (and caused the circuit to open) does not mean the Stripe call within it failed. The Stripe call succeeded. The failure was in the DB layer, which is subsequent to and independent of the Stripe HTTP layer.

The developer mistake is treating CallNotPermittedException as a label for the entire batch of failed items. It is only a label for items rejected after the circuit opened. The item that opened the circuit received a different exception (DataAccessException in this case), and it had already called Stripe.

There is a secondary subtlety: the developer cannot know from inside the fallback which exception type corresponds to a “Stripe call made” vs. “Stripe call not made” scenario. DataAccessException from paymentRepository.save() always means a Stripe call was made first (the code calls Stripe before saving). But DataAccessException could also come from a connection pool exhaustion before the Stripe call (if the connection is acquired before Stripe is called). CallNotPermittedException always means no Stripe call. A generic Stripe ApiException means a Stripe call was made but failed on Stripe’s side. Distinguishing these in a uniform fallback is difficult without inspecting the exception type hierarchy.

The fix is the content-hash key, which eliminates the need to distinguish. Whether or not ch_A was committed, the retry with the same key either creates ch_A (first-time success) or retrieves ch_A from Stripe’s idempotency cache (deduplicated). The fallback exception type becomes irrelevant to idempotency safety:

// SubscriptionBatchJob.java — fixed: content-hash key makes fallback type irrelevant
public void processBatch(List<Subscription> subscriptions) {
    List<Subscription> failedItems = new ArrayList<>();

    for (Subscription sub : subscriptions) {
        // Stable content-hash key — same value on every retry for this sub + period
        String idempotencyKey = ContentHashKey.forCharge(
            sub.getCustomerId(),
            sub.getId(),
            sub.getCurrentBillingPeriod(),
            sub.getAmountCents()
        );

        CircuitBreaker cb = circuitBreakerFactory.create("stripe");
        ChargeResult result = cb.run(
            () -> {
                Charge charge = stripeClient.charges().create(
                    buildParams(sub),
                    RequestOptions.builder()
                        .setIdempotencyKey(idempotencyKey)  // stable
                        .build()
                );
                paymentRepository.save(buildRecord(sub, charge.getId(), idempotencyKey));
                return new ChargeResult(charge.getId(), ChargeStatus.SUCCEEDED);
            },
            throwable -> {
                // CallNotPermittedException: circuit OPEN, no Stripe call → retry safe
                // DataAccessException: Stripe call may have committed ch_A → retry
                //   sends same idempotencyKey → Stripe deduplicates → no ch_B
                // Either way: same key → same behavior
                return new ChargeResult(null, ChargeStatus.FAILED);
            }
        );

        if (result.getStatus() == ChargeStatus.FAILED) {
            failedItems.add(sub);
        }
    }

    // Retry: each sub sends the same content-hash key.
    // Sub #8 (threshold-crosser with ch_A): Stripe returns ch_A from cache → no ch_B.
    // Subs #9-50 (CallNotPermittedException, no prior Stripe call): Stripe creates
    //   ch_X fresh — first-time charge, no duplication risk.
    for (Subscription sub : failedItems) {
        retryQueue.scheduleRetry(sub);
    }
}

An additional defensive measure is to log the exception type in the fallback before the fix is deployed, so that the threshold-crossing call can be identified in logs after a circuit-open event:

// Diagnostic fallback — log exception class to identify threshold-crosser post-incident
throwable -> {
    if (throwable instanceof CallNotPermittedException) {
        log.info("Circuit OPEN — sub {} skipped (no Stripe call)", sub.getId());
    } else {
        // DataAccessException, StripeException, TimeoutException, etc.
        // These items may have committed a Stripe charge.
        log.warn("Supplier threw for sub {} — Stripe call may have committed: {}",
            sub.getId(), throwable.getClass().getName(), throwable);
    }
    return new ChargeResult(null, ChargeStatus.FAILED);
}

Integration test for Mode 3

@SpringBootTest
@WireMockTest(httpPort = 8089)
@Testcontainers
class SubscriptionBatchCircuitBreakerTest {

    @Container
    static PostgreSQLContainer<?> pg = new PostgreSQLContainer<>("postgres:15");

    @DynamicPropertySource
    static void pgProps(DynamicPropertyRegistry r) {
        r.add("spring.datasource.url", pg::getJdbcUrl);
        r.add("spring.datasource.username", pg::getUsername);
        r.add("spring.datasource.password", pg::getPassword);
        // Small sliding window for test speed
        r.add("resilience4j.circuitbreaker.instances.stripe.sliding-window-size", "4");
        r.add("resilience4j.circuitbreaker.instances.stripe.failure-rate-threshold", "75");
        // 75% of 4 = 3 failures → open
    }

    @Autowired
    SubscriptionBatchJob job;

    @Test
    void thresholdCrosserCommittedChA_retryWithSameKeyDeduplicates() {
        // Subs 1-2: DB write succeeds — SUCCEEDED
        // Sub 3 (failure 1): Stripe succeeds, DB deadlock → FAILED
        // Sub 4 (failure 2): Stripe succeeds, DB deadlock → FAILED
        // Sub 5 (failure 3 = threshold): Stripe succeeds (ch_X committed),
        //        DB deadlock → circuit opens
        // Subs 6-10: CallNotPermittedException → FAILED (no Stripe call)

        // Arrange: Stripe stubs — all calls return 200 with unique charge IDs
        for (int i = 1; i <= 10; i++) {
            stubFor(post(urlEqualTo("/v1/charges"))
                .withHeader("Idempotency-Key", matching("key-sub-" + i + "-.*"))
                .willReturn(aResponse()
                    .withStatus(200)
                    .withBody(chargeJson("ch_sub_" + i, "succeeded"))));
        }

        // Arrange: simulate DB deadlock for subs 3-5 (will cause circuit to open at sub 5)
        simulateDeadlockForSubs("sub-3", "sub-4", "sub-5");

        List<Subscription> batch = buildBatch(10);
        job.processBatch(batch);

        // Sub 5 (threshold-crosser): Stripe received a call, ch_sub_5 committed
        // Verify exactly one Stripe call was made for sub-5 in the first pass
        verify(1, postRequestedFor(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", matching("key-sub-5-.*")));

        // Verify subs 6-10 had NO Stripe calls (circuit was OPEN)
        for (int i = 6; i <= 10; i++) {
            verify(0, postRequestedFor(urlEqualTo("/v1/charges"))
                .withHeader("Idempotency-Key", matching("key-sub-" + i + "-.*")));
        }

        // Now verify that sub-5 retry uses the SAME key (content-hash)
        // and Stripe returns from idempotency cache
        resetAllScenarios();
        stubFor(post(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", matching("key-sub-5-.*"))
            .willReturn(aResponse()
                .withStatus(200)
                .withBody(chargeJson("ch_sub_5", "succeeded"))
                .withHeader("Idempotency-Replayed", "true")));

        // Simulate DB deadlock resolved for sub-5 on retry
        clearDeadlockForSubs("sub-5");
        job.processBatch(List.of(findSub("sub-5")));

        // Sub-5 retry: Stripe received the SAME key → returned ch_sub_5 (no ch_B)
        List<LoggedRequest> sub5Calls = findAll(postRequestedFor(urlEqualTo("/v1/charges"))
            .withHeader("Idempotency-Key", matching("key-sub-5-.*")));
        // All calls used the same key
        Set<String> distinctKeys = sub5Calls.stream()
            .map(r -> r.getHeader("Idempotency-Key"))
            .collect(Collectors.toSet());
        assertThat(distinctKeys).hasSize(1);
    }
}

Comparison: three Spring Cloud Circuit Breaker failure modes

What Spring Cloud Circuit Breaker is and is not

Spring Cloud Circuit Breaker is a resilience abstraction. It prevents cascading failures by detecting when a downstream service is unhealthy and stopping calls to it for a cooldown period. It is designed to protect against latency and availability problems in downstream services. It is not designed to provide idempotency for calls to those services. The circuit breaker’s state machine (CLOSED, OPEN, HALF_OPEN) describes the application’s health assessment of the downstream service. It does not describe what the downstream service committed in response to any particular call.

The three failure modes in this post arise when developers conflate two distinct concepts: “the circuit breaker reported this call as failed” and “the downstream service did not commit this call.” These two statements are independent. A circuit breaker records a failure when an exception propagates out of the supplier. A Stripe charge is committed when Stripe’s API server processes the POST /v1/charges request and returns HTTP 200. The exception that the circuit breaker sees may have propagated from anywhere in the supplier — from the Stripe SDK (before commitment), from the network layer (after commitment but before the response arrived), or from application code running after the Stripe call (after commitment). Only the first case corresponds to “Stripe did not commit.”

The content-hash idempotency key pattern solves all three modes because it removes the dependency on the circuit breaker’s outcome signal. Whether the circuit reports success, failure, or CallNotPermittedException, the idempotency key for a given logical billing event is the same string on every attempt. If Stripe committed the charge on an earlier attempt, Stripe’s 24-hour idempotency cache returns the original charge on all subsequent attempts with the same key. The circuit breaker’s outcome determines whether the retry happens. The idempotency key determines whether the retry creates a new charge or retrieves the existing one. These two responsibilities should be kept separate.

Keybrake and Spring Cloud Circuit Breaker Stripe integrations

All three failure modes in this post share a detection challenge: the circuit breaker’s fallback returns a controlled value (ChargeStatus.FAILED, ChargeStatus.PENDING_RETRY), and the application continues to function without throwing an unhandled exception. No alert fires. No error appears in the application logs beyond the expected “fallback invoked” warning. The duplicate charge is committed on Stripe and the customer is billed twice, but from the application’s perspective the billing cycle completed “with some transient failures that were retried.”

The Stripe dashboard will show two charges for the same customer within a short time window, each with a distinct idempotency key and a distinct charge ID. In a subscription billing system processing hundreds of subscriptions per cycle, this pattern — one customer with two charges — can be difficult to spot against the noise of normal billing activity. A customer may not notice the duplicate for days, until their bank statement arrives. By then, the root cause (a circuit transition mid-batch, a TimeLimiter timeout, or a proxy-order fallback) may no longer be reproducible from logs.

Keybrake operates as a proxy between your Spring application and Stripe. Every request through the proxy is logged with its idempotency key, the resulting charge ID, and the response latency. The Mode 1 failure (TimeLimiter timeout after ch_A commits) produces a specific signature in Keybrake’s log: a request with idempotency key key_A that received a response (HTTP 200 with ch_A) followed, seconds later, by a request with idempotency key key_B for the same customer and amount. Keybrake flags this as a potential duplicate and records both events in the audit log. The Mode 3 failure (threshold-crossing mid-batch) is similarly detectable: Keybrake sees key_A succeed for sub 8, then later sees key_B for the same sub 8 amount. The per-vendor daily spend cap provides a hard backstop: a stuck retry loop that replays an entire failed batch with new UUIDs on each attempt will trip the cap after a configurable number of charges and halt further Stripe calls until the operator reviews the audit log.