تاب‌آوری: Resilience4j و الگوهای مقاومتResilience: Resilience4j & Fault-Tolerance Patterns

Let's start with a hard truth that anyone who has run a real system in production has felt in their bones: in a distributed system, failure is not an exception — it is a certainty. On your laptop, where everything lives in one process, a method call either works or the code has a bug. But the moment your service reaches across a network to another service, a whole world of danger sits between them: cables, switches, DNS, queues, a slow database, a service that just restarted. This chapter is about how, instead of wishing the network stays healthy, we accept that it will break and plan for it in advance.

You won't just memorize a list of pattern names here. You'll understand what pain each pattern cures, why it exists, and how to implement it with real code in Resilience4j — today's standard resilience library for Java — and how to talk about it in a senior interview.

Part 0 — a few words you must feel before we start

Before any code, a few terms recur throughout this chapter. Let me plant them with analogies now so you don't get lost later.

• Fault: a bad event inside the system — say, the payment service stops responding. Faults always happen; the art is not letting a fault become a failure (a total collapse of the system).
• Resilience: a system's ability to keep working in the presence of faults. A resilient system doesn't stay perfect, but instead of collapsing it degrades gracefully — e.g. showing a cached price instead of the live one.
• Latency: how long a call takes to return. Our enemy isn't always the "error"; very often slowness is more dangerous than errors, because it locks up resources while nobody is even shouting.
• Blast radius: when one piece breaks, how far the damage spreads. The whole philosophy of this chapter is "keep the blast radius small" — don't let one broken service drag the entire system down with it.
• Idempotent: an operation that, run multiple times, has the same effect as running once. Hold onto this word; it is the backbone of retry.

Why do remote calls fail?

Before we reach for cures, we must know the disease well. Junior developers unconsciously assume the network behaves like a local method call: fast, reliable, free. These assumptions are so common and so wrong that they have a name — the Fallacies of Distributed Computing, a list Sun Microsystems engineers compiled in the '90s:

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

Now the key question: why can one broken remote call drag down the entire system? The answer is a chain.

Cascading failure

Suppose service A calls service B, and B calls the database. Now the database slows down and each query, instead of 10 milliseconds, takes 30 seconds. What happens?

• Every thread in B waiting on the database is "locked" for 30 seconds.
• Fresh requests pour in, but there are no free threads; the thread pool fills.
• Now B also responds slowly to everyone, or not at all.
• A's threads waiting on B lock up too, and A goes down.
• Like dominoes, one database's slowness climbs all the way to the end user.

Bottom line: in a distributed system you can't stop your dependencies from breaking; you can only decide how their breakage affects you. That is precisely the definition of resilience.

Meet Resilience4j

Resilience4j is a lightweight fault-tolerance library for Java, inspired by Netflix's Hystrix but built with a more modern philosophy. Its design rests on functional programming: you have a function (say, a remote call), and Resilience4j wraps (decorates) it in a protective layer and hands you back a new, resilient function.

Why did Hystrix retire, and Resilience4j take its place?

Hystrix was the industry standard for years, but Netflix announced in 2018 that it was moving to maintenance mode — no new features, active development stopped. New projects should not adopt it. Let's see the differences in a table:

Modules and setup

Resilience4j has a pure core plus integrations. The main modules:

• resilience4j-circuitbreaker
• resilience4j-retry
• resilience4j-ratelimiter
• resilience4j-bulkhead
• resilience4j-timelimiter
• resilience4j-spring-boot3 — Spring Boot 3 integration (annotations + application.yml config)
• resilience4j-micrometer — metrics for monitoring

For a Spring Boot 3 project, usually this one dependency is enough:

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
    <version>2.4.0</version>
</dependency>
<!-- @CircuitBreaker/@Retry on methods also need AOP -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

Circuit Breaker

This is the most important pattern in the chapter. Its name comes from an electrical fuse, and that analogy makes everything click.

The three main states: closed, open, half-open

The heart of a circuit breaker is a state machine with three states:

• CLOSED: normal mode. All calls pass through, but the breaker silently records each result (success/failure) in a "sliding window."
• OPEN: if the failure rate crosses a threshold, the circuit "opens" and, like a blown fuse, no call passes; everyone is rejected instantly with CallNotPermittedException. This is the fail-fast that saves your threads.
• HALF_OPEN: after a wait period, the breaker permits a limited number of trial calls. If they succeed, it returns to CLOSED; if they fail again, it goes back to OPEN and waits once more.

Resilience4j also has three special, less-used states: DISABLED (always pass), FORCED_OPEN (always reject), and METRICS_ONLY (record only, never break).

        failure rate crossed the threshold
CLOSED ───────────────────────────────────▶ OPEN
   ▲                                           │
   │ trial call succeeded                      │ wait duration elapsed
   │                                           ▼
   └───────────────── HALF_OPEN ◀──────────────┘
        trial call failed ──▶ OPEN

The sliding window: the decision-making brain

How does the breaker measure "failure rate"? With a sliding window, which comes in two types:

• COUNT_BASED: keeps the last N calls. "Of the last 100 calls, what percent failed?"
• TIME_BASED: keeps calls from the last N seconds. "In the past 60 seconds, what percent failed?"

Config and functional code

Let's build a breaker by hand so we see every knob:

CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .slidingWindowType(SlidingWindowType.COUNT_BASED)
    .slidingWindowSize(10)                 // measure the last 10 calls
    .minimumNumberOfCalls(5)               // don't decide until 5 calls
    .failureRateThreshold(50f)             // open if 50% fail
    .slowCallRateThreshold(80f)            // also open if 80% are slow
    .slowCallDurationThreshold(Duration.ofSeconds(2)) // "slow" = over 2s
    .waitDurationInOpenState(Duration.ofSeconds(10))  // stay OPEN for 10s
    .permittedNumberOfCallsInHalfOpenState(3)         // allow 3 trial calls
    .automaticTransitionFromOpenToHalfOpenEnabled(true)
    .recordExceptions(IOException.class, TimeoutException.class)
    .ignoreExceptions(BusinessException.class)         // don't count this as failure
    .build();

CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);
CircuitBreaker breaker = registry.circuitBreaker("paymentService");

// now wrap the function:
Supplier<String> decorated =
    CircuitBreaker.decorateSupplier(breaker, () -> paymentClient.charge(order));

// and run it with a safe fallback:
String result = Try.ofSupplier(decorated)
    .recover(throwable -> "payment-unavailable")   // on failure, return this
    .get();

The same thing with an annotation in Spring Boot

In Spring, instead of the code above, you write the config in application.yml:

resilience4j:
  circuitbreaker:
    instances:
      paymentService:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 10
        minimum-number-of-calls: 5
        failure-rate-threshold: 50
        slow-call-rate-threshold: 80
        slow-call-duration-threshold: 2s
        wait-duration-in-open-state: 10s
        permitted-number-of-calls-in-half-open-state: 3
        automatic-transition-from-open-to-half-open-enabled: true

@Service
public class PaymentService {

    @CircuitBreaker(name = "paymentService", fallbackMethod = "chargeFallback")
    public String charge(Order order) {
        return paymentClient.charge(order);   // real remote call
    }

    // fallback signature = original signature + a trailing Throwable
    public String chargeFallback(Order order, Throwable t) {
        log.warn("payment circuit open, using fallback: {}", t.toString());
        return "payment-deferred";
    }
}

Observing events

The breaker has an event publisher, great for debugging and monitoring:

breaker.getEventPublisher()
    .onStateTransition(e -> log.info("circuit {} -> {}",
        e.getStateTransition().getFromState(),
        e.getStateTransition().getToState()))
    .onCallNotPermitted(e -> log.warn("call blocked, circuit is OPEN"))
    .onError(e -> log.debug("recorded failure: {}", e.getThrowable().toString()));

Retry — trying again (but carefully)

Some errors are transient: a momentary network blip, a random timeout, a brief restart. For these it makes sense not to give up instantly but to try a few more times. But naive retry is more dangerous than the failure itself.

Exponential backoff: wait longer each time

Backoff means waiting between attempts, and exponential backoff means that wait doubles each time: 1 second, then 2, then 4, then 8. The logic: if the service didn't answer after an instant retry, it probably really has a problem and needs more breathing room.

Why jitter (random spread) is vital

Now picture a subtle catastrophe: a thousand services all call one database at the same time, and the database blips for a moment. All thousand get errors simultaneously, and all, with the same backoff, retry together exactly 1 second later — a giant wave that hammers the just-recovering database again. This is called a thundering herd, or a retry storm.

RetryConfig config = RetryConfig.<String>custom()
    .maxAttempts(4)                        // one initial + 3 retries
    .intervalFunction(IntervalFunction
        .ofExponentialRandomBackoff(
            Duration.ofMillis(500),        // start at 500ms
            2.0,                           // doubling multiplier
            0.5))                          // jitter up to 50%
    .retryExceptions(IOException.class, TimeoutException.class)
    .ignoreExceptions(BusinessException.class)
    // retry only when the result is null (retry on result, not just exceptions):
    .retryOnResult(result -> result == null)
    .build();

Retry retry = Retry.of("inventory", config);

Supplier<String> decorated = Retry.decorateSupplier(retry,
    () -> inventoryClient.check(sku));

The application.yml equivalent:

resilience4j:
  retry:
    instances:
      inventory:
        max-attempts: 4
        wait-duration: 500ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2
        enable-randomized-wait: true       # this is jitter
        randomized-wait-factor: 0.5
        retry-exceptions:
          - java.io.IOException
        ignore-exceptions:
          - com.example.BusinessException

Combining retry and circuit breaker

These two are best together, but order matters: usually you want the circuit breaker inside the retry, so when the circuit opens, CallNotPermittedException is thrown fast; and you put that exception in the retry's ignoreExceptions so retry doesn't pointlessly hammer an open circuit.

Rate Limiter

A circuit breaker protects you from a broken service. A rate limiter is the opposite: it protects someone else (or your quota) from you, by capping the number of calls per unit of time.

RateLimiterConfig config = RateLimiterConfig.custom()
    .limitForPeriod(10)                         // 10 permits per period
    .limitRefreshPeriod(Duration.ofSeconds(1))  // refill the quota every 1s
    .timeoutDuration(Duration.ofMillis(500))    // wait at most 500ms for a permit
    .build();

RateLimiter limiter = RateLimiter.of("reportApi", config);

Supplier<String> decorated = RateLimiter.decorateSupplier(limiter,
    () -> reportClient.generate());

If the quota is exhausted during the period and no slot opens within timeoutDuration, RequestNotPermitted is thrown.

Bulkhead

The name bulkhead comes from a ship's watertight partitions, and that analogy carries the whole idea.

Resilience4j has two kinds of bulkhead:

• SemaphoreBulkhead: uses a counter (semaphore) to cap concurrent calls. Lightweight, no extra threads; the call runs on the caller's own thread.
• ThreadPoolBulkhead: creates a dedicated thread pool and queue for this dependency. The call runs on another thread, giving fuller isolation and the ability to truly cancel on timeout — but it's heavier and works only with asynchronous calls (CompletableFuture).

// semaphore type
BulkheadConfig config = BulkheadConfig.custom()
    .maxConcurrentCalls(20)                     // at most 20 concurrent calls
    .maxWaitDuration(Duration.ofMillis(100))    // wait 100ms for a free slot
    .build();

Bulkhead bulkhead = Bulkhead.of("searchService", config);

Supplier<String> decorated = Bulkhead.decorateSupplier(bulkhead,
    () -> searchClient.query(term));

If all slots are full and none frees within maxWaitDuration, BulkheadFullException is thrown.

Time Limiter

Remember we said the real enemy is slowness? The time limiter targets it directly: don't give a call more than X seconds. If no answer arrives, cut the call and throw a TimeoutException.

Unlike the others, the time limiter works on a CompletableFuture (an asynchronous call), because to truly cancel a running task you need a separate thread:

TimeLimiterConfig config = TimeLimiterConfig.custom()
    .timeoutDuration(Duration.ofSeconds(3))     // don't wait beyond 3s
    .cancelRunningFuture(true)                   // actually cancel the future
    .build();

TimeLimiter timeLimiter = TimeLimiter.of("mapsApi", config);
ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();

Supplier<CompletableFuture<String>> futureSupplier =
    () -> CompletableFuture.supplyAsync(() -> mapsClient.route(from, to));

Callable<String> decorated = TimeLimiter.decorateFutureSupplier(
    timeLimiter, futureSupplier);

In Spring, @TimeLimiter works only on methods that return a CompletableFuture:

@TimeLimiter(name = "mapsApi")
@CircuitBreaker(name = "mapsApi", fallbackMethod = "routeFallback")
public CompletableFuture<String> route(String from, String to) {
    return CompletableFuture.supplyAsync(() -> mapsClient.route(from, to));
}

public CompletableFuture<String> routeFallback(String from, String to, Throwable t) {
    return CompletableFuture.completedFuture("route-unavailable");
}

Fallback — the plan B

A fallback means "when the main path fails, instead of throwing an error at the user, give a graceful alternative answer." This is graceful degradation.

Kinds of good fallbacks:

• Cached value: return the last successful value (e.g. a price from 5 minutes ago).
• Default value: a conservative, safe response.
• Functional degradation: turn off part of the capability but keep the rest alive (e.g. hide "personal recommendations" but still show the product).

Combining decorators — and why order matters

Resilience4j's real power is unleashed when you stack several protections. The Decorators class makes this fluent:

Supplier<String> resilient = Decorators.ofSupplier(() -> paymentClient.charge(order))
    .withBulkhead(bulkhead)              // innermost
    .withTimeLimiter(timeLimiter, scheduler)
    .withRateLimiter(rateLimiter)
    .withCircuitBreaker(circuitBreaker)
    .withRetry(retry)                    // outermost
    .withFallback(List.of(CallNotPermittedException.class),
                  t -> "payment-fallback")
    .decorate();

The order of layers gives different meanings. In Spring, the default aspect order is (outer to inner):

Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( the function ) ) ) ) )

Idempotency — why retry is a bomb without it

I promised to come back to this, because without grasping it, retry causes disease instead of curing it.

The fundamental problem: when a call times out, you don't know whether the server didn't do the work, or did it but the response got lost. If the operation is idempotent, it doesn't matter and you retry safely. If it isn't, retry might deduct money twice or place two orders.

The standard fix is an idempotency key: the client generates a unique id (e.g. a UUID) for each logical operation and sends the same one across all retries; if the server has seen that key before, it returns the previous result instead of executing again.

public PaymentResult charge(Order order) {
    // a stable key for this logical operation; constant across all retries
    String idempotencyKey = order.getId() + ":charge";
    return httpClient.post("/charge")
        .header("Idempotency-Key", idempotencyKey)   // server dedupes with this key
        .body(order)
        .retrieve();
}

Testing resilience

Writing resilient code isn't enough; you must prove it works. Three levels of testing:

1) Unit-testing the state machine itself. You can force the breaker through state transitions and check its behavior:

@Test
void breakerOpensAfterFailures() {
    CircuitBreaker cb = CircuitBreaker.of("test",
        CircuitBreakerConfig.custom()
            .slidingWindowSize(4)
            .minimumNumberOfCalls(4)
            .failureRateThreshold(50f)
            .build());

    // simulate 4 failing calls
    for (int i = 0; i < 4; i++) {
        cb.onError(0, TimeUnit.MILLISECONDS, new IOException("boom"));
    }

    assertThat(cb.getState()).isEqualTo(CircuitBreaker.State.OPEN);
    // now every call should be rejected:
    assertThatThrownBy(() -> cb.executeSupplier(() -> "x"))
        .isInstanceOf(CallNotPermittedException.class);
}

2) Integration testing with a fake server. With a tool like WireMock you spin up a fake HTTP server and deliberately make it delay or error, to see whether the real circuit breaker and retry react correctly:

// make WireMock delay 3 seconds so the time limiter fires
stubFor(get("/inventory")
    .willReturn(aResponse()
        .withFixedDelay(3000)
        .withStatus(200)
        .withBody("ok")));
// or repeated errors to open the circuit:
stubFor(get("/inventory").willReturn(aResponse().withStatus(500)));

3) Observing in production. With resilience4j-micrometer, feed metrics to Prometheus/Grafana to watch circuit states, failure rates, and retry counts live. Without observability, you can't tell when your circuit even opens and closes.

Pitfalls and best practices

• Don't blindly accept defaults. The default slidingWindowSize is 100 and minimumNumberOfCalls is also 100; if your traffic is low, the circuit effectively never opens. Align config with real traffic.
• Never use backoff without jitter — you'll create a retry storm.
• Don't retry non-idempotent operations — or solve idempotency first.
• Keep fallbacks light and harmless — not another remote call.
• Choose decorator order deliberately — retry outside, breaker inside.
• Observe everything — an open circuit nobody sees is as bad as having no circuit at all.
• Don't treat breaker and timeout as separate concerns — catch slowness with the slow-call threshold and the time limiter, not just failure counting.
• Set ignoreExceptions correctly — business errors (like "insufficient balance") should not open the circuit; they aren't infrastructure failures.

Interview Questions

Now it's time to drill everything with real senior-interview questions. Answer each yourself first, then open the answer.