میکروسرویس، سیستم‌های توزیع‌شده و طراحی سیستمMicroservices, Distributed Systems & System Design

Let us be honest: "microservices" is one of those words everyone puts on a résumé and few can define precisely in an interview. This chapter is going to move you from someone who has heard these patterns to someone who knows which pain each one cures, what it costs, and when NOT to use it. I will never drop a term cold — the moment you see a new word like idempotent, partition, or linearizable, I unpack it in the same breath with an analogy.

Part 0 — words you must know first

Before we start, let us nail a few base words with analogies so we never stumble later.

• Atomic transaction: "all or nothing." Like transferring money between two accounts; either both halves (debit one, credit the other) happen together or neither does. No half-done state allowed.
• ACID: the four guarantees of a classic database — Atomicity, Consistency (data rules stay valid), Isolation (transactions don't interfere), Durability (once confirmed, data survives even a power cut).
• hop: every time a request crosses the network from one service to another, that's a hop. Like a phone call between two offices; each call takes time and can drop.
• latency vs throughput: latency is "how long one request takes" (delivery time of one pizza); throughput is "how many you can serve per unit time" (pizzas per hour). They are different axes.
• node: one machine/server in the distributed system — one branch office of the company.
• replica: a copy of data on another node, so if one breaks or gets busy, others can answer.
• partition (network partition): when network communication between two groups of nodes is cut but the nodes themselves are alive — like the phone line between two branches going down while both branches stay open and busy.
• idempotent: an operation that yields the same result whether you apply it once or a hundred times. An elevator's "floor 3" button is idempotent: press it a hundred times, you still go to floor 3.

Distribution is a cost, not a goal

A monolith that fits in one process gives you three luxuries for free: atomic transactions, synchronous in-memory calls (call a function, get the answer right there), and one deployable that's easy to reason about.

Every network boundary you add sells those three away in exchange for independent scaling and independent deployment. So the governing question of this whole chapter is:

Is the organizational and scaling benefit of this split worth giving up the free ACID transaction and the reliable function call?

If you can't articulate the benefit of a given split in a sentence, don't split.

The eight fallacies of distributed computing

These eight false beliefs (the "fallacies of distributed computing," from Deutsch and Gosling) are a checklist every design must respect. Each is a comfortable assumption that reality violates:

1. The network is not reliable (packets get lost).
2. Latency is not zero (round trips take time).
3. Bandwidth is not infinite.
4. The network is not secure.
5. Topology does change (nodes come and go).
6. There is not one administrator.
7. Transport cost is not zero.
8. The network is not homogeneous (machines and versions differ).

Decomposition and the distributed-monolith trap

Good service boundaries follow business capabilities and Domain-Driven Design bounded contexts, not technical layers. "Order service, Payment service, Inventory service" is a capability split. "Controller service, Repository service, Database service" is a layered split that just adds network hops with none of the autonomy.

What's a bounded context? A boundary inside which a word has one precise meaning. "Customer" in Sales is a different model from "Customer" in Support. Each service speaks its own language and model inside its border and doesn't leak them.

The failure mode to fear is the distributed monolith: the worst of both worlds — the complexity of distribution without its independence. Symptoms:

• Shared database. Two services writing the same tables means you can no longer change a schema, deploy, or scale independently. Every change needs everyone's coordination. This is the single biggest anti-pattern.
• Chatty synchronous chains. If serving one request means A → B → C → D synchronously, your availability is the product of each hop's availability, and your latency is the sum. A number: if each service is 99% up, 0.99⁴ ≈ 0.96 — a four-hop chain is less trustworthy than any of its links.
• Lockstep releases. If you can't deploy service X without simultaneously deploying Y, they are really one service wearing two hats.

Heuristics for decomposition: start coarse-grained (a "modular monolith" — one deployable but with clean, well-bounded internal modules) and extract a service only when a module has one of these three: a distinct scaling profile, a distinct team, or a distinct rate of change.

CAP and PACELC — state the theorem precisely

CAP theorem: during a network partition (P), a distributed system must choose one of:

• Consistency (C): every read sees the latest committed write. This property is called linearizability — it's as if only one copy of the data exists and everyone sees it in the same real-time order.
• Availability (A): every request gets a non-error response (not necessarily the freshest data, but no error).

While partitioned you can't have both. The common mistake is treating CAP as "pick two of three." But partitions are not optional; they happen in the real world. So the real choice is: CP vs AP, when a partition occurs.

• CP (e.g., a strongly-consistent store like ZooKeeper/etcd, or a single-leader RDBMS): on partition, the minority side refuses writes so data doesn't diverge. You lose availability, keep correctness.
• AP (e.g., Dynamo-style stores, or Cassandra with low consistency): on partition, both sides keep serving, and you reconcile later. You keep availability, accept the risk of stale or conflicting reads.

PACELC — the fuller lens

CAP only talks about the moment of partition, but partitions are rare. PACELC (from Daniel Abadi) completes the picture:

If Partition, choose A or C; Else (normal operation), choose Latency or Consistency.

This is the more useful day-to-day lens, because partitions are rare but the latency-vs-consistency trade is always present. To guarantee a read sees the latest write, you must wait for several replicas to coordinate — and that wait is latency.

• Cassandra by default is PA/EL: on partition favors availability, in normal operation favors low latency (speed).
• A traditional RDBMS is usually PC/EC: favors consistency in both states.
• Tune Cassandra with QUORUM reads+writes (a majority of replicas must acknowledge) and it moves toward EC, at the cost of higher latency.

Consistency models

Consistency isn't an on/off switch; it's a spectrum, from strongest (most expensive) to weakest (cheapest and most scalable):

Most large systems are eventually consistent with targeted strong consistency only where money or safety demands it. "Strong everywhere" doesn't scale (too expensive); "eventual everywhere" corrupts invariants (rules that must always hold, like "balance never goes negative"). The senior skill is choosing consistency per operation, not one global decision.

Idempotency and the "exactly-once" myth

There is no exactly-once delivery over an unreliable network. A sender that gets no ack can't tell whether the message was lost or the ack was lost, so it must retry — meaning the receiver sometimes sees duplicates. What you can build is effectively-once processing:

Effectively-once processing = at-least-once delivery + an idempotent handler.

This distinction is one of the most popular interview traps. You can't make delivery once; you can make the effect of processing once.

Idempotency means applying an operation N≥1 times has the same effect as applying it once. Three techniques:

1. Idempotency keys. The client generates a unique key per logical operation; the server records the key + result. On retry with the same key, it returns the stored result instead of re-executing. Stripe's API works exactly this way.
2. Dedup table / inbox pattern. The consumer stores processed message IDs (with a TTL or watermark) and skips duplicates.
3. Natural idempotency. SET balance = 500 is inherently idempotent (run it any number of times, it's 500); balance += 100 is not (it adds each time). So design commands to carry absolute state or a version, not a relative delta.

// Idempotent charge endpoint using an idempotency key + a unique constraint.
@Transactional
public ChargeResult charge(String idempotencyKey, Money amount, String account) {
    // Fast path: was this exact request already processed?
    var existing = idempotencyRepo.findByKey(idempotencyKey);
    if (existing != null) {
        return existing.result();          // return prior result, do NOT charge again
    }
    var result = ledger.debit(account, amount);
    try {
        // UNIQUE(idempotency_key) in DB is the real guard against a race
        idempotencyRepo.save(new IdempotencyRecord(idempotencyKey, result));
    } catch (DataIntegrityViolationException race) {
        // Concurrent duplicate won the insert; reload and return its result
        return idempotencyRepo.findByKey(idempotencyKey).result();
    }
    return result;
}

Distributed transactions: Saga and the outbox

In a monolith you wrapped everything in one @Transactional and were done. But across the databases of several microservices you can't use two-phase commit (2PC): it's blocking, it ties the whole transaction's availability to the slowest participant, and it locks resources across the network. Instead you use a Saga.

Two coordination styles for a Saga:

• Choreography: each service reacts to events and emits the next event. Decentralized, no central brain — like dancers who each take their next move from the one beside them. But the overall flow is implicit and hard to trace.
• Orchestration: a central coordinator (a state machine) issues commands and handles compensations — like a conductor. Explicit and observable, but the orchestrator is an extra dependency.

The dual-write problem and the outbox solution

The genuinely hard part of a Saga: how do you publish an event and atomically commit your DB write together?

The solution: the Transactional Outbox. Write the domain change and an outbox row in the same local database transaction. Now either both commit or neither does — it's one database, one transaction. Then a separate relay (either polling the outbox table, or CDC — Change Data Capture — like Debezium, which reads the database's change stream) picks up the outbox rows and publishes them to Kafka.

-- One local transaction writes both the state change and the event to publish.
BEGIN;
  UPDATE orders SET status = 'CONFIRMED' WHERE id = 42;
  INSERT INTO outbox (id, aggregate, type, payload, created_at)
  VALUES (gen_random_uuid(), 'order', 'OrderConfirmed',
          '{"orderId":42}', now());
COMMIT;
-- A separate relay (Debezium CDC or a poller) ships outbox rows to Kafka,
-- deletes/marks them, and retries on failure => at-least-once.

Because the relay retries on failure, delivery is at-least-once — which loops us right back to the previous lesson: so consumers must be idempotent. These patterns chain together.

CQRS and event sourcing

CQRS stands for Command Query Responsibility Segregation. Simplify it: separate the write model from the read model.

Writes go through a normalized, invariant-enforcing model (it guarantees the rules). Reads are served from one or more denormalized projections, each optimized for a specific query — a search index here, a wide table there.

• Benefits: you scale reads independently; you shape the read store per use-case; each side gets simpler.
• Cost: the read side is eventually consistent with the write side. After a write, the UI may show slightly stale data for a moment, and you must design for it (e.g., optimistically return the command's own result to the user).

Event sourcing goes one step further: it stores state as an append-only log of events — "OrderPlaced," "ItemAdded," "OrderShipped" — rather than current rows.

Benefits: a perfect, complete audit log; time-travel and debugging; trivially building new projections just by replaying events; and a natural fit with the outbox. But the costs are heavy, and interviewers probe exactly these:

• Event schema evolution: events are immutable and forever; when an event's format changes you need versioning/upcasting (you "upgrade" an old event to the new format when reading it).
• Rebuilding projections from a long log is expensive; you need snapshots (a point-in-time capture of state, so you replay only from there).
• No easy ad-hoc query over the event store; you must query the projections.
• GDPR's "right to be forgotten" conflicts with an immutable log. The usual escape is crypto-shredding: encrypt the data, and to "delete" it, just throw away the encryption key so the data is permanently unreadable.

The five resilience patterns

Mental rule: the caller must protect itself. Never assume your dependencies are up and fast. You have five tools.

1. Timeout

Every network call needs a timeout. A missing timeout is a latent outage: one slow dependency occupies your thread pool one by one until it's exhausted, and then the failure goes cascading — your service dies too because all its threads are stuck waiting on that slow dependency. Key point: set the timeout from the client's latency budget (how long the customer will wait), not from the server's happy path.

2. Retry — with backoff and jitter

Retry only idempotent operations and only retryable errors (timeouts, 503 — not 400 or 409, which won't fix on repeat). Naive fixed-interval retries cause two disasters:

• Retry storm: everyone multiplies the extra load onto a service that is right now collapsing.
• Thundering herd: without jitter, all clients retry at exactly the same instant and act like a DDoS against your own recovering service.

The fix: exponential backoff with jitter (grow the intervals exponentially and scatter them randomly), plus caps on the number of attempts and total time.

3. Circuit breaker

Track the failure rate to a dependency; when it crosses a threshold, open the circuit and fail fast immediately (no calls reach the sick service). After a cooldown, allow a few half-open trial calls; if they succeed, the circuit closes and returns to normal. This both saves your resources from being wasted on a dead service and gives that service room to recover.

4. Bulkhead

Isolate resources — a separate thread pool, connection pool, or semaphore per dependency — so one saturated downstream can't consume all your threads and drown unrelated traffic too. If the "recommendations" service slows down, it must not drag the "payments" service down with it.

5. Backpressure

When you can't keep up with the input, instead of buffering unboundedly until OOM (out of memory), signal upstream to slow down: bounded queues, rejecting with 429 (Too Many Requests), or demand signals in Java's Flow and Reactor. Backpressure's blunt cousin is load shedding: deliberately dropping low-priority work early so the system doesn't die under load.

All together in Resilience4j

// Resilience4j: exponential backoff with jitter + circuit breaker, composed.
IntervalFunction backoff = IntervalFunction
    .ofExponentialRandomBackoff(Duration.ofMillis(100), 2.0, 0.5); // base, multiplier, jitter
RetryConfig retryCfg = RetryConfig.custom()
    .maxAttempts(3)
    .retryExceptions(TimeoutException.class, IOException.class) // only retryable
    .intervalFunction(backoff)
    .build();
CircuitBreakerConfig cbCfg = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)                       // open at 50% failures
    .slowCallRateThreshold(80)
    .slowCallDurationThreshold(Duration.ofSeconds(1))
    .waitDurationInOpenState(Duration.ofSeconds(10)) // cooldown before half-open
    .slidingWindowSize(20)
    .build();

var cb = CircuitBreaker.of("inventory", cbCfg);
var retry = Retry.of("inventory", retryCfg);

Supplier<Stock> call = () -> inventoryClient.getStock(sku); // has its own timeout
Supplier<Stock> guarded = Retry.decorateSupplier(retry,
        CircuitBreaker.decorateSupplier(cb, call));         // CB inside retry
Stock stock = Try.ofSupplier(guarded)
        .recover(ex -> Stock.unknown())                     // graceful fallback
        .get();

API gateway, BFF, and service mesh

Three tools for managing traffic at two different boundaries: the edge (between clients and services) and the interior (between services).

• API gateway. A single entry point handling cross-cutting edge concerns: TLS termination (decrypting HTTPS), authentication/authorization (authn/authz), rate limiting, routing, and request aggregation. It keeps clients ignorant of internal topology. Risk: it becomes a smart, bottlenecked monolith — keep business logic out of it.
• BFF (Backend for Frontend). A gateway variant with one backend per client type (web, iOS, Android). Each BFF tailors payload shape and aggregation to its own client, avoiding a lowest-common-denominator API and chatty mobile calls. A phone on a weak network shouldn't fire 10 requests; the BFF sends one and aggregates the rest behind the scenes.
• Service mesh (e.g., Istio or Linkerd). It pulls service-to-service concerns — mTLS (mutual encryption between services), retries, timeouts, traffic splitting, telemetry — out of your code and into sidecar proxies (like Envoy) beside each pod. So you get uniform resilience and observability without library sprawl across languages — at the cost of operational complexity and a little added latency per hop. Prefer it when you have many services in many languages; a small homogeneous fleet can just use libraries.

Observability: logs, metrics, traces

You have three pillars:

• Logs — discrete events. Make them structured (JSON) and always carry a correlation / trace ID, so you can stitch one request together as it crosses several services.
• Metrics — aggregatable numbers (counters, gauges, histograms). Cheap and great for alerting and trends. Watch percentiles like p95/p99, never the average — the average hides the slow tail that the user actually feels.
• Traces — the causal path of one single request across services, with the timing of each span (each unit of work). OpenTelemetry is the vendor-neutral standard for all three pillars; propagate context via W3C traceparent headers. Distributed tracing is how you find which hop in the chain is slow.

Keep two classic metric sets: the RED method for services (Rate, Errors, Duration) and the USE method for resources (Utilization, Saturation, Errors). And crucially: alert on SLOs and error budgets (service-level objective and allowed error budget), not on raw CPU — the user doesn't care about a CPU number, they care that the service works.

A reusable system-design interview framework

In a system-design interview, the interviewer grades your process as much as the final answer. Every time, follow this same six-step skeleton:

1. Requirements & scope. Clarify functional requirements first (what the system does), then the non-functional (NFR) ones: scale, latency, availability, consistency, durability. And explicitly state what's out of scope.
2. Estimation (back-of-envelope). QPS (peak ≈ 2–3× average; read:write ratio), storage per day and per 5yr, bandwidth, cache memory. Use round numbers; the point is to justify component choices, not astronomical precision.
3. API design. The key endpoints/contracts (REST/gRPC/streaming), request/response shapes, and idempotency.
4. High-level architecture & data model. Draw the components and pick storage per access pattern. Choose SQL vs NoSQL by the query shape, and pick a partition key deliberately (the key that spreads data across nodes).
5. Scaling & deep dives. Caching layers, sharding (splitting data into pieces), replication, load balancing, async processing, geo-distribution.
6. Bottlenecks, failure modes, trade-offs. Where does it break at 10× load? What happens on a node or region failure? Name your CAP/PACELC choice out loud.

Worked example: real-time fleet tracking

Let's run this framework on a real problem.

1. Requirements. 1,000,000 vehicles emit GPS every 5 seconds. Features: show a vehicle's live position, replay a trip's history, and geofencing alerts (when a vehicle leaves a geographic zone). NFRs: live position may be eventually consistent (a couple seconds stale is fine → PACELC EL on the live path); trip history must be durable; the write path must never drop a point; and p99 map query must be < 300 ms.

2. Estimation.

• Write QPS = 1,000,000 / 5 = 200,000 writes/sec. Peak ≈ 2× = 400k/sec. This system is write-heavy — and that's the governing force of the entire design.
• Each point ≈ 50 bytes (id, lat, lon, ts, speed). Per day: 200k × 86,400 × 50 B ≈ 864 GB/day raw → time-series compression makes this far smaller.
• Live state: 1M vehicles × ~100 B latest position ≈ 100 MB → fits easily in memory (Redis).

3. API.

POST /v1/positions            # batched ingest from devices (idempotent by (vehicleId, ts))
GET  /v1/vehicles/{id}/live   # latest position, served from cache
GET  /v1/vehicles/{id}/trip?from=..&to=..   # historical path (time-series store)
WS   /v1/stream?bbox=...       # push updates for vehicles in a map viewport

4. Architecture.

Devices --> Load Balancer --> Ingest service --> Kafka (partitioned by vehicleId)
                                                   |            |
                                        (consumer) |            | (consumer)
                                                   v            v
                                       Redis (latest pos)   Time-series DB
                                       geo-index/hot state  (Cassandra/Timescale)
                                                   |
   Map clients <-- WebSocket fanout <-- Stream service <-- Kafka

• Ingest is stateless and horizontally scaled behind the LB; it only validates and produces to Kafka, absorbing spikes and giving a durable at-least-once buffer (backpressure lives here — bounded, and Kafka itself is the buffer).
• Partition Kafka by vehicleId so all points for one vehicle stay ordered and one hot vehicle can't reorder another.
• Hot path: a consumer updates Redis with each vehicle's latest position on every point (Redis's GEO index supports "vehicles in this bounding box"). GET /live reads Redis → fast, eventually consistent.
• Cold path: another consumer writes to a time-series store (TimescaleDB or Cassandra) partitioned by (vehicleId, day) — an ideal key for "vehicle X's trip on day D" that also spreads load. Old partitions roll off to object storage (tiered retention).
• Live map: clients subscribe over WebSocket by viewport (the map's visible area); the stream service filters Kafka updates down to that client's bounding box.

5. Scaling & bottlenecks.

• The write path scales by adding Kafka partitions and consumer instances; the choke point is the time-series DB's write throughput → batch the writes and pick a high-ingest store.
• Idempotency: device retries can duplicate points; dedup on (vehicleId, ts).
• Hot partition: a viewport over a dense city can overload one stream server → shard the fanout by geohash (geographic code), not by vehicle.
• Failure modes: if the time-series DB is down, Kafka retains the data (retention = your durability buffer) and consumers catch up later — and the live path (Redis) still works. If Redis dies, rebuild latest state by consuming the tail of Kafka. This is the payoff of putting a durable log at the center of the design.
• CAP stance: the live path is AP/EL (serve possibly-stale positions, never block); ingest+Kafka is durable and ordered per key. State this explicitly — it's the answer to "what consistency does this give?"

Pitfalls & gotchas (memorize these)

• Splitting into microservices for résumé reasons, ending with a distributed monolith and a shared DB.
• No timeout on a call → thread-pool exhaustion → cascading failure.
• Retrying non-idempotent operations → double charges / duplicate orders.
• Retries without jitter → synchronized retry storms that DDoS your own recovering service.
• Believing "exactly-once delivery" exists; it doesn't — engineer idempotent consumers.
• Dual-write (DB then Kafka) without an outbox → lost or phantom events.
• Alerting on averages instead of p99; the tail is what users experience.
• Distributed locks used for correctness without fencing tokens (a numbered token that increments each time). Why? If a paused GC (a brief stop-the-world for garbage collection) runs long, a service thinks it still "holds" the lock while it has actually expired and been handed to another — and without a fencing token its stale write corrupts the data.

Best practices

• Default to a modular monolith; extract a service only when scaling/team/change-rate demands it.
• One database per service; integrate via APIs and events, never a shared schema.
• Make every consumer idempotent and every producer use the outbox pattern.
• Timeouts on everything; retries only for idempotent + retryable, always with backoff + jitter; wrap flaky deps in circuit breakers and bulkheads.
• Propagate a trace ID end-to-end; instrument with OpenTelemetry; alert on SLOs.
• Choose consistency per operation, and write down the CAP/PACELC posture of each path.

Interview Questions

Answer each in your head like an exercise, then check the reference answer.

public void transfer(Account from, Account to, Money amt) {
    inventoryService.reserve(from, amt); // remote call, no timeout
    to.credit(amt);                       // local
}