کتابِ راهِ سنیور: نکاتِ واقعیِ پروداکشنThe Senior Playbook: Real-World Production Wisdom

Let's start with an uncomfortable truth: most of what you learned in tutorials and courses is either wrong or incomplete in production. Not because the instructor lied, but because a tutorial shows the happy path, and production is where everything breaks at once: the network gets slow, the database locks under pressure, a downstream service times out, a deploy stalls halfway, and right at that moment the pager goes off at 3 a.m.

A senior isn't someone who has memorized every pattern. A senior is the person who, when the team excitedly says "let's make this a microservice," asks "why? what problem does it solve? have you counted the cost?" — and when everyone says "monoliths are bad," calmly replies "a good monolith beats a bad microservice."

This chapter isn't going to teach you a new pattern. It's going to give you judgment: when something is good, when that same thing is a disaster, and how to articulate all of this loudly and clearly in an interview and in a design meeting.

Part One: Anti-patterns — where teams fall down

Before you learn what to do, you must know what not to do. I've seen each anti-pattern below set a real team back by months. Learn their names; because in an interview, when you can name a bad decision with the right term, you sound like a senior.

1) The Distributed Monolith

The distributed monolith is the worst of both worlds: you pay the operational cost of microservices (network, separate deploys, distributed debugging) but reap none of the benefits (independent deployability, independent scaling, fault isolation). Symptoms:

• To release one feature, you must deploy several services together in a specific order.
• Changing one field in one service breaks the build of three others.
• Services call each other in a synchronous chain: A waits on B, B waits on C, C waits on D.
• Everyone depends on one shared library for data models, so updating that library means updating everyone.

The right way: draw boundaries around business capabilities, not technical layers. Make inter-service communication asynchronous and event-based wherever possible, to break the synchronous chain.

Let's see the difference in topology.

Diagram: distributed monolith (synchronous chain) vs. event-driven independence — تفاوت زنجیره‌ی همگام با استقلالِ رویدادمحور.

flowchart LR
  subgraph Bad[Distributed Monolith]
    A1[Order] --> B1[Inventory]
    B1 --> C1[Pricing]
    C1 --> D1[Notification]
  end
  subgraph Good[Event-Driven]
    A2[Order] -- OrderPlaced --> K[(Event Bus)]
    K --> B2[Inventory]
    K --> C2[Pricing]
    K --> D2[Notification]
  end

On the left, total latency is the sum of everyone's latency, and if Notification falls over, the whole order-placement falls over. On the right, Order just publishes an event and responds immediately; the others consume in their own time, and one failure doesn't take the rest down.

2) Chatty Services — the same N+1, now over the network

In a monolith, a method call is a few nanoseconds. Over the network, each call is hundreds of microseconds to several milliseconds — thousands of times more expensive, and each one can fail. The code below looks innocent but is a killer in production:

// Anti-pattern: N+1 over the network
List<Order> orders = orderClient.findRecent(userId);      // 1 network call
for (Order o : orders) {
    // one call per order — 50 orders means 50 round-trips
    Product p = productClient.getById(o.productId());     // N network calls
    o.setProductName(p.name());
}

If each call is 20ms and the list has 50 items, this loop alone adds 1 second of latency — and that's optimistic. The fix is to design APIs for batch and coarse-grained responses:

// Right: one batch call instead of N
List<Order> orders = orderClient.findRecent(userId);
Set<Long> ids = orders.stream().map(Order::productId).collect(toSet());
Map<Long, Product> products = productClient.getByIds(ids);   // 1 call
orders.forEach(o -> o.setProductName(products.get(o.productId()).name()));

3) The Shared Database

The golden rule of microservices: each service owns its database. Not just separate tables, but ideally a separate schema or instance, and no service ever touches another service's tables directly. Why so strict?

• Independent evolution: if service A can query B's table, then B can never change that table's structure without fearing it breaks A. The schema becomes an unintended public API — the worst kind of coupling, because it's hidden.
• Data ownership: business invariants must be enforced in one place. If two services both write to the account table, the "balance must not go negative" rule is duplicated in two places, and one day one of them forgets it.

For cross-service reporting, the common pattern is that each service ships its events into a data pool. A reporting query against a read model, working in both dialects:

-- PostgreSQL: pagination over a read model
SELECT order_id, product_name, total
FROM   reporting.order_summary
WHERE  created_at >= now() - interval '7 days'
ORDER  BY total DESC
LIMIT  20 OFFSET 40;

-- Oracle: same query, standard pagination
SELECT order_id, product_name, total
FROM   reporting.order_summary
WHERE  created_at >= SYSTIMESTAMP - INTERVAL '7' DAY
ORDER  BY total DESC
OFFSET 40 ROWS FETCH FIRST 20 ROWS ONLY;

4) Nano-services — when you slice too thin

A nano-service is one so small that the cost of its existence (a separate process, network, deploy, monitoring, database) exceeds the value it produces. A service that does one function — say "currency conversion" or "date formatting" — should probably be a library, not a service.

5) Entity Services (CRUD-as-a-Service)

This one is subtler. An entity service is built around a table, not a capability: "UserService," "OrderService," "ProductService," each doing only CRUD on one entity, while all the business logic lives in an "orchestrator" on top of them.

The problem: the business logic isn't in any service — it's between them. Every real operation becomes chatty calls and distributed transactions. This is effectively a distributed monolith wearing a "clean" mask.

When are microservices simply wrong?

Microservices are usually wrong when:

• The team is small (under ~15–20 people): a well-built modular monolith with clean internal boundaries gives you 90% of the benefit at 10% of the cost.
• The domain is still immature: if you don't yet know where the right boundaries are (and in an early startup you never do), you'll cast wrong service boundaries in concrete. Moving a boundary in a monolith is a refactor; in microservices it's a multi-month project.
• You lack operational maturity: without automated CI/CD, good observability, and an on-call culture, microservices just mean distributed darkness.

Part Two: Production concerns — where a senior really shines

This is where the line is drawn between "someone who writes code" and "someone who keeps a system alive." Writing code is easy; keeping the system up at 3 a.m., when you're missing half the information, is an art.

Observability in practice

Observability is the ability to answer questions you didn't know in advance you'd need to ask, purely from the system's outputs. The three classic pillars: logs (what happened), metrics (how much, how many, how fast), traces (where one specific request went).

Correlation IDs and distributed tracing

The single biggest debugging pain in a distributed system is this: a user says "my order didn't go through," and you must trace that one request across the logs of 8 services. Without a shared identifier, that's impossible.

A correlation ID (or trace ID) is a unique identifier generated at the edge of the system (the gateway) that travels with every subsequent call — HTTP or message-on-a-queue alike. Now a single grep shows you that request's entire journey.

The good news: in modern Spring Boot this is nearly automatic. With Micrometer Tracing, you just add a bridge, and Boot injects traceId and spanId into the logging MDC (Mapped Diagnostic Context) and propagates them across service boundaries.

<!-- Bridge to a tracing system: either Brave/Zipkin or OpenTelemetry -->
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<!-- exporter: e.g. to Zipkin, or via OTLP to Tempo/Jaeger -->
<dependency>
  <groupId>io.zipkin.reporter2</groupId>
  <artifactId>zipkin-reporter-brave</artifactId>
</dependency>

Configure the log pattern so these IDs are visible:

# application.yml
logging:
  pattern:
    # [appName,traceId,spanId] — Boot reads these from the MDC
    level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]"
management:
  tracing:
    sampling:
      probability: 1.0    # everything in dev; usually 0.05–0.1 in prod

When a request moves between services, the RestClient or WebClient that Spring builds automatically carries the propagation headers (like traceparent in the W3C standard) — provided you get that client from Spring's factory bean, not via new. If you build the client by hand, that chain breaks.

Diagram: one shared trace ID flowing through several services — یک trace ID مشترک از میانِ چند سرویس.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant O as OrderSvc
    participant P as PaymentSvc
    participant K as Kafka
    C->>G: POST /orders
    Note over G: generate traceId=abc123
    G->>O: traceparent: abc123
    O->>P: traceparent: abc123
    P-->>O: 200
    O->>K: OrderPlaced (traceId=abc123 in header)
    O-->>G: 201
    G-->>C: 201

The golden point: propagation must survive even the message queue. When you publish OrderPlaced to Kafka, put the traceId in the message header so the consumer can continue the trace. Otherwise the request's trail is lost exactly where it goes async.

RED and USE — which metrics do I watch?

A beginner monitors everything and therefore sees nothing. A senior knows which few metrics truly matter. Two standard frameworks:

In Spring, Micrometer gives these metrics almost for free. Adding micrometer-registry-prometheus creates an endpoint at /actuator/prometheus that exports all HTTP metrics (including http.server.requests with status/uri tags and quantiles):

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus,metrics
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true   # histogram for Prometheus-side quantiles
      slo:
        http.server.requests: 100ms,300ms,500ms

Adding a business metric is just as easy — and a senior knows business metrics (like "failed-order rate") are sometimes more important than technical ones:

@Service
public class OrderService {
    private final Counter failedOrders;

    public OrderService(MeterRegistry registry) {
        this.failedOrders = Counter.builder("orders.failed")
            .description("Orders that failed to place")
            .tag("channel", "web")
            .register(registry);
    }

    public void place(Order order) {
        try {
            // ... business logic ...
        } catch (PaymentDeclinedException e) {
            failedOrders.increment();
            throw e;
        }
    }
}

On-call and incident management basics

A day comes when the pager rings and you're responsible. This is taught to engineers less often than anything, yet it's exactly the part that separates "senior" from "good programmer."

Concepts you must know:

• SLI / SLO / Error Budget: an SLI is what you measure (e.g. "percentage of requests under 300ms"). An SLO is the target you aim for (e.g. "99.9% of requests under 300ms"). The error budget is the failure allowance derived from the SLO: an SLO of 99.9% means 0.1% error allowed, which is roughly 43 minutes of permitted downtime per month. This number is extraordinarily powerful because it turns the "be more stable vs. ship faster" debate from an emotional argument into a shared number.

• Runbook: a simple document that says "if this alarm fires, do these things." Nobody feels creative on the night of an incident; the runbook is your backup brain.
• Blameless postmortem: a post-incident meeting with no personal blame. The goal is to find the system's flaw, not a culprit. Because if people are afraid, they hide incidents and the same mistake repeats.

Capacity and back-pressure

A common fatal error: the unbounded queue. An ExecutorService with an infinite queue slowly devours memory under pressure until OutOfMemoryError and death. The right way is a bounded queue with an explicit rejection policy:

// Right: bounded pool + bounded queue + rejection policy
ThreadPoolExecutor executor = new ThreadPoolExecutor(
    10, 20,                          // core / max threads
    60, TimeUnit.SECONDS,
    new ArrayBlockingQueue<>(500),   // bounded queue — not an infinite LinkedBlockingQueue
    new ThreadPoolExecutor.CallerRunsPolicy()  // back-pressure: the submitter runs it
);

CallerRunsPolicy is a subtle trick: when the queue fills, the thread that submitted the work is forced to run it itself. This effectively slows the accepting thread and naturally reduces the inflow rate — a simple, effective back-pressure.

At the service layer, bulkhead and rate limiter are the main tools. A bulkhead caps concurrent calls to a dependency so that one slow dependency doesn't devour the whole service's thread pool:

@Bulkhead(name = "inventory", type = Bulkhead.Type.SEMAPHORE)
@RateLimiter(name = "inventory")
@CircuitBreaker(name = "inventory", fallbackMethod = "fallbackStock")
public StockLevel checkStock(String sku) {
    return inventoryClient.getStock(sku);
}

public StockLevel fallbackStock(String sku, Throwable t) {
    // graceful degradation: a conservative value instead of an error
    return StockLevel.unknown(sku);
}

resilience4j:
  bulkhead:
    instances:
      inventory:
        max-concurrent-calls: 25       # at most 25 concurrent calls
  ratelimiter:
    instances:
      inventory:
        limit-for-period: 100          # 100 calls
        limit-refresh-period: 1s       # per second
        timeout-duration: 0            # if quota is exhausted, reject immediately (fail-fast)
  circuitbreaker:
    instances:
      inventory:
        sliding-window-size: 20
        failure-rate-threshold: 50     # if 50% of calls fail, open the circuit
        wait-duration-in-open-state: 10s

Diagram: the circuit breaker's state lifecycle — چرخه‌ی حالتِ circuit breaker.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failure rate > threshold
    Open --> HalfOpen: after wait duration
    HalfOpen --> Closed: trial calls succeed
    HalfOpen --> Open: trial calls fail
    note right of Open
        fail fast, run fallback
    end note

Graceful degradation

Real examples:

• The "personalized recommendations" service is down? Instead of an error, show best-sellers.
• The real-time pricing service got slow? Show the cached price from a few minutes ago with a banner: "prices may change."
• The image service is down? Show a placeholder, but keep the buy button alive.

Cost — the dimension engineers forget

A senior knows every architectural decision has a price tag, and someone must pay it. Microservices are inherently more expensive than a monolith:

• Network cost: every inter-service call is money (especially cross-availability-zone traffic in the cloud, which incurs egress cost).
• Duplication cost: each service wants its own CPU/RAM/database. 20 small services are often more expensive than one dense monolith, because each has a baseline footprint (JVM minimum heap, sidecar, etc.).
• Observability cost: this is the big surprise. Log, trace, and metric volume in a distributed system is explosive, and APM tools (like Datadog) charge by volume. I've seen a team whose observability bill exceeded its compute bill.

Cross-service data migration

The hardest operational task in the microservices world is this: you want to move a table, or ownership of some data, from service A to service B — with no downtime. This is where even fresh seniors fall down.

The expand/contract pattern for zero-downtime schema change has three phases:

1. Expand — add the new structure alongside the old (new column, new table). Nothing is deleted yet. The code writes to both (dual-write) but still reads from the old.
2. Migrate — gradually backfill existing data into the new structure, and switch the code to read from the new. Now both old and new are up to date.
3. Contract — once you're sure everything reads and writes from the new, drop the old structure.

Each phase deploys and rolls back on its own. The key is that no single deploy changes both schema and code incompatibly at once.

For moving data between two different services (not just a schema), the common pattern is Change Data Capture (CDC) or Outbox: the source service publishes data-change events, the target service consumes them and builds its own state, until they're aligned and you can switch traffic.

Diagram: zero-downtime ownership move via dual-write + backfill — مهاجرتِ مالکیتِ داده با dual-write و backfill.

flowchart TD
    subgraph Phase1[Expand: dual-write]
        W[Write path] --> Old[(Old Store)]
        W --> New[(New Store)]
    end
    subgraph Phase2[Migrate: backfill + read new]
        B[Backfill job] --> New
        R[Read path] --> New
    end
    subgraph Phase3[Contract: drop old]
        R2[All traffic] --> New
    end
    Phase1 --> Phase2 --> Phase3

SQL for the backfill phase, which must be idempotent and batched (so it doesn't lock the live table):

-- PostgreSQL: idempotent upsert in batches
INSERT INTO new_service.customer (id, email, tier)
SELECT id, email, tier FROM old_service.customer
WHERE  id BETWEEN :lo AND :hi
ON CONFLICT (id) DO UPDATE
   SET email = EXCLUDED.email, tier = EXCLUDED.tier;

-- Oracle: same idempotency with MERGE
MERGE INTO new_service.customer t
USING (SELECT id, email, tier FROM old_service.customer
       WHERE id BETWEEN :lo AND :hi) s
ON (t.id = s.id)
WHEN MATCHED THEN UPDATE SET t.email = s.email, t.tier = s.tier
WHEN NOT MATCHED THEN INSERT (id, email, tier)
     VALUES (s.id, s.email, s.tier);

Part Three: How to think out loud in a system-design interview

The system-design interview doesn't measure your knowledge; it measures your judgment and thought process. The interviewer wants to see how you clarify, decompose, and trade off an ambiguous problem. Going silent and jumping straight to a solution is the worst move.

A concrete example: if asked to "design a URL shortener," the senior path is:

• "How many URLs are created per day? How many reads?" (usually reads far exceed writes) → so the system is read-heavy → cache and read replicas matter.
• "Does the short length matter? Do links expire?" → the data model and TTL clarify.
• "Consistency: what if two people grab the same code at once?" → the key-generation strategy (central counter vs. hash).

Part Four: The senior readiness checklist

Before you ship a service to production, ask yourself these. If the answer to half of them is "I don't know," you're not ready for on-call yet.

Observability

• Does every request have a correlation/trace ID that propagates through all logs and across queues?
• Do I have RED metrics (rate/errors/duration with p99) for each endpoint?
• Do I have dashboards and actionable alarms tied to an SLO, not noisy alarms?

Resilience

• Does every external call have a timeout, retry (with backoff and jitter), and circuit breaker?
• Are timeouts chained (upstream shorter than downstream)?
• Do I have a fallback for each non-critical dependency?
• Are queues and pools bounded, and do I have back-pressure?

Data

• Does each service own its own database, and does no one touch another's tables?
• Are message consumers idempotent (do they have an idempotency key)?
• Can schema changes run via expand/contract with no downtime?
• Is event publishing atomic (transactional outbox), not fragile dual-write?

Operations

• Do I have runbooks for the main alarms?
• Is deployment gradual (canary/blue-green) with automatic rollback?
• Is the health check real (not just "the process is alive" but "I can reach my dependencies")?
• Do I know this service's cost (compute + network + observability)?