Elasticsearch و معماری جست‌وجوElasticsearch & Search Architecture

Almost every product you build eventually grows a search box: one plain input where a user types and expects the best results to float to the top — even with a typo, even with the wrong plural, even when they only half-remember the phrase. Behind that plain box sits a whole engineering world: text analysis, an inverted index, ranking, sharding, and a synchronisation architecture that carries data from your primary database into the search engine without losing anything.

This chapter takes you from "why LIKE '%x%' is not search" all the way to "how do I size a multi-terabyte cluster and keep it in sync with CDC".

1. Why LIKE is not search

A B-tree index on title is like a sorted phone book: it can tell you which records start with "data", but not which records contain that word in the middle. A leading % therefore makes the index useless and you fall back to a full scan — on 10 million rows that means seconds of work for every keystroke a user makes.

But the bigger problem is not performance. LIKE gives you a binary answer: present or absent. Search is not inherently binary; search means "which of these are most relevant" — it means ranking. And LIKE matches characters, not words: "cat" matches inside "concatenate", while "running" does not match "run".

Relational databases do have full-text search, and for small to medium workloads it is entirely sufficient:

CREATE INDEX idx_articles_fts
  ON articles USING GIN (to_tsvector('english', title || ' ' || body));

SELECT id,
       ts_rank(to_tsvector('english', title || ' ' || body),
               plainto_tsquery('english', 'database index')) AS rank
FROM   articles
WHERE  to_tsvector('english', title || ' ' || body)
       @@ plainto_tsquery('english', 'database index')
ORDER  BY rank DESC
FETCH FIRST 10 ROWS ONLY;

-- Oracle Text: a CONTEXT domain index
CREATE INDEX idx_articles_fts ON articles(body)
  INDEXTYPE IS CTXSYS.CONTEXT;

SELECT id, SCORE(1) AS rank
FROM   articles
WHERE  CONTAINS(body, 'database AND index', 1) > 0
ORDER  BY SCORE(1) DESC
FETCH FIRST 10 ROWS ONLY;

2. The inverted index from scratch

Take three documents: doc 1 is the quick brown fox, doc 2 is the quick blue car, doc 3 is brown fox jumps over the lazy dog. First we break each text into terms, then we invert the table — instead of "document → words" we write "word → documents":

Building the jargon from zero:

• term: the atomic unit of search after analysis. Not necessarily a "word" — whatever the analyzer emits is a term.
• postings list: the sorted list of document ids containing that term. Sortedness is crucial, because intersecting or unioning two sorted lists is a single linear walk (see the core-ds and complexity chapters).
• term frequency (tf): how many times the term occurs in that document — the basis for "this document is more about this word".
• document frequency (df): how many documents contain the term. the is in every document so it is near worthless; jumps is rare so it is valuable.
• positions: where the term sits in the document. Without positions, match_phrase is impossible.

The query brown fox is the intersection of {1,3} with {1,3}. Its cost depends on how many documents contain those words, not on the total size of the index — that is precisely what makes search scale.

The diagram below is the mental model of an inverted index — Inverted index model: the term dictionary points to postings lists.

flowchart LR
  subgraph Dict["Term dictionary (sorted, on disk)"]
    T1["brown"]
    T2["fox"]
  end
  subgraph Post["Postings"]
    P1["doc1 tf=1 pos=2 | doc3 tf=1 pos=0"]
    P2["doc1 tf=1 pos=3 | doc3 tf=1 pos=1"]
  end
  T1 --> P1
  T2 --> P2
  Q["Query: brown AND fox"] --> T1
  Q --> T2
  P1 --> M["Intersect sorted lists"]
  P2 --> M
  M --> R["Hits: doc1, doc3"]

Elasticsearch did not implement the inverted index itself; it sits on Apache Lucene. Each Elasticsearch shard is in fact a complete Lucene index, and each Lucene index is made of segments. A segment is an immutable bundle of files: once written, it never changes.

Immutability buys three things for free: lock-free concurrent reads, safe caching of whole files in the OS page cache, and aggressive compression. The price is that "update" does not exist — an update means marking the old document as deleted in a bitmap and writing the new version into a fresh segment. The old document's space is only reclaimed during a merge.

Alongside the inverted index, Lucene builds a second structure: doc values, which are columnar and used for sorting, aggregations and scripts.

You will be tempted to set "_source": {"enabled": false} to save disk; don't. It kills reindex, partial update, highlighting and document reconstruction — exactly the things you need on incident day. If size really hurts, reach for "codec": "best_compression" or _source.excludes first.

3. Analysis — the real heart of search quality

The diagram below is the analysis pipeline — Text analysis pipeline: raw text becomes indexed terms.

flowchart LR
  A["Raw field value"] --> B["Character filters (0..n)"]
  B --> C["Tokenizer (exactly 1)"]
  C --> D["Token filters (0..n)"]
  D --> E["Terms written to inverted index"]
  Q["Query string"] --> B2["Same analyzer at search time"]
  B2 --> F["Query terms"]
  E --> G["Match only if terms are identical"]
  F --> G

The single most useful debugging tool in search is _analyze. Before you guess anything, run it:

POST /_analyze
{
  "analyzer": "english",
  "text": "The Foxes were quickly Running through 3 Databases!"
}

You get terms like fox, quickli, run, 3, databas. Notice three things: The and were are gone (stop words); Foxes became fox and Databases became databas (stemming — an algorithmic root that is not necessarily a real word); and everything is lowercased. Instead of a named analyzer you can pass the parts directly: "char_filter": ["html_strip"], "tokenizer": "standard", "filter": ["lowercase","asciifolding"].

Stop words are high-frequency, low-information words (the, and). Removing them shrinks the index but breaks match_phrase — drop the and "to be or not to be" becomes effectively empty. Synonyms are for when the user types "cellphone" and you want "phone" to match:

PUT /products
{
  "settings": { "analysis": {
    "filter": { "product_synonyms": { "type": "synonym_graph", "synonyms": [
        "laptop, notebook",
        "cellphone, mobile phone, smartphone => phone" ] } },
    "analyzer": {
      "product_index":  { "tokenizer": "standard", "filter": ["lowercase"] },
      "product_search": { "tokenizer": "standard", "filter": ["lowercase", "product_synonyms"] }
    } } },
  "mappings": { "properties": {
    "name": { "type": "text", "analyzer": "product_index", "search_analyzer": "product_search" } } }
}

Notation: a, b means two-way equivalence; a, b => c rewrites everything on the left to c. Put synonyms in the search_analyzer, not at index time — otherwise every edit to the synonym list means a full reindex. synonym_graph, unlike the older synonym filter, handles multi-word phrases correctly and is designed for search time; keep the list outside the mapping via the synonyms set API so it can change without a deploy.

edge_ngram produces prefixes from the start of a word: search becomes se, sea, sear, searc, search. Plain ngram produces every substring and is used for "contains" matching.

PUT /catalog
{
  "settings": {
    "index": { "max_ngram_diff": 18 },
    "analysis": {
      "tokenizer": { "edge_2_20": { "type": "edge_ngram", "min_gram": 2, "max_gram": 20,
                                    "token_chars": ["letter", "digit"] } },
      "analyzer":  { "autocomplete_index": { "tokenizer": "edge_2_20", "filter": ["lowercase"] } }
    }
  },
  "mappings": { "properties": {
    "name": { "type": "text", "analyzer": "autocomplete_index", "search_analyzer": "standard" } } }
}

Non-English text: the Persian example

Persian, like many non-English scripts, has three concrete problems that will otherwise leave your search permanently "a bit off": the zero-width non-joiner (ZWNJ, U+200C), which makes one word look like three different strings depending on how it was typed; visually identical Arabic and Persian characters such as ي versus ی and ك versus ک; and digits written in Persian, Arabic or Latin forms. The built-in persian analyzer covers exactly these, and this is its official definition:

PUT /articles_fa
{
  "settings": { "analysis": {
    "char_filter": { "zero_width_spaces": { "type": "mapping", "mappings": [ "\\u200C=>\\u0020" ] } },
    "filter":      { "persian_stop": { "type": "stop", "stopwords": "_persian_" } },
    "analyzer": {
      "rebuilt_persian": {
        "tokenizer": "standard",
        "char_filter": [ "zero_width_spaces" ],
        "filter": [ "lowercase", "decimal_digit", "arabic_normalization",
                    "persian_normalization", "persian_stop", "persian_stem" ]
      }
    } } }
}

zero_width_spaces turns the ZWNJ into a normal space so a compound word splits into two tokens; decimal_digit folds any Unicode digit to its Latin equivalent; the two normalization filters unify Arabic character shapes; and persian_stem performs Persian stemming.

The senior judgement generalises to any language: a built-in language analyzer is a starting point, not a finished design. On short, high-stakes fields such as product or person names, index both a stemmed and a raw variant as a multi-field and boost the raw one higher in multi_match, because aggressive stemming mangles proper nouns. And install the analysis-icu plugin if you handle several languages: icu_normalizer and icu_folding cover far more of Unicode than asciifolding.

4. Mappings — the Elasticsearch schema

A mapping defines what type each field has and how it is analysed and stored. It is the equivalent of CREATE TABLE, with one big difference: if you do not write it, Elasticsearch guesses. A string becomes text plus a keyword sub-field with ignore_above: 256, an integer becomes long, a decimal float, and a date-looking string date.

PUT /_index_template/products-template
{
  "index_patterns": ["products-*"],
  "priority": 200,
  "template": {
    "settings": { "number_of_shards": 3, "number_of_replicas": 1, "refresh_interval": "5s" },
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "sku":         { "type": "keyword" },
        "name":        { "type": "text", "fields": {
                           "keyword": { "type": "keyword", "ignore_above": 256 },
                           "ngram":   { "type": "text", "analyzer": "autocomplete_index",
                                        "search_analyzer": "standard" } } },
        "description": { "type": "text", "analyzer": "english" },
        "price":       { "type": "scaled_float", "scaling_factor": 100 },
        "created_at":  { "type": "date", "format": "strict_date_optional_time||epoch_millis" },
        "location":    { "type": "geo_point" },
        "tags":        { "type": "keyword" },
        "attributes":  { "type": "flattened" }
      }
    }
  }
}

Values for dynamic: true (add it), runtime (add as a runtime field, not indexed), false (ignore it but keep it in _source — not searchable), strict (throw). For a public API strict is best because it surfaces producer-side bugs immediately.

The fields block is a multi-field: name is indexed as text for search, name.keyword for filtering and sorting, and name.ngram for autocomplete. One _source, three index structures — the main tool for resolving "I want both search and precision".

object vs nested vs join

Elasticsearch flattens nested JSON by default. The document {"orders": [{"product":"laptop","qty":1}, {"product":"mouse","qty":5}]} is stored internally as orders.product: ["laptop","mouse"] and orders.qty: [1,5] — and the link between laptop and 1 is lost. The consequence: a query for "an order where product = laptop and qty = 5" returns this document even though no such order exists. That is a bug which sits silently in reports for months.

With nested, each array element is stored as a separate hidden document in the same shard and the relationship is preserved:

PUT /carts
{ "mappings": { "properties": {
  "orders": { "type": "nested", "properties": {
    "product": { "type": "keyword" }, "qty": { "type": "integer" } } } } } }

GET /carts/_search
{ "query": { "nested": {
    "path": "orders",
    "query": { "bool": { "filter": [
      { "term":  { "orders.product": "laptop" } },
      { "range": { "orders.qty": { "gte": 5 } } } ] } },
    "inner_hits": { "size": 3 } } } }

inner_hits tells you which element matched — without it you only know the parent matched.

Do not underestimate the cost: every nested element is a separate Lucene document, so a parent with 200 elements is 201 documents, and updating one element rewrites the entire parent. The default limits are deliberate — index.mapping.nested_fields.limit is 50 and index.mapping.nested_objects.limit is 10000; if you are raising them, your data model is probably wrong. The join type gives you real parent/child so a child can be updated without rewriting the parent, but queries are slower. The rule of thumb: denormalise first, then nested, and join only for a genuine one-to-very-many relationship with independent updates.

Changing a mapping with zero downtime

You cannot change the type of an existing field, because the data on disk was written with that type. The standard pattern is: new index, reindex, atomic alias swap.

The diagram below is the zero-downtime path — Zero-downtime mapping change with an alias swap.

sequenceDiagram
    participant App
    participant Alias as "Alias: products"
    participant V1 as "products-v1"
    participant V2 as "products-v2"
    App->>Alias: read and write
    Alias->>V1: routes
    Note over V2: create with new mapping
    V2->>V2: _reindex from products-v1
    Note over App,V2: dual-write or replay changes since snapshot
    Alias->>V2: atomic _aliases swap
    Note over V1: delete after verification

curl -X POST "localhost:9200/_reindex?wait_for_completion=false" \
  -H 'Content-Type: application/json' \
  -d '{"source":{"index":"products-v1","size":2000},"dest":{"index":"products-v2","op_type":"index"}}'

curl -s "localhost:9200/_tasks?actions=*reindex&detailed"

# atomic swap — there is never a moment without an index
curl -X POST localhost:9200/_aliases -H 'Content-Type: application/json' -d '{
  "actions": [
    { "remove": { "index": "products-v1", "alias": "products" } },
    { "add":    { "index": "products-v2", "alias": "products" } } ] }'

5. The query DSL

GET /products/_search
{
  "query": {
    "bool": {
      "must": [ { "match": { "name": { "query": "wireless headphones", "operator": "and" } } } ],
      "should": [
        { "match_phrase": { "name": { "query": "noise cancelling", "boost": 3 } } },
        { "term": { "brand": { "value": "acme", "boost": 1.5 } } }
      ],
      "filter": [
        { "term":  { "active": true } },
        { "terms": { "category": ["audio", "accessories"] } },
        { "range": { "price": { "gte": 50, "lte": 400 } } },
        { "range": { "created_at": { "gte": "now-1y/d" } } }
      ],
      "must_not": [ { "term": { "discontinued": true } } ],
      "minimum_should_match": 1
    }
  },
  "size": 20,
  "track_total_hits": 1000
}

The four bool clauses: must must match and contributes to _score; filter must match but does not affect _score and is cacheable; should is optional and raises the score when it matches; must_not must not match and runs in filter context.

There are two caches. The node query cache keeps filter results as bitsets in node memory (10% of heap by default) and only kicks in for filters that are reused often enough on large enough segments. The shard request cache caches the whole response for requests with size: 0 — that is, exactly aggregations and dashboards — and is invalidated on every refresh.

GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "sony wireless headphone",
      "type": "best_fields",
      "fields": ["name^4", "brand^2", "description", "tags"],
      "tie_breaker": 0.3,
      "fuzziness": "AUTO",
      "prefix_length": 1,
      "minimum_should_match": "70%"
    }
  },
  "highlight": {
    "fields": { "name": { "number_of_fragments": 0 },
                "description": { "fragment_size": 120, "number_of_fragments": 2 } },
    "pre_tags": ["<mark>"], "post_tags": ["</mark>"]
  }
}

The multi_match types: best_fields (default) scores as the best single field plus tie_breaker times the rest, and is right when all the words usually live in one field; most_fields sums the scores of all fields and suits a single text indexed with several analyzers; cross_fields treats the fields as one big field and suits names and addresses where words are spread across fields; and phrase, phrase_prefix and bool_prefix cover phrase and prefix cases.

fuzziness: AUTO means terms of 0–2 characters must match exactly, 3–5 characters allow one edit, and longer than 5 allow two. The thresholds are configurable via AUTO:4,8; the default is equivalent to AUTO:3,6.

The real cost of wildcard, prefix and regexp: these perform term expansion — Elasticsearch has to walk the term dictionary, find every matching term and rewrite the query as an OR across them. {"wildcard": {"name": "*phone*"}} means scanning that field's entire term dictionary on every shard, which on a large index takes seconds. The right approach: replace prefix with edge_ngram at index time or the index_prefixes mapping parameter; replace *x* with an ngram field; and on multi-tenant clusters set search.allow_expensive_queries to false so nobody can accidentally take the cluster down.

6. Autocomplete and pagination

PUT /suggestions
{ "mappings": { "properties": {
  "title":   { "type": "search_as_you_type", "max_shingle_size": 3 },
  "suggest": { "type": "completion", "analyzer": "simple" } } } }

GET /suggestions/_search
{
  "suggest": {
    "product-suggest": {
      "prefix": "wirel",
      "completion": { "field": "suggest", "size": 8, "skip_duplicates": true,
                      "fuzzy": { "fuzziness": "AUTO", "prefix_length": 1 } } }
  },
  "query": {
    "multi_match": { "query": "wirel head", "type": "bool_prefix",
                     "fields": ["title", "title._2gram", "title._3gram"] }
  }
}

The suggest block uses the completion suggester while the query block does the same job through the auto-generated search_as_you_type sub-fields. Alongside these there are also the term suggester and phrase suggester, which serve "did you mean?" rather than autocomplete.

The completion suggester is not what you think it is. Three limitations that people discover late: (1) it matches prefixes only, so "headphone" will never find "wireless headphone" unless you write every rotation as an input; (2) the whole FST is loaded into heap, which costs gigabytes on a few million phrases; (3) filtering is only possible through predefined contexts, not an arbitrary bool — so "autocomplete only over items in stock in this warehouse" must have been anticipated in the context design. For most real products, edge_ngram on a multi-field combined with a bool carrying the business filters is the better choice.

Pagination — where systems go to die

{"from": 0, "size": 20} works up to page 500 and then throws Result window is too large, because index.max_result_window defaults to 10000. The reason: for from=9980, size=20 on a 5-shard index, every shard must sort and ship its own top 10000 hits, and the coordinating node merges 50000 records in memory just to take 20 from the middle. Cost grows linearly with from and multiplies by shard count; a handful of concurrent requests is enough to fill the heap and trip a circuit breaker. That ceiling is a guard rail, not an annoyance.

curl -X POST "localhost:9200/products/_pit?keep_alive=2m"
# response: {"id":"46ToAwMDaWR5..."}

GET /_search
{
  "size": 1000,
  "query": { "term": { "active": true } },
  "pit": { "id": "46ToAwMDaWR5...", "keep_alive": "2m" },
  "sort": [
    { "created_at": { "order": "asc", "format": "strict_date_optional_time_nanos" } },
    { "_shard_doc": "asc" }
  ],
  "track_total_hits": false
}

For the next page send the same body and simply add "search_after": ["2026-03-14T10:22:31.001Z", 4294967298] — the sort array of the last hit of the previous page.

Three details that separate a professional from an amateur: a tiebreaker is mandatory — the sort key must become unique or records duplicate or vanish between pages; with a PIT the _shard_doc field is added automatically, and without one you must append _id as the last sort key. track_total_hits stops counting at 10000 by default and reports hits.total.relation as "gte"; set it to true for an exact count (expensive) and to false while paginating. And close the PIT with DELETE /_pit, because an open PIT pins segments and prevents merges from reclaiming space.

7. Relevance — why is this document on top?

The older TF-IDF formula scored roughly tf × idf × 1/√length, and its flaw was that tf grew without bound, so keyword-stuffed documents scored absurdly high. BM25 — the Elasticsearch default since version 5 — adds two parameters: k1, default 1.2, controlling how quickly term frequency saturates (from the third occurrence onward each extra one matters less as the curve approaches a ceiling); and b, default 0.75, controlling the strength of length normalisation (b = 0 ignores document length, b = 1 weights it fully).

PUT /articles
{
  "settings": { "index": { "similarity": {
    "tuned_bm25": { "type": "BM25", "k1": 1.1, "b": 0.4 } } } },
  "mappings": { "properties": {
    "title": { "type": "text", "similarity": "tuned_bm25" },
    "body":  { "type": "text" } } }
}

Touch k1 and b almost never — certainly not before you have fixed mappings, analyzers and field boosts. The one genuine case is fields whose length varies enormously; for short titles a smaller b makes sense so a one-word title does not win purely by being short.

Textual relevance is only half the story. In a shop, "relevant" means combining text with stock, ratings, recency and margin:

GET /products/_search
{
  "query": {
    "function_score": {
      "query": { "multi_match": { "query": "running shoes", "fields": ["name^3", "description"] } },
      "functions": [
        { "filter": { "term": { "in_stock": true } }, "weight": 2 },
        { "field_value_factor": { "field": "rating", "factor": 1.2,
                                  "modifier": "sqrt", "missing": 3 } },
        { "gauss": { "created_at": { "origin": "now", "scale": "30d", "decay": 0.5 } } }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply"
    }
  }
}

score_mode says how the individual functions combine with each other and boost_mode says how their result combines with the textual score. gauss is a decay curve: the further from origin, the lower the score — the standard tool for recency and geographic proximity.

These functions are expensive: they run for every candidate document, not just the top 10. If the query matches a million documents, the function runs a million times. Two escapes: shrink the candidate set with filter first; and instead of scripts use the rank_feature and rank_features field types, which are optimised in Lucene and let you fold a numeric signal such as "view count" into the score without executing a script per document. Never sort by _script on a user-facing path; it is both slow and it defeats Lucene's early-termination optimisations.

When the wrong result is on top, do not guess — ask for an explanation:

curl -X GET localhost:9200/products/_explain/SKU-123 -H 'Content-Type: application/json' -d '{
  "query": { "multi_match": { "query": "running shoes", "fields": ["name^3","description"] } } }'

The output is a tree: each node has a value, a description and children. There you can see exactly what boost, idf (with n and N) and tf (with freq, k1, b, dl, avgdl) contributed. To see the same for every hit, add "explain": true to the _search body. For a latency problem rather than a ranking problem you have a different tool: "profile": true, which breaks down where time went in each shard.

Scores are not comparable across shards. By default idf is computed from the statistics of that shard, not the whole index. With few documents and an uneven distribution, two identical documents in two shards get different scores. This usually appears in a test environment with 20 documents and 5 shards and sends you hunting for an imaginary bug. The temporary fix for testing is ?search_type=dfs_query_then_fetch, which gathers distributed statistics first; the real fix in production is fewer shards and more data.

8. Aggregations — the analytics half of the engine

Elasticsearch is also a columnar analytics engine working over doc values. There are three families: metric (avg, sum, cardinality, percentiles, top_hits), bucket (terms, range, date_histogram, filters, nested, composite), and pipeline, which operates on the output of other aggregations (derivative, moving_fn, cumulative_sum, bucket_selector, bucket_sort).

GET /orders/_search
{
  "size": 0,
  "query": { "bool": { "filter": [ { "range": { "created_at": { "gte": "now-90d/d" } } } ] } },
  "aggs": {
    "per_day": {
      "date_histogram": { "field": "created_at", "calendar_interval": "day",
                          "time_zone": "Europe/Berlin", "min_doc_count": 0 },
      "aggs": {
        "revenue":        { "sum": { "field": "total" } },
        "unique_buyers":  { "cardinality": { "field": "customer_id", "precision_threshold": 3000 } },
        "revenue_growth": { "derivative": { "buckets_path": "revenue" } }
      }
    },
    "top_categories": {
      "terms": { "field": "category", "size": 10, "shard_size": 100,
                 "order": { "revenue": "desc" } },
      "aggs": {
        "revenue":  { "sum": { "field": "total" } },
        "big_only": { "bucket_selector": { "buckets_path": { "r": "revenue" },
                                           "script": "params.r > 10000000" } }
      }
    }
  }
}

"size": 0 means "no hits, just aggregations" — and that is exactly what enables the shard request cache.

9. Cluster mechanics

A node is a JVM process, and you set its responsibilities with node.roles: master (manages cluster state, not data), data plus the tier roles data_hot, data_warm, data_cold, data_frozen, data_content, and also ingest (runs pipelines before indexing), ml, transform and remote_cluster_client. With node.roles: [] you get a pure coordinating node that only routes requests and merges results.

A primary shard is one slice of an index and is itself a complete Lucene index; its count is fixed at index creation and can only change via _split, _shrink or a reindex. A replica is a copy of a primary on another node, giving both redundancy and read capacity, and its count can be changed at any time.

Which shard does a document land on? A deterministic formula: shard = hash(routing) % number_of_primary_shards, where routing defaults to the _id. That formula is exactly why the primary shard count cannot change after creation: changing the divisor would relocate every document.

The diagram below is what a search actually does — Distributed search: query phase gathers ranked ids, fetch phase loads documents.

sequenceDiagram
    participant C as Client
    participant Co as "Coordinating node"
    participant S1 as "Shard A copy"
    participant S2 as "Shard B copy"
    C->>Co: POST /idx/_search size=10 from=0
    Co->>S1: Query phase (local top 10 ids and scores)
    Co->>S2: Query phase (local top 10 ids and scores)
    S1-->>Co: 10 doc ids plus scores
    S2-->>Co: 10 doc ids plus scores
    Co->>Co: Merge and sort, keep global top 10
    Co->>S1: Fetch phase (get _source for chosen ids)
    Co->>S2: Fetch phase (get _source for chosen ids)
    S1-->>Co: documents
    Co-->>C: Final ranked hits

Three consequences. (1) from + size is multiplied by shard count — for from=9000, size=10 on 5 shards, the first phase merges 5 × 9010 records; that is the root of the 10000 ceiling. (2) The slowest shard sets the overall latency — one oversized shard raises p99 for every query. (3) The fetch phase only reads _source for the winning documents, so use _source: {"includes": [...]} or fields to take only what you need.

refresh, flush, merge — and what "near real-time" means

The diagram below is the life of a write — Document write lifecycle: buffer, refresh, flush, merge.

stateDiagram-v2
    [*] --> InMemoryBuffer: index request
    InMemoryBuffer --> Translog: appended for durability
    InMemoryBuffer --> NewSegment: refresh (default 1s)
    NewSegment --> Searchable: visible to search
    Searchable --> DiskCommit: flush (Lucene commit, translog truncated)
    DiskCommit --> MergedSegment: background merge
    MergedSegment --> [*]: deleted docs reclaimed

• refresh turns the in-memory buffer into a new segment so it becomes searchable. index.refresh_interval defaults to 1s. This is the only reason we call Elasticsearch near real-time: a document you just wrote is not in _search results for about a second — but GET /index/_doc/<id> returns it immediately, because that path reads from the translog.
• translog is an append-only log for durability. With index.translog.durability set to request (the default) every request is fsynced before it is acknowledged; with async it is flushed every 5 seconds — faster, but with a data-loss window.
• flush is a real Lucene commit; segments become durable and the translog is truncated. It happens automatically.
• merge combines several small segments into a larger one and genuinely reclaims the space of deleted documents.

curl -X PUT localhost:9200/products/_settings -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"-1","number_of_replicas":0}}'
# ... bulk-load the data here ...
curl -X PUT localhost:9200/products/_settings -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"1s","number_of_replicas":1}}'
curl -X POST "localhost:9200/products/_forcemerge?max_num_segments=1"

How many shards? — and the classic over-sharding mistake

A back-of-the-envelope for a new index: final data volume including a year of growth divided by 40 GB, rounded up, and at least as many shards as you have data nodes. For 600 GB of data that is roughly 15 primary shards. If the index is small and static, number_of_shards: 1 is entirely correct.

For time-based data (logs, metrics, events) you should never have a single ever-growing index. The right pattern is a data stream: one logical name backed by a series of hidden indices, with rollover creating a new one whenever the current index crosses a threshold.

The diagram below is the data lifecycle under ILM — Index lifecycle: hot to warm to cold to frozen to delete.

flowchart LR
  H["Hot: active writes, fast SSD"] -->|rollover 50gb or 1d| W["Warm: read only, forcemerge, shrink"]
  W -->|min_age 30d| C["Cold: cheaper nodes, searchable snapshot"]
  C -->|min_age 90d| F["Frozen: object storage, slow queries"]
  F -->|min_age 365d| D["Delete"]

PUT /_ilm/policy/logs-lifecycle
{
  "policy": { "phases": {
    "hot":  { "min_age": "0ms", "actions": {
                "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" },
                "set_priority": { "priority": 100 } } },
    "warm": { "min_age": "2d", "actions": {
                "shrink": { "number_of_shards": 1 },
                "forcemerge": { "max_num_segments": 1 },
                "set_priority": { "priority": 50 } } },
    "cold": { "min_age": "30d", "actions": { "set_priority": { "priority": 0 } } },
    "delete": { "min_age": "90d", "actions": { "delete": {} } }
  } }
}

Prefer max_primary_shard_size over max_size: max_size measures the whole index, so changing shard count pushes each shard outside the healthy range. max_primary_shard_size controls the thing that actually matters. Always pair it with a max_age so a low-traffic index does not stay open forever.

10. The Elastic Stack and the Java clients

Elasticsearch on its own is only the engine. Beats are lightweight single-purpose collectors (Filebeat for log files, Metricbeat for metrics, Heartbeat for uptime), but today Elastic Agent replaces all of them: one unified agent centrally managed through Fleet in Kibana. Logstash is a heavyweight transformation pipeline with dozens of input and output plugins; if your transformation is simple (grok, renaming a field, adding geo data) you do not need it, because an ingest pipeline inside Elasticsearch does the same job with one fewer service to run. Kibana is the UI: data exploration, dashboards, ILM and template management, and Dev Tools.

The diagram below shows where each piece sits — Elastic Stack data flow from source to dashboard.

flowchart LR
  A["App logs / metrics / DB rows"] --> B["Elastic Agent or Beats"]
  B --> C["Logstash (optional heavy transform)"]
  B --> D["Ingest pipeline (inside Elasticsearch)"]
  C --> D
  D --> E["Elasticsearch indices / data streams"]
  E --> F["Kibana: Discover, Lens, Alerts"]
  E --> G["Your application search API"]

PUT /_ingest/pipeline/normalize-logs
{
  "processors": [
    { "grok": { "field": "message",
                "patterns": ["%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}"] } },
    { "date": { "field": "ts", "formats": ["ISO8601"], "target_field": "@timestamp" } },
    { "lowercase": { "field": "level" } },
    { "remove": { "field": ["ts", "message"], "ignore_missing": true } }
  ],
  "on_failure": [ { "set": { "field": "error.pipeline", "value": "normalize-logs" } } ]
}

The old TransportClient and RestHighLevelClient have been removed. Today's official client is the Elasticsearch Java API Client under the co.elastic.clients group:

<dependency>
  <groupId>co.elastic.clients</groupId>
  <artifactId>elasticsearch-java</artifactId>
  <version>9.3.0</version>
</dependency>

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch._types.query_dsl.Query;
import co.elastic.clients.elasticsearch.core.SearchResponse;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import co.elastic.clients.transport.rest_client.RestClientTransport;
import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;

public class ProductSearch {

    private final ElasticsearchClient client;

    public ProductSearch(String host, int port) {
        RestClient rest = RestClient.builder(new HttpHost(host, port, "https")).build();
        this.client = new ElasticsearchClient(new RestClientTransport(rest, new JacksonJsonpMapper()));
    }

    public SearchResponse<Product> search(String text, double maxPrice) throws Exception {
        Query byText = Query.of(q -> q.multiMatch(m -> m
                .query(text).fields("name^4", "description").fuzziness("AUTO")));
        Query priceFilter = Query.of(q -> q.range(r -> r.number(n -> n.field("price").lte(maxPrice))));
        Query activeFilter = Query.of(q -> q.term(t -> t.field("active").value(true)));

        return client.search(s -> s
                .index("products")               // an alias, never a real index name
                .query(q -> q.bool(b -> b.must(byText).filter(priceFilter, activeFilter)))
                .size(20)
                .trackTotalHits(t -> t.count(1000)),
            Product.class);
    }
}

Batch writes must always go through bulk, never one at a time:

public void indexAll(List<Product> batch) throws Exception {
    BulkRequest.Builder br = new BulkRequest.Builder().index("products");
    for (Product p : batch) {
        br.operations(op -> op.index(i -> i.id(p.sku()).document(p)));
    }
    BulkResponse res = client.bulk(br.build());
    if (res.errors()) {
        res.items().stream()
           .filter(it -> it.error() != null)
           .forEach(it -> log.error("bulk item {} failed: {}", it.id(), it.error().reason()));
    }
}

In Spring, Spring Data Elasticsearch sits on top of that same client:

@Document(indexName = "products")
public class ProductDoc {

    @Id
    private String sku;

    @Field(type = FieldType.Text, analyzer = "standard")
    private String name;

    @Field(type = FieldType.Keyword)
    private String brand;

    @Field(type = FieldType.Scaled_Float, scalingFactor = 100)
    private BigDecimal price;

    @Field(type = FieldType.Date, format = DateFormat.date_optional_time)
    private Instant createdAt;
}

For anything beyond trivial queries, use NativeQuery with ElasticsearchOperations — the full query DSL, with type safety:

public SearchHits<ProductDoc> topProducts(String text, ElasticsearchOperations ops) {
    NativeQuery query = NativeQuery.builder()
        .withQuery(q -> q.bool(b -> b
            .must(m -> m.multiMatch(mm -> mm.query(text).fields("name^4", "description")))
            .filter(f -> f.term(t -> t.field("active").value(true)))))
        .withPageable(PageRequest.of(0, 20))
        .build();
    return ops.search(query, ProductDoc.class);
}

Take the version compatibility matrix seriously. Spring Data Elasticsearch is pinned to a specific version of the official client: 6.1.x targets Elasticsearch 9.4.2 and Spring Framework 7.0.x, 6.0.x targets Elasticsearch 9.2.2, and 5.5.x targets Elasticsearch 8.18.1. Do not manually override the elasticsearch-java version, because the incompatibility shows up as a runtime NoSuchMethodError rather than a compile error; and when upgrading, raise the cluster first and the application second. For testing, rather than mocking the client, start a real Elasticsearch with Testcontainers — no mock reproduces analyzer and mapping behaviour (see the testing chapter).

11. Search architecture — keeping in sync with the primary database

The diagram below puts the three strategies side by side — Three ways to keep the search index in sync with the primary database.

flowchart TD
  subgraph DW["1. Dual write (avoid)"]
    A1["Service"] --> B1[(Primary DB)]
    A1 --> C1[(Elasticsearch)]
  end
  subgraph OB["2. Transactional outbox"]
    A2["Service"] -->|one local transaction| B2[(Primary DB plus outbox table)]
    B2 --> R2["Relay / poller"]
    R2 --> C2[(Elasticsearch)]
  end
  subgraph CDC["3. Log-based CDC"]
    B3[(Primary DB)] -->|WAL / redo log| K3["CDC connector"]
    K3 --> Q3["Message broker"]
    Q3 --> I3["Indexer service"]
    I3 --> C3[(Elasticsearch)]
  end

In the outbox pattern you write an event row into an outbox table inside the very same transaction that changes the data. Because both happen in one local transaction, either both commit or neither does. A separate relay then reads those rows and bulk-ships them to Elasticsearch.

CREATE TABLE outbox (
  id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  aggregate_id text        NOT NULL,
  event_type   text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);

-- partial index: only unpublished rows
CREATE INDEX idx_outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

CREATE TABLE outbox (
  id           NUMBER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  aggregate_id VARCHAR2(64) NOT NULL,
  event_type   VARCHAR2(64) NOT NULL,
  payload      CLOB         NOT NULL CHECK (payload IS JSON),
  created_at   TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
  published_at TIMESTAMP WITH TIME ZONE
);

-- Oracle has no partial index; emulate it with a function-based index
CREATE INDEX idx_outbox_unpublished
  ON outbox (CASE WHEN published_at IS NULL THEN id END);

In both engines, an index holding only unpublished rows stays small even when outbox holds millions of historical rows. In PostgreSQL that is done directly with WHERE; in Oracle, because a B-tree does not index all-NULL entries, the CASE expression achieves the same thing (see the sql-mastery and oracle-postgres-dialects chapters).

If you would rather poll, at least do it properly — with keyset pagination, never OFFSET:

SELECT id, updated_at
FROM   products
WHERE  (updated_at, id) > ($1, $2)
ORDER  BY updated_at, id
FETCH FIRST 1000 ROWS ONLY;

SELECT id, updated_at
FROM   products
WHERE  (updated_at, id) > ((:last_ts, :last_id))
ORDER  BY updated_at, id
FETCH FIRST 1000 ROWS ONLY;

Polling on updated_at has three holes: it never sees deletes (a deleted row simply stops appearing and lives in Elasticsearch forever — the fix is a soft delete); clock skew between servers can drop records; and long transactions that commit late with an older updated_at fall outside the pull window, which is why teams deliberately overlap the window by a few seconds and rely on idempotency.

Log-based CDC means reading the database transaction log directly — the WAL in PostgreSQL, redo via LogMiner or XStream in Oracle — and turning each change into an event. No application code change is needed, deletes are visible, and the event order is exactly commit order.

Broker and delivery semantics are covered in the messaging and ms-data chapters.

curl -X PUT "localhost:9200/products/_doc/SKU-123?version=1712345678901&version_type=external" \
  -H 'Content-Type: application/json' -d '{"sku":"SKU-123","name":"Wireless Headphones"}'

Every sync architecture needs a full rebuild path that can be run at any moment: create a new index with the fresh mapping; configure the event consumer to write to both the old and the new index (this is dual indexing, not dual writing — it is safe because both destinations are fed from the same event source); backfill with search_after plus a PIT from the primary database, or _reindex from the previous index; verify with document counts and random sampling; then swap the alias atomically and keep the old index for a week.

12. Production checklist