Skip to content

Search

Most search you have used finds books. You type a phrase, and it hands back a list of titles that might contain what you meant, leaving you to open each one and hunt for the passage. Search over the Content Lake is built to do the harder, more useful thing: find the paragraph. Because the lake holds content as structured blocks rather than opaque files (see Content format), a search can return the exact heading, paragraph, or code sample that answers a question, and tell you which Resource it came from — without opening a single book.

That block-level precision is what makes the same search facility serve an editor avoiding duplication, a product surface building a "more like this" rail, and an AI assistant assembling context for an answer. All three ask the same question — where, across everything we hold, is the content about X? — and all three want the passage, not the shelf.

Finding the right passage well takes more than matching words. A reader who asks for "securing service-to-service calls" should also find a paragraph that only ever says "mutual TLS between microservices", even though the two share almost no words. To do that, search blends three different kinds of signal in a single query — keyword, meaning, and knowledge-graph context — and this page explains each by analogy before getting into the shape of a request. It is a business-facing tour; the exact request and response payloads live in collapsible blocks for the technically curious, and the full contract is in the Public API.

Every request is scoped to one Tenant: the token you send carries a tenant_id, and a search can only ever see that tenant's content. And every request already knows about rights — the resolved rights of each Resource travel with it into the index, so results come back knowing what may be done with them, which matters enormously the moment an AI surface is involved.

Hybrid retrieval, by analogy

The heart of search is hybrid retrieval: three ways of looking for the same thing, run together, then merged so the best answer wins no matter which signal found it. Think of three researchers sent to the same library with the same question.

  • The keyword researcher scans the index for your exact words and phrases. Ask for "OAuth token exchange" and they find every passage that literally says "OAuth token exchange". They are unbeatable for known-item lookups — an exact title, an error message, a function name — and useless when the answer is phrased differently from the question. This is classic full-text matching.

  • The meaning researcher ignores the exact words and looks for passages that are about the same idea. They understand that "securing service-to-service calls" and "mutual TLS between microservices" are neighbours in meaning, and surface the second even though it shares no words with the first. This is semantic search: each block is turned into a numerical fingerprint of its meaning (an embedding), and the query is matched against the closest fingerprints. It is superb at recall and hopeless at precision on its own — "close in meaning" casts a wide net.

  • The knowledge-graph researcher knows the entities your question is about. If your query mentions a specific tool, library, or person that the platform recognises — say the github/chi router — they go straight to every passage that discusses that entity, even ones whose wording matches neither the keywords nor the meaning of your phrasing. This signal draws on the shared Knowledge Graph and the entity links attached to each block during enrichment. It is what lets a search for "routing with chi" find a code sample that never spells out the word "routing".

No single researcher is right often enough. The keyword one misses paraphrases; the meaning one drowns in near-misses; the entity one only helps when the question names something known. Hybrid retrieval sends all three at once and pools their findings into one candidate set, so a paragraph found by any of them gets a fair hearing.

flowchart LR
    Q["Your question<br/>'securing service-to-service calls'"]
    Q --> K["Keyword<br/>exact words & phrases"]
    Q --> S["Meaning<br/>semantic / embeddings"]
    Q --> G["Knowledge graph<br/>entities the query names"]
    K --> POOL["One candidate pool<br/>merged & de-duplicated"]
    S --> POOL
    G --> POOL
    POOL --> RR["Re-ranking<br/>best passages first"]
    RR --> R["Block-level results"]

Which signals run is up to the query

Hybrid is the default because it is the safest general answer, but a caller can lean the blend one way. A copilot after pure conceptual matches can ask for meaning only; a tool looking up an exact error string can ask for keyword only. The knowledge-graph signal joins automatically whenever the query names an entity the platform recognises — you never have to ask for it by hand.

Block-level results

Results come back as blocks — the individual paragraphs, headings, code samples, tables, and images that make up a Resource — each with a relevance score and a pointer to its parent Resource. That is the whole point: you see the passage that matched, and you learn which book, chapter, or article it lives in, in one round trip.

Because a single long, highly relevant Resource could otherwise fill an entire page of results, a query can cap how many blocks come from any one Resource, so a page of results spreads across several sources rather than being dominated by one. A "more like this" rail, for instance, usually wants one block per Resource so that every card is a different piece of content.

If a caller genuinely needs the full text of the parent Resources — an AI assistant that wants whole chapters, say — it can ask for the parent bodies to be returned alongside the block hits in the same response, rather than fetching each Resource separately afterwards.

Clips from video and audio

Text isn't the only thing search understands. Video and audio are embedded natively — multimodal models watch and listen to the recording itself, capturing tone, visuals, and pacing alongside the words — and the recording is indexed as time-coded segments rather than blocks.

Ask a question and, when the best answer lives in a recording, the result is a clip: the relevant span, between 10 seconds and two minutes, aligned to natural boundaries such as scene changes and speaker turns. The hit carries the parent Resource, the time range, and a playable reference — no scrubbing through an hour of video to find the demo you half-remember.

Transcripts still exist and still matter: every recording's speech is transcribed (and its scenes narrated) into a text document that is searched like any other — and the transcript is timed, every utterance carrying its place on the recording's timeline. A keyword match on words someone spoke at 3:20 doesn't hand you a wall of text to scrub through; it returns the quote and a playable clip of roughly 3:15–3:25. The two routes complement each other — the transcript catches what was said; the native segments catch what was shown but never said.

And a single search can blend text and media: ask once, and paragraphs from books compete with clips from courses in one ranked list, so "where was raft leader election explained?" is answered by whichever passage or moment explains it best.

Filters

Retrieval decides what is relevant; filters decide what is allowed to appear at all. Filters run natively inside retrieval, so a narrowed search is not a big search with the misfits thrown away afterwards — the index never considers the excluded blocks in the first place, which keeps filtered queries fast regardless of how much content the tenant holds.

The filters a business user reaches for most:

  • Tags — the user-applied tags on a Resource, e.g. only content marked as published, or only a given topic. Match all of a list, or any.
  • Labels — system-generated labels: editorial and analytics signals like trending or high-completion.
  • Language — the Resource's primary language, a first-class field on the Resource itself (not a label), e.g. English-only results.
  • Block types and styles — restrict to prose paragraphs and code samples but not images, or to headings only. Useful when an AI assistant wants prose and code but not figure captions.
  • Specific Resources — scope a search to a chosen set of Resources, for "search within this book".
  • Dates — filter by when the parent Resource was created or last updated.
  • Rights — the most consequential filter. Ask only for content whose resolved rights permit a given usage type — most often AI_RETRIEVAL — or that is cleared for a given territory. Because rights are indexed attributes, this returns pre-cleared results in one pass, with no client-side sifting.

Scoped search: catalogues, entitlements, libraries

The scoping question every consumer product hits eventually is "search everything this user has access to" — their subscription tier, the products they've bought, a free-access promotion that ends on Sunday. The Content Lake answers it with collections: neutral membership keys carried on every block, projected by the services that own the concepts — product membership by Product Management, catalogue membership for storefront corpora, entitlement pools by the Entitlement service.

At query time, the platform resolves what the caller holds — a few pools plus their purchases — into one filter. The elegant consequence: a purchase indexes nothing, and a free-content weekend starting or ending re-indexes nothing. Users gain or lose a key; the content never moves. "Search my library" stays one fast query however the commercial packaging shifts.

Rights are enforced, not merely offered as a filter

On consequential surfaces — AI retrieval endpoints, distribution tooling — the rights predicate is applied server-side and is not optional: a caller cannot ask an AI surface to skip it. Content whose rights prohibit AI_RETRIEVAL stays in the index and stays findable in ordinary search — it is filtered out at query time on every AI surface, so it never reaches a model. Rights are held on the Work, projected onto each Resource, and revalidated at the point of use — search is one link in that chain, not the whole of it. See Works & rights.

Search sessions and stable pagination

A single search can match more passages than fit on one page. Rather than cursors or numbered offsets, search uses a search session: the first response carries a session ID (a readable srch_… identifier), and asking for the next page means re-sending the same query with that ID attached.

The session remembers every block it has already handed you, so the service automatically excludes results you have already seen when it fetches the next page. You never get the same paragraph twice, and you never have to track offsets yourself — the platform does the bookkeeping. Sessions expire on their own after a period of inactivity.

sequenceDiagram
    participant C as Caller
    participant S as Search
    C->>S: Query (no session)
    S-->>C: Page 1 + session srch_microservices_auth
    C->>S: Same query + srch_microservices_auth
    Note over S: Excludes page-1 blocks automatically
    S-->>C: Page 2 (only new blocks)

Sessions are bounded on purpose

A search session returns up to a fixed ceiling of total results across all its pages. Search finds the best passages, not an exhaustive dump; if you find yourself paging deep into the hundreds, a tighter query or a filter is almost always the better tool. The ceiling keeps the exclusion list — and therefore every page — fast.

Re-ranking

The three researchers between them return far more candidates than a caller wants — a wide pool chosen for recall, so nothing relevant is missed. A final re-ranking pass reads each candidate passage against the original question and scores how well it actually answers it, then sorts best-first and trims anything below a caller-set relevance floor.

The division of labour is deliberate: retrieval casts a wide net cheaply, and re-ranking spends more effort judging a smaller set precisely. It is the difference between "these hundreds of passages are in the right neighbourhood" and "these ten actually answer the question". The relevance score on each result is the re-ranker's verdict; a copilot can demand a high floor to keep only high-confidence context and reduce the risk of an AI answer drifting.

Depth is a dial, not a fixed cost. The default re-ranks the top hundred candidates — enough precision for interactive search while keeping it comfortably sub-second. A RAG caller chasing maximum precision can ask for a deeper (and costlier) pass over the whole candidate pool; a high-volume browse surface can skip re-ranking entirely and take the merged retrieval order.

Ranking beyond relevance

Relevance is rarely the whole story a product wants to tell. Search supports the rest of it without a second system:

  • Newest first. Results can be ordered by creation or update date instead of relevance — the browse listings behind "latest additions".
  • Best-selling and most-read. The platform refreshes performance data onto the index daily — sales over 30/90/365 days, views and completed reads — so "best sellers this quarter" is a native ordering, and a relevance search can boost popular content without drowning out the precise answer.
  • Trending. Editorial signals land as labels, applied and removed by the platform as consumption shifts — a "trending now" rail is just a filter.
  • Facets and counts. A search can return the counts behind a filtered browse UI — how many results per tag, per language, per content type, and the total — in the same round trip as the results.

The performance data closes a loop: search reports what users clicked, read, added to basket, and bought (attributed back to the search that surfaced it), the analytics estate aggregates it nightly, and tomorrow's ranking is a little better informed than today's.

Adopting better models without re-indexing downtime

The meaning signal depends on the model that turns text into fingerprints, and those models keep improving. The platform is built so that adopting a better one is routine rather than disruptive. Each embedding model is registered as a named embedding space — configuration, not architecture — and the index can hold more than one space at a time.

Practically, that means a new model can be introduced and back-filled across content quietly, in the background, while live search keeps serving from the current space. Once its quality is proven side by side with the incumbent, search switches over to it. There is no window where search is offline for a re-index, and no moment where results are half old-model and half new. The doc deliberately names no model as a commitment: which model backs the default space is an operational choice that can change without any change to how you search or to the shape of your results.

Model-agnostic by design

Because the embedding model is registered configuration rather than something baked into the query, the same request works unchanged across model generations. Callers evaluating a candidate space can aim a query at it explicitly for a side-by-side comparison, but no caller ever has to know which model is in play. This is the property that lets the platform keep search quality current over years without re-indexing downtime.

Worked examples

Two common shapes of search, with the exact request and response for the technically curious. Both are copy-paste valid against the Public API; IDs use readable slugs in place of the random suffix.

An editor planning a new chapter wants to see every existing passage across published content that covers microservices authentication — to avoid duplication and spot gaps. Hybrid mode gives the broadest net; block-level results mean they see the paragraphs themselves, not a list of book titles; a modest relevance floor keeps recall generous for manual review.

Request — editorial hybrid search (JSON) 
POST /v0/content-lake/search
Authorization: Bearer <workos-jwt>
Content-Type: application/json

{
  "query": "microservices authentication patterns",
  "search_mode": "hybrid",
  "limit": 10,
  "filters": {
    "tags": ["publishing/status/published"],
    "languages": ["en"]
  },
  "min_score": 0.3
}
Response — block-level hits with a session (JSON) 
{
  "search_id": "srch_microservices_auth",
  "results": [
    {
      "score": 0.91,
      "block": {
        "block_key": "b5",
        "block_type": "block",
        "style": "h2",
        "order": 5,
        "content": "OAuth 2.0 Token Exchange for Service-to-Service Authentication",
        "highlight": "OAuth 2.0 Token Exchange for Service-to-Service <<Authentication>>"
      },
      "resource": {
        "resource_id": "res_securing_microservices",
        "title": "Securing Microservices Architectures",
        "version": "ver_secmsvc_ed1",
        "tags": ["publishing/status/published"],
        "language": "en"
      }
    },
    {
      "score": 0.84,
      "block": {
        "block_key": "b6",
        "block_type": "block",
        "style": "normal",
        "order": 6,
        "content": "In a microservices architecture, each service must independently verify the identity of callers. Mutual TLS provides transport-level authentication, while JWT bearer tokens handle application-level identity propagation across service boundaries.",
        "highlight": "In a <<microservices>> architecture ... Mutual TLS provides transport-level <<authentication>>..."
      },
      "resource": {
        "resource_id": "res_securing_microservices",
        "title": "Securing Microservices Architectures",
        "version": "ver_secmsvc_ed1",
        "tags": ["publishing/status/published"],
        "language": "en"
      }
    }
  ],
  "total_candidates": 342,
  "cumulative_returned": 10,
  "has_more": true
}

To fetch page two, the editor re-sends the same query with "search_id": "srch_microservices_auth". The service excludes the two blocks already returned automatically — no offsets, no duplicates.

An AI assistant assembling context for an answer wants a handful of high-confidence passages, restricted to prose and code, and — this is non-negotiable on an AI surface — only content contractually cleared for AI retrieval. It leans the blend toward meaning, sets a high relevance floor to keep only strong matches, and filters on the AI_RETRIEVAL usage right. The rights predicate is enforced server-side on this surface; it is shown explicitly here for clarity.

Request — RAG retrieval, rights-gated (JSON) 
POST /v0/content-lake/search
Authorization: Bearer <workos-jwt>
Content-Type: application/json

{
  "query": "how to configure TLS certificates in Nginx",
  "search_mode": "vector",
  "limit": 5,
  "candidate_pool_size": 1000,
  "min_score": 0.5,
  "filters": {
    "rights": { "usage_type": "AI_RETRIEVAL", "status": "PERMITTED" },
    "block_types": ["block", "codeBlock"]
  }
}
Response — high-confidence, rights-cleared blocks (JSON) 
{
  "search_id": "srch_nginx_tls",
  "results": [
    {
      "score": 0.78,
      "block": {
        "block_key": "b24",
        "block_type": "codeBlock",
        "order": 24,
        "content": "server {\n  listen 443 ssl;\n  ssl_certificate /etc/nginx/ssl/cert.pem;\n  ssl_certificate_key /etc/nginx/ssl/key.pem;\n  ssl_protocols TLSv1.2 TLSv1.3;\n}",
        "highlight": null
      },
      "resource": {
        "resource_id": "res_nginx_config",
        "title": "Production-Ready Nginx Configuration",
        "version": "ver_nginx_ed1",
        "language": "en",
        "rights": { "usable": true }
      }
    },
    {
      "score": 0.72,
      "block": {
        "block_key": "b23",
        "block_type": "block",
        "style": "normal",
        "order": 23,
        "content": "To enable HTTPS, generate a certificate signing request, obtain a signed certificate from your CA, and configure Nginx to reference both the certificate chain and private key files.",
        "highlight": "To enable HTTPS, generate a certificate signing request ... configure <<Nginx>> to reference both the <<certificate>> chain and private key files."
      },
      "resource": {
        "resource_id": "res_nginx_config",
        "title": "Production-Ready Nginx Configuration",
        "version": "ver_nginx_ed1",
        "language": "en",
        "rights": { "usable": true }
      }
    }
  ],
  "total_candidates": 1000,
  "cumulative_returned": 5,
  "has_more": true
}

A copilot typically consumes only the first page. The search_id is still returned, so the assistant can pull more context if an answer needs it — without re-seeing what it already has.

For the technically curious — what runs under the hood

The three signals are executed as parallel sub-queries and merged into one candidate pool before re-ranking, so an entity-matched block competes on relevance with keyword- and meaning-matched blocks rather than being appended. Rights are first-class index attributes, so rights predicates run during retrieval, not as an afterthought. Access control (who may see a Resource, as opposed to what the contract permits) is checked after retrieval and before re-ranking, per parent Resource — the re-ranker never scores what the caller cannot see, and staged or private content an owner can search never leaks into anyone else's results. A page may still return fewer than the requested number of hits after relevance trimming — the session model tolerates short pages. Search never invents a second content format: it points at the same PCF blocks the Content format page describes.