Skip to content

Content Lake

The Content Lake is the platform's single content estate: one shared place where every piece of content Packt holds lives, from a manuscript that arrived this morning to a chapter that has been translated, narrated, and shipped in a dozen products. It is the foundation the rest of the platform stands on — Contracts govern the content held here, Products are composed from it, and Distribution sends it out. If you understand the lake, the rest of the site falls into place.

The core idea is that content should be held once and reused everywhere, rather than copied into a fresh file for every format and project. When a book arrives, its chapters become Resources in the lake. From that single copy the platform can produce an ebook, an audiobook, a course, a newsletter, or an answer given by an AI assistant — without anyone re-keying the text, and without losing track of who wrote it or what may be done with it. That last part matters as much as the reuse: every Resource carries its attribution and its resolved rights, so reuse never means losing the thread back to the author or the contract.

Two things make the lake more than a file store. First, content is kept as structured blocks in the Packt Content Format (PCF) — not opaque documents — so any part of it can be addressed, reused, and rendered into any output. Second, content is connected: Resources link to the shared Knowledge Graph so a mention of a technology or a person joins a live, shared record, and every transformation is written down as an Activity so the estate knows its own history.

This page is the overview: what the estate is, the journey content takes through it, the handful of entities it is built from, the goals that shaped it, and where to go next. Anything unfamiliar links to the Glossary.

The content journey

Content moves through the estate in four stages. It is ingested from wherever it lives today, enriched with context, derived into new forms, and distributed to the products and channels that reach customers. The same single copy underlies all four — nothing is duplicated as it moves.

flowchart LR
    ING["Ingest<br/>Sources → Resources"] --> ENR["Enrich<br/>Knowledge Graph · metadata"]
    ENR --> DER["Derive<br/>translations · narrations · courses"]
    DER --> DIS["Distribute<br/>Products · Channels"]

  1. Ingest. Content is imported from external Sources — SharePoint, Google Drive, a Git repository, or a plain upload — and registered as Resources under a Work. A big manuscript is typically split into one Resource per chapter, all sharing one Source. See Ingestion.

  2. Enrich. Resources are linked to the shared Knowledge Graph so content gains context — currency, credibility, relationships — without that context being copied into the text. Enrichment is recorded as log-only Activities; it adds knowledge, it does not create new content.

  3. Derive. New content is produced from existing content: translations, audiobook narrations, course adaptations, compilations, assessments. Each derivation is a graph Activity that creates a new node and records what it came from — so lineage and royalty attribution are recorded fact, not later investigation.

  4. Distribute. Manifestations are assembled from the Work's Resources into sellable Products and offered to Channels. The content itself never leaves the lake as anything but PCF; the shape a customer receives is produced at the edge (see PCF only, rendered at the edge).

Rights and provenance travel the whole way

The single thread running through all four stages is the Contract. Rights are resolved on the Work and carried with every Resource, so a term set once — say, ebooks permitted, audiobooks prohibited — constrains everything downstream automatically. Provenance is the same: every step writes an Activity, so the estate always knows where a piece came from and who is owed.

The entity model

The estate is built from a small number of entity types. Five of them matter for a first read; the rest of this section links to the pages that go deep.

flowchart TD
    S["Source<br/>where content came from"] -->|split into| R["Resource<br/>the unit of content"]
    R -->|belongs to| W["Work<br/>the IP · holds rights"]
    R -->|has parts| CU["ContentUnit<br/>chapter · section · segment"]
    A["Activity<br/>the provenance record"] -->|produced| R
    CU -->|belongs to| W
  • A Resource is the unit of content: one document, image, video, audio file, or code repository. A single chapter is a Resource (res_ml_go_ch3). Resources are stored lean and normalised, then stitched back together on reads.
  • A Source is the external artifact a Resource was ingested from — the original manuscript file. When one file is split into many chapter Resources, they all share one Source (src_ml_go_manuscript).
  • A Work is the intellectual-property anchor. It holds the title and the rights, and it is where a Contract attaches. Identity rolls up from Manifestation to Expression to Work (wk_ml_go).
  • A ContentUnit is an addressable part of a Resource — a chapter, section, or segment — with its own stable identity so it can be reused, cited, and attributed independently (cu_ml_go_ch3_intro).
  • An Activity is an immutable record of how content came to be: created, translated, narrated, compiled, enriched. Every Activity names the agent responsible and any tooling used (act_ml_go_ch3_translate_fr).

Content is only usable inside a Work under a Contract

A Resource that is not yet attached to a Work sits in staging: owner-only, usable: false, so work can begin before the paperwork is settled. It becomes properly usable — in a Product, in distribution, or on any AI surface — only once it is promoted into a Work that a Contract governs. There is no other route in.

See Resources & versions for the full entity schemas and Provenance & attribution for the Activity taxonomy.

What the lake is for

Six goals shaped the estate. They are summarised here; the Principles page carries the reasoning in full.

  • Liquid content. Ingest content once and let it flow into any downstream format — audiobooks, courses, short videos, newsletters — instead of starting each format from a fresh copy.
  • Publisher context. Beyond raw text, the Knowledge Graph adds context: whether a library is still current, whether a technique has been superseded, whether a known vulnerability applies. Context is what makes content worth more than the words alone.
  • Meaningful AI integrations. Retrieval and MCP integrations should go beyond keyword or similarity matching. The Knowledge Graph gives an AI assistant the sourced context it needs to answer trustworthily rather than regurgitate.
  • Attribution. Every piece of content traces back to its author. This is an author-motivation concern as much as an end-user one: if AI consumes content without attribution or click-through, authors lose the reason to publish. Attribution is first-class here.
  • Contractual alignment. All content is held with explicit contractual terms, enforced at the platform level. Resolved rights travel with every read and search hit; consequential surfaces (AI retrieval, distribution, publish) apply rights predicates server-side; point-of-use revalidation backs the projections. Enforcement is not left to downstream apps.
  • Provenance. Every transformation is recorded as an immutable Activity with full agent attribution, so "what did AI touch?", "what is this derived from?", and "who is owed?" are queries against recorded fact, not investigations.

Rights: held on the Work, cached per Resource

Content must belong to a Work under a Contract before it can be used. Rights are held once on the Work — the source of truth — and projected onto each of the Work's Resources, so every read and filtered query is self-contained and already knows what may be done. Contract changes trigger reprojection, and consequential actions revalidate against the contract versions pinned in each Resource's rights_basis.

In one line: held on the Work, cached per Resource, revalidated at the point of use. The headline modern example is AI — a contract can permit AI_RETRIEVAL while prohibiting AI_TRAINING, and the platform enforces the difference on every retrieval.

See Works & rights for the rights schema and Contracts & Rights for how contracts resolve.

PCF only, rendered at the edge

The Content Lake serves PCF (JSON) only. PCF is the highest-fidelity representation of content, and every internal service operates on it directly. A read returns either a whole PCF document or a streamed sequence of PCF blocks — nothing else.

Rendered formats — Markdown, HTML, SSML, EPUB, print — are not produced by the lake. They are generated at the Public API edge through content negotiation: the caller states the shape it wants with an Accept header, and the edge renders PCF into that shape on the way out. Keeping rendering at the edge means the lake stays a single, canonical store, and new output formats can be added without touching the content itself.

Correction from earlier drafts

Earlier documentation described a Markdown endpoint on the Content Lake. That is not how it works: the lake serves PCF JSON only, and Markdown (and every other rendered format) is produced at the Public API edge via content negotiation. If you need Markdown, ask the Public API for it — do not expect it from the lake directly.

Asking the edge for a rendered format (HTTP)

The Content Lake read returns PCF by default. To get Markdown, the caller negotiates it at the Public API edge with an Accept header — the edge renders the same PCF on the way out.

GET /v0/content-lake/resources/res_ml_go_ch3 HTTP/1.1
Host: api.packtpub.services
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6...
Accept: text/markdown
Accept value You receive
application/json PCF JSON (the default)
application/x-ndjson Streamed PCF blocks
text/markdown Markdown rendered at the edge

The tenant is taken from the token's tenant_id claim — there is no tenant header. See Build → API.

See Content format (PCF) for the block model and Build → API for content negotiation in full.

Explore the Content Lake

The estate spans several pages. Start wherever your question sits.

  • Works & rights — the Work as the IP anchor, how Contracts attach, and how rights reach every Resource.
  • Resources & versions — the Resource model, ContentUnits, Sources, and the explicit snapshot versioning model (working state latest).
  • Content format (PCF) — how content is held as ordered blocks and spans, and why entities live in an overlay rather than in the text.
  • Ingestion — how content is imported from Sources and registered as Resources under a Work.
  • Enrichment & the Knowledge Graph — how content gains shared, live context.
  • Search — the hybrid pipeline (lexical, semantic, Knowledge Graph) and how rights filter results.
  • Provenance & attribution — the Activity graph, lineage, and answering who is owed?
  • Access & labels — user tags, system labels, and how visibility and access are controlled inside one tenant.