Composition & format generation

Every product on the platform has to answer two questions: what is it made of? and how does a customer get a file? This page covers both. A product's composition is the declared list of what goes into it — the parts, chapters, and lessons, in order — and it points at content that lives in the Content Lake rather than holding copies of its own. From that one declaration the platform generates whatever file a customer needs: the PDF, the ePub, the SCORM package. The book is never stored as a finished file; it is assembled each time it is asked for.

That is the deliberate departure from how publishing usually works. In the old model each format is a separate file that someone produces, stores, and keeps in sync by hand — so the same book ends up as a dozen slightly different files, and fixing a typo means re-exporting all of them. Here there is one source of truth for the content (Resources in the Content Lake) and one declaration of structure (the composition). The files come out of a pipeline on demand, so a correction to a chapter flows automatically into every format that draws on it.

Not every product works this way, and that is fine. A format can also be an uploaded file — an ePub delivered by a partner, a video, an audiobook, a print-ready PDF from an external typesetter. The platform stores those as they are and never picks them apart. The choice is made per format, not per product, so a single book can pair a generated ePub with an uploaded print PDF. This page walks through the composition tree, how each piece of content is pinned to a version (or left tracking the latest), how generated files are produced with no storage in between, and how an uploaded-file product is migrated to a generated one without any disruptive cutover.

The vocabulary here — Work, Expression, Manifestation, and Product — is the four-rung ladder introduced in Core concepts. If those terms are new, read that page first; this one assumes them.

The composition tree

The composition is a product's answer to "what is this made of?". It is a single ordered tree, held on the product, and it is the one source every generated format derives from — the PDF, the ePub, and the SCORM package of a product are all projections of the same tree. Reorder a chapter once and every format reorders with it.

The tree is built from three kinds of node:

• Content nodes are the leaves that reference actual content. Each points at a Content Lake Resource (a chapter, say) plus a version specifier, or — when a product reuses only part of an existing Work — at a ContentUnit, a named, durably-identified part of a Resource.
• Container nodes are the structural groupings that give the product its shape: a PART holding several chapters, a MODULE holding lessons. They carry a title and an ordered list of children but no content of their own.
• Generated nodes are placement markers for material produced during rendering rather than drawn from the lake — the imprint page, the table of contents, the index. A generated node says "the table of contents renders here", not "here is the table of contents".

The node types are borrowed straight from the Content Lake's ContentUnit vocabulary — PART, CHAPTER, SECTION, FRONT_MATTER, BACK_MATTER, APPENDIX for books; MODULE and LESSON for courses — so the product side and the content side speak the same language. Books compose PART → CHAPTER → SECTION; courses compose MODULE → LESSON.

flowchart TD
    P["Product · Book<br/>Hands-On Machine Learning with Go"]
    P --> FM["Front matter<br/>generated: imprint, contents"]
    P --> PT["Part I — Foundations<br/>(container)"]
    P --> BM["Back matter<br/>generated: index"]
    PT --> C1["Chapter 1<br/>Resource · pinned version"]
    PT --> C2["Chapter 2<br/>Resource · tracks latest"]
    PT --> C3["Chapter 3<br/>ContentUnit · reused part"]

A content node holds a reference, not a copy. If the Content Lake Resource it points at is deleted, the reference simply breaks — the product never cached the content, because a single source of truth is the entire point. This is why the composition is safe to reorder and re-version freely: it is a lightweight declaration, not a store of bytes.

{
  "composition": [
    {
      "node_type": "FRONT_MATTER",
      "generated": ["imprint", "toc"]
    },
    {
      "node_type": "PART",
      "title": "Part I — Foundations",
      "children": [
        {
          "node_type": "CHAPTER",
          "resource_id": "res_ml_go_ch1",
          "version": "ver_0195aaa"
        },
        {
          "node_type": "CHAPTER",
          "resource_id": "res_ml_go_ch2",
          "version": "latest"
        }
      ]
    },
    {
      "node_type": "BACK_MATTER",
      "generated": ["index"]
    }
  ]
}

The table of contents is a projection

The table of contents is never stored — it is projected from the composition every time a format is generated. The container and content nodes supply the top levels; the heading blocks (H1, H2, H3) inside each referenced Resource supply the deeper levels as content streams through the pipeline. How deep it goes is a packaging setting on the format, so a print table of contents can stop at chapters while the ePub navigation reaches three levels. The same projection drives an ePub's navigation document and a SCORM package's organisation tree — one tree, many renderings.

Composition and ContentUnits

The composition tree looks a lot like the Content Lake's own HAS_PART ContentUnit tree, and that resemblance is deliberate — but the two do different jobs. The composition is a mutable product concern: editors reorder chapters and swap versions freely while a product is in draft. ContentUnits are minted in the lake only when a part needs durable identity — for reuse, citation, attribution, or a royalty split.

The two meet when a product draws on parts of existing Works. A course compiled from chapters of several books references those chapters as ContentUnits; the compilation is recorded as an Activity that names exactly which units went in, and a new Work is created for the course with its own rights basis. Composition nodes reference ContentUnits when that traceability is needed; nothing forces every chapter node to mint one.

Version pinning

Content changes over time, and a product has to decide, chapter by chapter, whether it wants those changes. Every content node carries a version specifier that answers one question: should this piece change when its source content is updated?

• version=latest — the node tracks the latest Content Lake version of its Resource. When the Resource is updated, the product reflects the new content automatically.
• version=<id> — the node is pinned to one specific version. Updates to the source are ignored until someone deliberately re-pins to a newer one.

A single product can mix both. A common pattern pins most chapters for stability while letting one fast-moving chapter — a security advisory, a framework migration guide — track latest.

flowchart LR
    subgraph prod["Product: Go Microservices"]
        direction TB
        A["Chapter 1<br/>pinned · ver_0195aaa"]
        B["Chapter 2<br/>pinned · ver_0196bbb"]
        C["Chapter 3<br/>tracks latest"]
    end
    CL["Content Lake<br/>Resource updated"] -. no effect .-> A
    CL -. no effect .-> B
    CL -- flows through --> C

The product-level concurrent flag sets the default for new content nodes. When concurrent is true, new nodes default to version=latest; when false, every new node requires an explicit version ID. It is only a convenience for the common case — either way, any individual node can be overridden.

Re-pinning

Re-pinning moves a content node from one specific version to another. It is always a deliberate action — it never happens on its own — and it changes the specifier from version=<old-id> to version=<new-id>. Re-pinning emits a platform.products.content.v1.updated event on the platform-products bus so downstream consumers know the product's effective content has changed.

That event is distinct from platform.products.product.v1.updated, which fires when product metadata changes. A consumer that cares about content freshness — a storefront invalidating a cached page — subscribes to content.v1.updated; one that cares only about titles and pricing subscribes to product.v1.updated; one that cares about both subscribes to both.

Generating a file on the fly

Every generated format is produced the same way, on demand, and stored nowhere. There is no pre-built PDF sitting on a disk waiting to be downloaded — the request itself triggers a pipeline that walks the composition, pulls content from the Content Lake, injects the product's own assets, renders the target format, and streams the bytes back as they are produced. The customer starts receiving the file before it has finished assembling.

flowchart LR
    REQ["Format<br/>requested"] --> RET["Retrieve<br/>walk composition,<br/>stream from<br/>Content Lake"]
    RET --> INJ["Inject assets<br/>covers, imprint,<br/>contents, index"]
    INJ --> REN["Render<br/>PDF · ePub ·<br/>InDesign · SCORM"]
    REN --> OUT["Stream<br/>to client"]

1. Retrieve. The pipeline walks the composition tree and fetches each content node's Resource from the Content Lake's streaming endpoint, which delivers content block by block with sub-100ms time-to-first-byte. Pinned nodes request their specific version; latest-tracking nodes request the current one.
2. Inject assets. The product's own assets are placed where the composition's generated nodes declare — the front cover first, the imprint after it, the index at the end. The table of contents is projected from the composition and the heading structure as it streams past; the index is built from entity and keyword data in the Content Lake's Knowledge Graph.
3. Render and stream. The assembled stream is rendered into the target format and written to the response as it is produced. Nothing buffers the whole file; the client receives bytes as they are ready.

A SCORM course follows the identical shape: the pipeline walks the MODULE/LESSON tree, renders each lesson to a self-contained HTML learning object, generates the package manifest from the tree and the packaging settings, and streams the resulting zip — the same no-stored-files principle as every book format.

Determinism

When every content node is pinned, generation is deterministic: the same request always produces byte-identical output, which is what a downstream system needs when it verifies an artifact's checksum or caches it behind a CDN. When any node tracks latest, the output reflects the Content Lake state at the moment of generation — two requests seconds apart can differ if a Resource was updated in between, so that output should not be treated as a stable artifact.

The Product Management service does not itself cache generated files. For a fully-pinned product the response is deterministic, so a downstream CDN can cache it with ordinary HTTP caching headers. A system that genuinely needs a persistent file — uploading a fixed PDF to a retail channel, say — generates it, stores it, and manages its own cache. The platform provides the generation capability; storage of the result is the consumer's concern.

Generated assets vs uploaded assets

A product's parts fall into two camps: things the platform generates each time and never keeps, and things that are uploaded once and stored. The dividing line matters because it explains why a fully-generated book stores almost nothing.

For a product whose formats are all generated, the cover images are the only stored binaries. The imprint, table of contents, and index are never persisted — they exist only in the generation stream, produced fresh on every request. This is the whole promise of the no-stored-files design: correct the content or the metadata once, and every future download reflects it, because there is no stale file anywhere to fall out of sync.

Uploaded assets carry product-asset IDs (ast_). An uploaded format artifact is the exception to "no stored files" — the platform keeps the distributable exactly as delivered and treats it as opaque, never decomposing it into Content Lake content. Its internal structure (an ePub's navigation, a SCORM manifest, a PDF outline) is still made visible through a structural descriptor extracted at upload, but the bytes are the deliverable and they are never regenerated. See Product types for the full format and descriptor detail.

Migrating an uploaded-file product to generated

A title that arrived as an uploaded ePub — an acquired backlist item, say — has no Content Lake presence beyond its Work and its Contract. Making it fully platform-native (searchable at block level, rights-filterable in catalogues, with a generated index from the Knowledge Graph) does not require a disruptive cutover. Because sourcing is decided per format, the migration is incremental and nothing has to be retired up front.

flowchart LR
    A["1 · Ingest<br/>source into the<br/>Content Lake<br/>(same Work)"] --> B["2 · Build the<br/>composition<br/>referencing new<br/>Resources"]
    B --> C["3 · Flip the ebook<br/>format from<br/>ARTIFACT to<br/>RENDERED"]
    C --> D["4 · Retire the<br/>stored artifact<br/>once verified"]

1. Ingest the source. Bring the editable source (or a conversion of the ePub) into the Content Lake under the title's existing Work. It gets structured-content conversion, enrichment, and block-level indexing like any other ingested content.
2. Build the composition. Add a composition referencing the new Resources, pinning each node to its reviewed version.
3. Flip the format's sourcing. Switch the ebook format from ARTIFACT to RENDERED. The Manifestation and its ISBN do not change — the ISBN identifies the sellable embodiment, not the file behind it — so nothing downstream sees a new SKU.
4. Retire the stored artifact. Once the generated output is verified, retire the uploaded asset. Retire nothing else: no Manifestation, no ISBN, no product record disappears.

During the transition the product is legitimately mixed — its product_model reflects that some formats are generated and some are still uploaded. A print Manifestation can stay artifact-backed for years after the ebook goes rendered; there is no requirement to convert every format at once.

What the migration buys is what the generated model gives every product: corrections flow through version tracking instead of file re-uploads, the table of contents and index become live projections rather than content baked into a file, rights-aware and content-based catalogue rules can finally see the product (it has a composition now), and the title joins search and AI-retrieval scopes at block level.

Related pages

• Product types — the eight types, their formats, and the full format-configuration and descriptor reference.
• Products overview — the Work → Expression → Manifestation → Product ladder in commercial terms.
• Catalogues — how composed products are grouped for each storefront and channel.
• Core concepts — the vocabulary this page assumes, in dependency order.
• Content Lake — where the content a composition points at actually lives.
• Explorer — Chapter VII — products and the W/E/M ladder, visualised.