Ingestion

Most content does not start life on the platform. It starts as a Word file in a shared drive, a Google Doc an editor is still writing, a PDF a partner emailed over, an ePub from a back catalogue. Ingestion is how all of that becomes usable content in the Content Lake: connect a source once, and the platform watches it, fetches what changes, and turns each document into clean, structured, versioned Resources — with a full record of where every piece came from.

The promise is connect once. You point ingestion at a SharePoint library, a Google Drive folder, or a Dropbox account, and from then on it keeps up: when a document is added or edited, the platform notices, pulls the new bytes, and re-runs the same pipeline. Nobody re-uploads anything; nobody exports to a special format. An author keeps working the way they always have, and their content stays in step on the platform.

The other half of the promise is any document. Manuscripts arrive in wildly different shapes — a born-digital DOCX with real headings, a scanned PDF with no structure at all, an ePub, a Google Doc, a plain text file. Ingestion routes each one down the path best suited to it and produces the same output every time: a single stream of PCF blocks, split into one Resource per chapter, ready to be enriched, composed into products, and searched.

This page walks the journey a document takes — from a source connection to written Resources — and then covers the sources themselves, how re-sync keeps content in step without losing its identity, and the limits and safeguards that apply along the way.

The journey, end to end

Every document takes the same journey, whichever source it came from and whatever shape it arrived in. Ingestion watches the source for changes, fetches the document, extracts structure from it, transforms it into PCF, splits it into sections, and writes those sections to the Content Lake as Resources — each carrying a record of how it got there.

flowchart LR
    S["Source<br/>connected once"] -->|edit detected| W["Watch<br/>webhook fires"]
    W --> F["Fetch<br/>download the file"]
    F --> A["Archive<br/>keep the original"]
    A --> X["Extract<br/>recover structure"]
    X --> T["Transform<br/>to PCF"]
    T --> SP["Split<br/>into sections"]
    SP --> WR["Write<br/>Resources + versions"]

The rest of this section walks each step. None of it is something you operate by hand — once a source is connected, the whole journey runs on its own, every time a document changes.

Watch

A connected source is a watched source. Where the source supports it — SharePoint, Google Drive, Google Docs, and Dropbox all do — ingestion registers for change notifications, so an edit at the source triggers a fetch within minutes rather than waiting for a scheduled scan. You do not poll and you do not re-upload: editing the file where it already lives is the whole workflow.

Notifications are only ever a signal that something changed — they rarely say what. Ingestion holds a per-source cursor (a bookmark into the source's own change feed) and, when a notification arrives, asks the source for the list of documents that changed since the cursor last moved. That keeps the platform in step even when many edits land at once.

Fetch and archive

Once ingestion knows which documents changed, it downloads each one using the credentials of the user who connected the source — so a document is only ever fetched by someone already allowed to see it. The original bytes are then archived permanently, exactly as they arrived, and a checksum of them is recorded.

That permanent archive is deliberate. It is never served to anyone; it exists so that if the platform's extraction tooling improves, historical documents can be reprocessed into better content without going back to the source system at all. The checksum, meanwhile, is what makes re-sync cheap: if a document comes back with the same checksum, nothing has changed and the pipeline stops right there (see Re-sync).

Extract — routing by document type

Extraction is where the platform earns its keep. Different document types carry structure in completely different ways, so ingestion routes each one down the path that recovers the most structure with the least guesswork. The decision is made from the file's type — its source, name, and format — not a flag someone sets by hand.

flowchart TD
    D["A document arrives"] --> Q{"What kind<br/>of document?"}
    Q -->|"Born-digital<br/>DOCX · Markdown · text"| DET["Deterministic converters<br/>structure read directly"]
    Q -->|"ePub"| EPUB["ePub extraction pipeline<br/>spine-ordered content"]
    Q -->|"PDF or scan"| GLM["Vision extraction model<br/>structure inferred from layout"]
    Q -->|"Google Docs"| GDOC["Structured API<br/>maps straight to PCF"]
    Q -->|"No deterministic route<br/>e.g. PPTX, Excel"| REND["Rendering fallback<br/>render → PDF → vision model"]
    DET --> PCF["PCF"]
    EPUB --> PCF
    GLM --> PCF
    GDOC --> PCF
    REND --> GLM

The routes, and why each exists:

• Born-digital documents — DOCX, Markdown, and plain text — already carry the structure PCF wants: real heading levels, list nesting, code fences, table semantics. They are converted deterministically (Word through a document converter, Markdown and text through a normalisation pass), never sent through a model. Laundering clean structure through a layout engine and hoping it survives would only lose fidelity.
• ePub takes a dedicated ePub extraction pipeline — a cheap, deterministic route that unpacks the package and emits its chapters in spine order.
• PDFs and scans are the hard case: their structure lives only in the visual layout, so ingestion sends them to a vision extraction model that reads each page and recovers headings, paragraphs, tables, code, and figures. Images inside the document are pulled out and stored as assets the lake can serve.
• Google Docs skip extraction entirely. The Google Docs API returns a typed document tree, so ingestion maps that structure straight to PCF — no model, no intermediate format, faster and cheaper.
• Anything with no deterministic converter — a PowerPoint deck, a spreadsheet — takes the rendering fallback: the document is rendered and printed to PDF, then handed to the vision model like any other PDF. Some connectors do this conversion for you at fetch time (see Sources).

Transform to PCF

Whichever route a document took, it converges here. Transformation is the single stage that turns every extracted document into PCF — the one representation the Content Lake stores. There is one shared converter, so a chapter arrives as the same PCF whether it started as a DOCX, a PDF, or an ePub. That is what "any document" means in practice: the differences between formats are resolved before content reaches the lake, not carried into it.

For PDFs, transformation also does page assembly — quietly undoing the artefacts of print. It strips running heads and page numbers, re-joins paragraphs and code blocks split across a page break, and repairs words hyphenated at line ends, so what reaches the lake reads as continuous content rather than a sequence of pages.

Every block of content gets a stable key at this stage — an internal identity that later lets citations, links, and enrichment point at an exact paragraph. Keeping those keys stable across re-syncs is what makes re-sync safe, and it is covered there.

Split into Resources

A whole manuscript is not the useful unit — a chapter is. Splitting divides the assembled PCF stream into typed sections, normally one per top-level heading, and each section becomes one Content Lake Resource. A ten-chapter book yields at least ten Resources, which is what lets product tooling later assemble books, courses, and reports from atomic parts rather than one monolithic file.

Splitting also classifies each section — is this a chapter, a table of contents, a copyright page, an index, an appendix, a cover? — using ordered heuristics, with anything ambiguous sent to an editorial review queue rather than blocking the pipeline. Everything is written either way; review reclassifies after the fact, so ingestion never waits on a human.

Write — with full provenance

The final stage writes each section to the Content Lake through the public API. It imports the document's images and figures as assets, registers the Source the content came from, creates or updates one Resource per section, and snapshots a Version of each — so a completed ingestion is an immutable, pinnable moment: this is what the source said at this point in time.

Crucially, every Resource is written with an Activity that records exactly how it came to be: the ingestion pipeline as the system agent, the person who connected the source as the human agent, the tools and their versions, the checksum of the original bytes, and the source location. Provenance is not added later — it is part of the write. That is what makes "what did this come from?" a single query rather than an investigation.

The sources

A source is where documents live. Ingestion ships connectors for the places Packt content actually sits, plus a connector-free direct-upload path for everything else. Each connector handles its own authentication, fetching, and change detection, but they all feed the same pipeline above — so once a document has been fetched, its origin no longer matters.

A few things worth calling out:

• You connect as yourself. Every connector uses delegated access — ingestion acts as the user who connected the source, so it can only ever reach documents that user is already allowed to see. There is no shared service account quietly reading everything.
• SharePoint and Google Drive can hand you a PDF. Both can convert native Office files to PDF at fetch time, which brings formats with no deterministic converter (PowerPoint, Excel) neatly into the vision route without any local tooling.
• Google Docs rides the Drive change feed. A single Google connection watches everything; a change to a native Google Doc is routed to the structured-API path, and everything else to the standard path, purely by file type.
• Direct upload is the escape hatch. No account, no OAuth, no webhook: request a short-lived upload link, push the bytes, and import. It covers one-off manuscripts, files from systems without a connector, and content pushed programmatically by other Packt services. Because it has no change feed, re-importing a corrected file is always an explicit action.

Re-sync: keeping content in step

Here is the property that makes connect once actually work: edit the file in SharePoint and the platform keeps the same content identity. A re-sync is not a fresh import that leaves a duplicate behind. It updates the same Resources, preserves the links and citations pointing into them, and adds a new version — so the history of a document stays continuous no matter how many times it is edited at the source.

When a change arrives — by webhook, by scheduled crawl, or by a manual refresh — the same logic runs every time:

flowchart TD
    C["Change detected<br/>at the source"] --> H{"Same checksum<br/>as last time?"}
    H -->|Yes| NOOP["No-op sync<br/>nothing re-written"]
    H -->|No| RUN["Re-run the pipeline"]
    RUN --> M{"Match each section<br/>to an existing Resource"}
    M -->|"Heading matches"| U["Update that Resource<br/>keys preserved"]
    M -->|"Same position"| U
    M -->|"No match"| N["New Resource"]
    M -->|"Old section gone"| K["Keep it<br/>tag 'source-removed'"]
    U --> V["Snapshot a new version"]
    N --> V

The pieces that give re-sync its guarantees:

• Unchanged files cost almost nothing. The checksum recorded at archiving is compared against last time. If the bytes are identical, the sync short-circuits — no extraction, no writes, no new version. This is what makes watching a large estate affordable: an unchanged document is one cheap check.
• Sections keep their identity. When a document does change, each new section is matched to the Resource it was before — first by its heading, then by its position — so a reordered chapter or a renamed one keeps the same Resource. New sections mint new Resources; only genuinely new content gets a new identity.
• Internal anchors survive. Within a matched Resource, blocks that did not change keep their stable keys, so entity overlays, internal links, and citations that point at an exact paragraph stay anchored across the edit. Only content that actually changed gets a new key.
• Nothing is silently deleted. A section removed at the source does not delete its Resource — it might be pinned into a product or cited elsewhere, which ingestion cannot see. The Resource is kept and tagged ingestion/source-removed for an editor to judge. If the section comes back, the tag is removed and updates resume.

The upshot: an editor works in SharePoint or Google Docs exactly as they always have, and the platform mirrors their changes onto stable, versioned Resources — no re-uploads, no duplicates, no broken links.

Limits and safeguards

A few boundaries and protections apply to everything ingestion processes, whatever the source. They are collapsed here because they rarely need to be front-of-mind — but they are worth knowing before you point ingestion at a very large estate or accept files from outside.

Related pages

• Content Lake overview — what ingestion feeds, and why content is held once.
• Content format (PCF) — the structured representation every document is converted into.
• Resources & versions — the units ingestion writes and the snapshots it takes.
• Works & rights — how ingested content becomes usable under a Contract.
• Provenance & attribution — the Activity record every written Resource carries.
• Explorer — Chapter II — the ingestion journey, visualised.