How Magellan Provides Data Lineage and Provenance

Two Questions Every Data Team Eventually Faces

“Where did this field come from?” is usually the first question. The harder follow-up is: “Who decided to transform it this way, which definitions were in force, and can we trust the record?”

When those answers live in disconnected spreadsheets and scripts, every investigation repeats the work. Source changes create uncertainty, retired engineers take context with them, and a current diagram cannot explain a historical decision.

Magellan treats mappings as first-class metadata assets. Its lineage model describes the relationships between source fields, transformation rules, and target fields. Its provenance architecture preserves the versions, actors, and decisions behind those relationships. Lineage shows the path; provenance explains the path’s origin and history.

Scope matters: this article describes Magellan’s metadata model and its documented Cryptographic Audit & Trust Ledger architecture. The ledger and evidence-pack export are the next module being built, not capabilities already deployed at Optum; that deployment uses predictive mapping. The public evidence pack is a synthetic, independently verifiable example, not a customer production record.

Lineage and Provenance Are Related, Not Interchangeable

Neither a useful graph nor a protected history automatically proves that the source data was correct. They make the transformation and its governance inspectable.

Start with the Mapping, Not a Drawing

Normalize the source schema(s) and target schema(s)

Magellan’s unified MetaModel gives schemas imported from different formats a common representation. Fields, hierarchy, types, and constraints become metadata that a mapping can reference rather than labels drawn on an unrelated diagram.

When a mapping spreadsheet is imported, it creates the source schema(s), target schema(s), and mapping at one shared version at import only. Afterwards, they version independently, with each mapping version pinning the specific schema versions it references.

That pinning is essential to historical lineage. “Patient identifier maps to subject identifier” is incomplete without knowing which source definition, which target definition, and which mapping version established the relationship.

Preserve the field relationships and rule context

A useful lineage edge identifies more than two system names. It connects a source field to a target field in the context of a mapping and any associated transformation rule. A normalization, lookup, or combination of fields is part of the explanation, not something to hide behind a line.

The persistent knowledge graph organizes those relationships for lineage exploration. Backward traversal helps investigate a target field’s dependencies. Forward traversal helps assess the effect of changing a source field. Where intermediate mappings are represented, traversal can follow multiple hops rather than stop at the first connection.

Coverage is bounded by the metadata captured. An undocumented transformation in an external job does not become visible merely because Magellan knows the schemas on either side. External dependencies must be imported or integrated to extend the graph’s coverage.

Make the Decision History the Basis of the Graph

The documented automated lineage design derives lineage from ledger events instead of maintaining a separate, unexplained diagram. Mapping events such as MappingDrafted and MappingRevised provide the context from which the lineage projector records LineageEdgeDerived entries.

This separates two things that are easy to confuse:

• The event history records the decisions and changes.
• The graph projection presents the relationships derived from that history.

Pinning a projection to its ledger range gives a historical view a reference point. A later mapping revision should not silently replace the explanation for an earlier release. Historical reconstruction also depends on retaining the referenced schema versions, mapping artifacts, and projection logic—not just the event identifiers.

Capture Why a Mapping Was Accepted—or Rejected

The Semantic Match Engine proposes candidate field relationships with confidence, algorithm class, and rationale. A suggestion is evidence of what the engine proposed; it is not evidence that a reviewer approved it.

The provenance design keeps those stages distinct. Suggestion context includes the model and policy versions used. Review events preserve the human decision and its rationale. Rejected candidates matter too: they explain why a superficially plausible field pairing was not used.

Approval has a strict dependency order: every source schema version and target schema version pinned by a mapping version must be Approved & Published before that mapping version can be submitted for approval. Schemas approve first; the mapping approves afterwards.

That order gives an approved mapping a stable semantic foundation. If a target schema later changes, the older mapping still references the version that was actually reviewed.

A Healthcare Example: The Same Label, Different Meaning

Consider a hypothetical EHR export with both patient.local_id and patient.enterprise_id. A downstream registry has a field named subject.identifier. Matching on the word “identifier” alone does not establish which source field is appropriate.

Lineage should show the chosen source field, any identifier normalization rule, and the target field, tied to the relevant mapping version.

Provenance should explain the decision: the local identifier was rejected because its scope was limited to one facility; the enterprise identifier was selected under the registry’s agreed definition; the reviewer recorded the rationale against the pinned schema versions.

If the registry’s definition changes later, the team can examine which mapping versions depend on it. If a governance reviewer asks about a previous release, the team can inspect the definitions and decision history for that release rather than substitute today’s mapping.

This is an illustrative example, not a claim about a particular customer’s data or workflow.

Protect the History—and State What Protection Means

The Cryptographic Audit & Trust Ledger design assigns events ULIDs and records actor context, role at the time of action, timestamps, subjects, and version references. Canonical payloads are hashed with SHA-256, and entries are linked using:

entry_hash = sha256(prev_hash || payload_sha256)

Changing a payload without changing its recorded hash breaks verification. Recomputing a modified chain can be detected by comparison with a separately retained, trusted chain head. Append-only database controls and externally retained seals complement the mathematics; a hash chain alone does not prevent a privileged operator from rewriting both records and hashes.

Tamper-evident is not the same as true. Hashes can establish consistency with a trusted record, not whether an original reviewer’s judgment was correct. They also do not, by themselves, guarantee an identical rerun of a model whose runtime or inputs are no longer available.

The architecture targets audit evidence for HIPAA §164.312(b), 21 CFR Part 11 §11.10(e), and CMMC L2 §3.3 AU-family controls. It does not confer certification or replace an organization’s access controls, validation, retention policy, or assessment.

Carry Provenance into Exported Artifacts

The export design records an ArtifactExported event with the mapping ULID, artifact format, consumer identifier, and SHA-256 digest of the emitted bytes. A downstream team can associate a received artifact with the recorded export and compare its digest.

That establishes artifact-level provenance. It does not prove that a downstream job executed the artifact unchanged, or trace each individual patient or transaction record. Record-level execution lineage requires runtime instrumentation and correlation with the consuming pipeline.

Keeping that boundary explicit helps teams ask for the right evidence: metadata lineage for the intended transformation, artifact provenance for what was delivered, and execution evidence for what actually ran.

Inspect the Evidence Yourself

The sample evidence pack illustrates a mapping lifecycle with synthetic identities and payloads. Its machine-readable JSON and independent Python verifier let you check the example’s hashes without running Magellan.

• Download the JSON and verifier into the same folder.
• Run python3 verify_chain.py magellan_sample_evidence_pack.json to verify the sample chain.
• Add --tamper to demonstrate detection of a changed payload.

The sample proves the mechanics of that supplied record. It is not evidence of a production ledger deployment or a complete implementation of every lineage event described above.

The practical outcome: fewer investigations that start from memory, mapping versions with explicit dependencies, and a historical record that can be checked rather than merely trusted.

Request a demo to discuss lineage coverage for your source schema(s), target schema(s), and consuming pipelines. For the broader governance argument, read Transformation Debt, and the Record That Pays It Off.

Magellan AI Corporation — Charting every data connection.