Inside the Semantic Match Engine

Why a Companion Post

The previous post — Predictive Mapping — Why We Don't Say "AI‑Powered" — argued that the engine *proposes* and a named subject-matter expert *decides*. That post deliberately stopped at the boundary of the pipeline. This one opens the box.

Specifically: what happens between the moment two schema field paths arrive at the /semantic-match endpoint and the moment a top‑k list of candidate mappings, each tagged with an integer confidence in [0, 100], lands back in the SME's review queue.

Three stages do the work: embedding, calibrated cosine ranking, and confidence calibration. Each is deterministic at a pinned model version, and each is reproducible from the decision record after the fact.

Stage 1 — Embedding

The engine never compares strings directly. It compares vectors.

Path normalization first

Before embedding, every field is rewritten as a normalized dot-delimited path that preserves hierarchical context — the same convention the Semantic Match API post introduced. patient.demographics.name.given carries more signal than given alone, because the parent segments tell the model *what kind of "given"* this is. Array notation (orders[0].lines[1].item) is preserved so structurally repeated fields don't collapse.

A small set of structural transforms run before the embedding call:

• Casing is folded; underscores, hyphens, and camelCase boundaries are turned into single spaces.
• Known abbreviations from the active domain dictionary (e.g. DOB, MRN, NPI, SSN) are emitted alongside their expansions so both forms enter the embedding context.
• Type metadata, when present in the source schema, is appended as a short suffix (:: date, :: identifier, :: codeable_concept). This is what lets the engine know that patient.id :: identifier is closer to subject.mrn :: identifier than to patient.id :: integer.

What the model returns

The normalized strings are passed to a sentence-level embedding model and returned as L2-normalized vectors. Two properties matter:

1. L2 normalization means cosine similarity reduces to a dot product, which is fast and numerically well-behaved.

2. A pinned model version (semantic-match-v2.4.1) is recorded in every decision record. If the embedding model changes, the version tag changes, and downstream replay against historical decisions stays unambiguous.

The output of Stage 1 is a pair of vectors per candidate: (v_source, v_target). Nothing has been ranked or scored yet.

Stage 2 — Calibrated Cosine Ranking

Raw cosine similarity between two embeddings is a real number in [-1, 1]. In practice, almost all schema-field pairs land in [0.20, 0.99] because they're all "data field" strings in some sense. That compressed band is the reason raw cosine is a *terrible* user-facing score — a 0.71 looks important until you discover that 0.71 is the median of the entire candidate pool.

The engine ranks in three substeps.

2a. Top-k retrieval

For a given source field, every candidate target field is scored with cosine. The top *k* (by default k = 3) survive. Below-threshold candidates are dropped entirely so the SME's queue isn't padded with noise.

2b. Algorithm-class arbitration

A high cosine score isn't enough on its own. Each surviving candidate is also evaluated against five algorithm classes — the same five the Semantic Match API post lists:

Each candidate is tagged with the *highest-precedence* class that fires. Two candidates with the same cosine but different classes do not rank equally — exact_match outranks synonym_match outranks the rest. The class label travels with the candidate into the decision record so an auditor can reconstruct *why* the engine surfaced what it did.

2c. Calibrated mapping to [0, 100]

The raw cosine is then mapped to an integer in [0, 100] through a per-class calibration curve rather than a single global scale. The curves are isotonic regressions fit on Magellan's labeled corpus — they monotonically map raw cosine to the empirical probability that the candidate matches the steward-validated gold label, then multiply by 100 and round.

Two practical effects:

• A 0.94 cosine in the exact_match class is calibrated near 99. The same 0.94 cosine in the context_hierarchy class might calibrate to 78, because that class is empirically noisier.
• The number an SME sees is never raw cosine. It is a calibrated integer the SME can compare across classes without having to remember which class is more reliable.

The output of Stage 2 is the ranked top‑k, each carrying *(class, calibrated_score)*.

Stage 3 — Confidence Calibration

This is the stage most often misunderstood. The integer in the decision record's confidence field is not the same thing as the engine's accuracy. They answer different questions, on different units of analysis.

The two scores you must not conflate

• Per-pair confidence (the integer 0–100 in the decision record) is the engine's calibrated estimate, for one specific candidate, that an SME would accept this proposal as the correct mapping.
• Engine-level accuracy is a population statistic over a held-out evaluation set. Magellan's published numbers are 87% top‑1 accuracy on a mixed-industry corpus and 92% on a healthcare-specific 500-pair set drawn from FHIR R4 and X12 834 / 835 / 837[^bench].

A candidate with a calibrated confidence of 95 does not mean the engine is right ninety-five times in a hundred at the population level. It is a candidate the engine believes — based on the calibration curve fitted to historical SME decisions — has a high probability of being accepted by a steward. Engine-level accuracy answers a separate question: across the whole evaluation set, how often did the engine's top‑1 candidate match the gold pair?

Calibration as a discipline, not a number

Calibration in this engine means three concrete things:

1. Isotonic, not Platt. The mapping from raw cosine to confidence is monotone but not parametric, so it can fit the actual bend in the data without forcing a sigmoid that doesn't belong.

2. Per-class curves. Each algorithm class has its own curve, refit when the labeled corpus grows materially.

3. Versioned curves. The active calibration set is part of the model_version tag (semantic-match-v2.4.1 includes the curve revision). A decision record from 2024 can be replayed against the 2024 curves; a decision record from today is replayed against today's. Versioning is what makes the historical confidence values actually mean something.

Confidence routing

The calibrated integer drives queue routing — not commit decisions. The engine does not commit anything regardless of confidence. The thresholds Magellan ships by default look like this:

The dual-SME rule on low scores is not a UI nicety. It is a control surface for the hash-chained ledger: a low-confidence proposal that becomes a decision will carry two signatures in the approved_by field, and that fact is visible to anyone replaying the chain later.

What Lands in the Decision Record

Tying it back to the predictive-mapping pipeline, the artifacts each stage contributes are:

Every field in that table is reproducible from the inputs plus the pinned versions. That is what "deterministic by construction" means in the previous post — and Stage 1 through Stage 3 are where the determinism is actually made.

Failure Modes the Engine Surfaces (Not Hides)

A short, non-exhaustive list — these are written into the API response, not buried in logs:

• Embedding model unreachable. No partial fallback to a different model. The request fails closed and the engine refuses to emit a confidence score it cannot calibrate.
• All candidates below floor threshold. The response returns an empty candidates array with a reason of no_candidates_above_floor, not an apologetic guess.
• Class arbitration tie that calibration cannot break. Both candidates are returned at the same confidence, with a tie flag, and the SME chooses.
• Calibration curve missing for the active model version. The engine refuses to score and emits a calibration_unavailable error rather than fall back to raw cosine.

These are the same "blocked, not silently recovered" semantics described in the predictive-mapping post. Stage 1–3 enforce them at the engine boundary.

What This Buys You

• An auditor can recompute, years later, the exact ranked list any historical SME saw — given the pinned schemas, the pinned model version, and the calibration set tagged in that version.
• A steward sees an integer that means the same thing across algorithm classes, because the calibration is per-class.
• A dual-SME control fires on low confidence without anyone having to remember to invoke it.
• The engine is incapable of silently committing a mapping. There is no "auto-accept above 99" mode and there will not be one.

The predictive-mapping post said the engine proposes and the SME decides. This post is the receipt: every step from raw schema string to calibrated integer is named, versioned, and replayable, so the SME's decision is the *only* unconstrained step in the chain.

^bench]: Magellan Semantic Match Engine v1.0.0 internal benchmark, google/gemini-2.5-flash, temperature 0.1. Accuracy is defined as top-1 algorithm-class agreement with steward-validated gold pairs on a 500-pair healthcare set drawn from FHIR R4 and X12 834 / 835 / 837. Full methodology in the [ETL Best Practices benchmark post.