April 20, 20269 min read

Why the Magellan Audit Ledger Lives in the Same Container as magellan-data

> Where to read this in context: the integrity model below is the same one referenced from /audit (healthcare framing) and /audit/regulated-industries (SOX, GxP, GDPR, FedRAMP, ISO 27001 framing). The container decision does not change with the framework.

The Question Worth Answering Honestly

When platform engineers see "audit ledger" and a regulator's name in the same sentence — HIPAA §164.312(b), SOX §404 ITGCs, EU GMP Annex 11, GDPR Art. 30, FedRAMP AU family, ISO 27001 A.8.15 — the reflex is usually to draw a separate box on the architecture diagram. A dedicated magellan-audit-svc. Its own image, its own deploy, its own pager rotation.

We considered that. We did not ship it. The audit ledger writer lives inside magellan-data — the same container that handles schema imports, mappings, and approval workflows. This post explains why, what we did instead to satisfy the controls a separate service would have addressed, and the three specific conditions under which we would split it out later.

If you are evaluating Magellan for a regulated environment of any kind, this is the architectural decision your security reviewer will ask about first.

The Co-Location Decision

Three things drove the call to keep the ledger writer in magellan-data. None of them depend on which regulator is reviewing.

Hash-chain integrity is a property of the data, not the process. A reviewer verifies H(n) == sha256(H(n-1) || canonical(entry_n)) against the stored rows. Whether those bytes were assembled in magellan-data or in a separate magellan-audit-svc is irrelevant to the check. A separate process does not make the chain more verifiable; it just adds a network hop that can fail.

A missing entry is worse than a slow one. Every mapping decision must produce a ledger entry. If that write has to cross a service boundary synchronously, you have added a new failure mode — the decision succeeded, the ledger write was lost in flight — that is exactly the failure an audit ledger is supposed to prevent. Co-location removes that class of bug. The write happens in the same transaction as the decision it records, or the decision rolls back. NIST SP 800-53 AU-5 ("Response to Audit Logging Process Failures") is satisfied by the transaction boundary, not by the deployment boundary.

RBAC is enforced at the database, not at the service boundary. Magellan's seven roles — Reader, Contributor, Approver, Auditor, Compliance Officer, Security Officer, Administrator — map to Postgres roles and row-level security policies. A separate container does not strengthen Approver-vs-Auditor separation; the database already does that. Splitting the process would be ceremony, not control. SOX segregation-of-duties, FedRAMP AC-5, and HIPAA §164.308(a)(3) all read the same database privileges.

The summary, in one line: a separate audit container would add cost without adding assurance. The properties an assessor verifies are properties of the rows, the privileges, and the chain — none of which become stronger by living behind a different process boundary.

What "Co-Located" Does Not Mean

Co-located does not mean entangled. The single most common mistake is to think "same container" implies "same code path with the same privileges." That would be the wrong design — and it is the design that would justifiably alarm a security reviewer.

We made four structural splits inside magellan-data that give the ledger the isolation a separate service would have given it, without paying for the network hop. None of these require a new container. All of them are visible to an assessor on day one.

Split 1 — A separate Postgres role for ledger writes

magellan-data holds connections under two Postgres roles, not one.

magellan_app — the application role. Has SELECT, INSERT, UPDATE, DELETE on mapping tables. Has no privileges at all on audit_ledger. Cannot write a ledger entry even by mistake. The privilege does not exist.
magellan_ledger_writer — the audit module role. Has INSERT only on audit_ledger. No UPDATE. No DELETE. No access to mapping tables.

A bug in the application's mapping code cannot rewrite history because the connection it holds physically does not have the privilege. That is the property an assessor is looking for under SOX SoD, FedRAMP AC-6 (least privilege), and HIPAA §164.312(a)(1) — and it is enforced by the database, not by code.

Split 2 — Append-only is enforced twice

Privileges can be granted by an operator with SUPERUSER. So we don't rely on privileges alone. audit_ledger carries a BEFORE UPDATE OR DELETE OR TRUNCATE trigger that raises insufficient_privilege unconditionally. Even an operator who accidentally re-grants UPDATE cannot mutate the table without first dropping the trigger — and dropping the trigger is itself a DDL event that will land in your DBA audit log.

Belt and suspenders. The assessor question is "what would have to be true for the ledger to be silently rewritten?" The answer is: a SUPERUSER would have to (a) re-grant privileges, (b) drop the trigger, and (c) suppress the resulting WORM divergence alert (Split 4). Three deliberate, logged actions — not one careless commit. This is the answer to NIST AU-9(2)/(3) and ISO 27001 A.8.15.

Split 3 — A separate read pool for auditor traffic

Auditor and Compliance Officer dashboards run heavy scans. The hot mapping path runs short, high-concurrency queries. Letting them share a connection pool means an aggressive audit query can starve user-facing latency.

The audit module exposes a magellan_auditor role with SELECT only, served from its own connection pool against a Postgres read replica. Same container, separate pool, separate replica. An auditor exporting a quarter of ledger entries cannot affect a Contributor saving a draft.

This is the kind of split that is trivial when the code lives together and surprisingly fiddly when it lives in a separate service — because then "separate pool" turns into "separate deployment with its own scaling story."

Split 4 — A WORM secondary sink for the chain head

This is the structural split that does the most work for the smallest diagram change. The application can never overwrite it.

Every N entries (default 500), or every 60 seconds — whichever comes first — the audit module flushes the current chain head to S3 with Object Lock in compliance mode. Default retention is set to seven years to cover both HIPAA §164.316(b)(2)(i) (six-year minimum) and SOX §103 (seven-year external-audit retention); GxP, FedRAMP, and ISO 27001 deployments override per their own policies. The application has PutObject permission only; it has no permission to delete or overwrite. The bytes, once written, are immutable for the retention period.

If the database's computed chain head ever disagrees with the most recent WORM record covering the same partition, the WORM head wins and the divergence is treated as a P1 incident. That is the answer to "what if a database administrator silently rewrites history?" — they cannot, because the second sink is outside the database's control plane and the disagreement is detectable in a single comparison. NIST SP 800-53 AU-9 ("Protection of Audit Information") is structurally satisfied; it is not a process commitment.

A separate magellan-audit-svc would not give you this. WORM-as-second-sink is the actual integrity boundary. The container layout is downstream of that decision.

When We Would Split It Out

Defaults are not laws. There are three specific conditions under which we would move the ledger writer into its own container — or, more likely, into a sidecar process on the same pod. Each is a real boundary, not a vague "for separation of concerns."

1. magellan-data gains a less-trusted code path. If we ever add customer-written transformation plugins, or any code path that runs untrusted bytes inside magellan-data, the ledger writer must not share an address space with that code. A memory corruption or RCE in a plugin host would otherwise reach the magellan_ledger_writer connection by virtue of living in the same process. At that point we move the ledger writer to a sidecar — same pod, separate process, separate filesystem mount for its DSN — and the plugin host talks to it over a local Unix socket.

2. A FedRAMP boundary requires a distinct authorization decision. Not "FedRAMP makes us nervous" — a specific control statement in the System Security Plan that draws the audit subsystem as its own boundary with its own authorization. That is unusual but not unheard of for FedRAMP High deployments. Until that statement exists, splitting the container does not change a control conclusion an assessor will reach.

3. Multi-tenant SaaS with per-tenant KMS isolation for entry signing. If individual tenants demand that ledger entries be signed with a tenant-controlled KMS key — common in GxP CDMOs serving multiple sponsors, in financial services with cross-tenant SOX scope, and in any GDPR deployment where joint-controllers want cryptographic separation — the signer becomes a per-tenant component with its own key access. That is a ledger-signer concern; storage can still be co-located. The signer becomes a tiny service whose only job is to call KMS:Sign on entry hashes for the tenant whose request initiated the write.

None of these apply at v1. They are year-two concerns. They are listed here so the answer to "why don't you split it?" is "we will, under these specific conditions" — not "we never will."

What an Assessor Sees

A pre-audit walkthrough of the production deployment shows, in this order:

1. magellan-data container holds two distinct Postgres connections under two distinct roles. The connection strings are sourced from two distinct env vars. The pools are sized differently and have different statement timeouts.

2. audit_ledger is partitioned by month. REVOKE statements are visible in the migration history. The append-only trigger is present and executable. INSERT succeeds under magellan_ledger_writer; UPDATE fails with insufficient_privilege under any role.

3. The most recent S3 Object Lock record for the current partition is visible, signed with the configured KMS key, and its head_hash field matches the database's most recent entry_hash. The bucket lifecycle policy shows compliance-mode lock with the configured retention.

4. A read against audit_ledger from the auditor dashboard hits the read replica via a different connection pool. The hot mapping path remains responsive while the read runs.

That walkthrough is the assurance story. It is the same walkthrough whether the assessor's cover page says HIPAA, SOX, GxP, GDPR, FedRAMP, or ISO 27001 — the controls these frameworks name are the same four observations in different language.

The container count is incidental. We chose one container because adding a second would not change any of those four observations — and would have introduced a write path that could quietly fail.

The Trap to Avoid

The mistake is not "co-locate." The mistake is to co-locate without the four splits and to claim defensibility on the basis of "we have an audit ledger." A single Postgres role that writes to mapping tables and audit_ledger is not an audit ledger; it is a logging table the application can rewrite at will. The privileges, the trigger, the WORM sink, and the role-at-time-of-action stamp on each entry are what make the ledger defensible. The container layout sits downstream of all four.

If you are designing your own audit subsystem and you are debating "separate microservice or same container?", that is not the question. The question is "which structural splits am I committing to?" Once those are clear, the container count answers itself — and for most deployments, the answer will be one.