Integration
NiFi, Azure Data Factory & AWS Glue
Emit NiFi flow templates, Azure Data Factory linked-service JSON, and AWS Glue job scripts — each artifact embeds the source mapping ULID and is regenerable byte-for-byte from the ledger, so a change in the runtime that didn't originate in Magellan is immediately detectable.
The NiFi adapter emits flow template XML for processor groups, processors, controller services, and parameter contexts. Magellan mappings translate to ConvertRecord, UpdateRecord, JoltTransformRecord, and EvaluateJsonPath chains depending on the source/target shapes; transformation rules from Visual Rules for Targets translate to ScriptedTransformRecord with the rule body emitted inline.
The Azure Data Factory adapter emits ARM templates for linked services, datasets, data flows, and pipelines. Mapping data flows carry per-column derivations directly from the Magellan mapping; integration runtimes are referenced by name from the deployment configuration so no ADF-specific credentials live in the ledger.
The AWS Glue adapter emits Glue Studio visual job JSON and the equivalent PySpark or Spark SQL scripts for the same job. Both artifacts are produced from the same mapping ULID and are byte-deterministic, so a Glue job that was edited in the Glue console will diff cleanly against a regenerated artifact — the diff is the change, not formatting noise.
All three adapters embed the same provenance triple in the artifact: source mapping ULID, `ApproverReleaseSigned` ULID, and SHA-256 chain head at export time. Where the runtime supports a metadata field (NiFi `Process Group` description, ADF `annotations`, Glue `Tags`) the triple is also surfaced for in-runtime inspection, not only in the source artifact.
Each export writes `ArtifactExported` carrying the runtime identifier (`NiFi@1.x`, `AzureDataFactory@v2`, `AWSGlue@4.0`), the artifact format, and the SHA-256 of the emitted bytes. Subsequent re-imports — a NiFi flow template ingested back through a Custom Import adapter — write `ArtifactImported` with the same triple, so a round-trip drift is detectable as a hash mismatch.
The strongest property the cloud-runtime adapters offer is regenerability. A production NiFi flow, an ADF pipeline, or a Glue job can be rebuilt from scratch from the ledger alone, given the mapping ULID. This is what closes the FedRAMP CM-2 (baseline configuration) and AC-3 (access enforcement) story for cloud-native data pipelines: the running configuration is the signed configuration, and the proof is a hash equality check.
Audit ledger entries written by this capability
Each action in this capability appends one of the following entry types to the ULID‑keyed, SHA‑256 hash‑chained audit ledger. A reviewer can replay the chain to reconstruct exactly what happened, when, and under which role.
ArtifactExportedArtifactImportedApproverReleaseSignedSee nifi, azure data factory & aws glue demonstrated against your data
Pick a sample dataset or bring your own. We will walk the ledger from ingest through Approver release in a single working session.
Request a demoRelated across the platform
Where this capability hands off — upstream adapters, downstream governance, and the underlying audit primitives.