collibra-governance-automation
Collibra Governance Automation
Automates the journey from technical metadata coming from multiple sources to a common governance model that can be reviewed, analyzed and synchronized with Collibra in a controlled way.
Problem it solves
PostgreSQL, ODCS, dbt and OpenLineage describe different parts of the same ecosystem. The challenge is not simply collecting that metadata, but giving it a common identity, preserving where it came from and understanding how it relates before sending it to a governance platform.
How I approached it
The project first builds a neutral governance model that is independent of Collibra. That model can then be used to check rules, analyze the impact of changes and prepare reviewable actions. Collibra remains at the end of the flow as the synchronization target rather than the source of truth for the model.
See how the project works
How the project works
This project automates two different workflows that share principles of governance, reviewability and determinism, but not the same internal model. The first starts from PostgreSQL, builds a GovernanceModel and controls which changes can be prepared and synchronized with Collibra. The second composes ODCS, dbt and OpenLineage into a GovernanceGraph to preserve provenance, build lineage and analyze which assets may be affected by a change.
Keeping both workflows separate is an important part of the current architecture: PostgreSQL does not feed the impact graph, and the multi-source graph does not feed Collibra synchronization. The portfolio must show that separation exactly as it exists in the code rather than turning it into a single pipeline that does not exist today.
POSTGRESQL → GOVERNANCEMODEL → COLLIBRA
ODCS + DBT + OPENLINEAGE → GOVERNANCEGRAPH → IMPACT
Two workflows, two models
GovernanceModel and GovernanceGraph are two different vendor-neutral cores. GovernanceModel represents physical metadata discovered from PostgreSQL and is the basis for the policy and Collibra synchronization path. GovernanceGraph represents nodes, relationships and provenance coming from ODCS, dbt and OpenLineage and is the basis for impact analysis.
There is currently no GovernanceModel → GovernanceGraph conversion. Keeping that boundary visible avoids attributing an end-to-end integration to the project that is not implemented yet and makes it possible to explain precisely what problem each workflow solves.
ROOT
Control and synchronization with Collibra
Context, lineage and impact
WORKFLOW A
Control and synchronization with Collibra
This workflow answers an operational question: what physical metadata exists in PostgreSQL, whether it satisfies a small set of governance checks and which changes would need to be prepared for the managed Collibra state to represent it without applying writes by default.
RUNNING EXAMPLE · commerce.orders
PostgreSQL discovery
The workflow starts by observing PostgreSQL catalogs directly. The scanner works inside a REPEATABLE READ, READ ONLY transaction to obtain a consistent view of the structure without modifying the database.
WHAT GOES IN
A PostgreSQL connection and the catalogs of the configured database.
WHAT PYTHON DOES
Python executes five catalog queries and reconstructs databases, user schemas, tables, columns, types, nullability, primary keys, foreign keys, comments and ownership. Views, indexes and CHECK constraints are not part of the discovery currently implemented.
RESULT
A GovernanceModel with a Data Source → Database → Schema → Table → Column hierarchy and structural relationships derived from foreign keys.
CODE EVIDENCE
PostgresMetadataScanner.scan · build_governance_model
commerce
orders
order_id
customer_id
…
orders.customer_idcustomers.customer_id
In the commerce demo, Python observes the orders table and the orders.customer_id → customers.customer_id foreign key. The relationship is preserved inside the GovernanceModel as structural metadata.
Model and stable identities
Discovered metadata is not kept as loose catalog rows. Python turns it into domain objects with stable identities so the same asset can be recognized reproducibly across executions.
WHAT GOES IN
The normalized rows obtained by the PostgreSQL scanner.
WHAT PYTHON DOES
Python builds canonical IDs for each model level, validates reference integrity and orders collections deterministically. The PostgreSQL host and port are not part of table and column identity, so moving the connection does not by itself change the logical identity of an asset.
RESULT
A stable, reviewable vendor-neutral GovernanceModel that can be used without depending on Collibra yet.
CODE EVIDENCE
GovernanceModel · make_table_id · make_column_id · make_relationship_id
RAW CATALOG ROWS
CANONICAL IDS
GOVERNANCEMODEL
The example table is identified as tbl:governance-demo/governance_demo/commerce/orders and the customer_id column as col:governance-demo/governance_demo/commerce/orders/customer_id.
Governance checks
Before preparing changes for Collibra, the project can apply a small and explicit set of rules to the GovernanceModel. This is not a general-purpose policy engine: the project currently defines three check types.
WHAT GOES IN
The GovernanceModel and the rules declared in the Governance-as-Code configuration.
WHAT PYTHON DOES
Python evaluates whether selected assets have an owner, a description and relationships when require_owner, require_description or require_relationship demand them. Violations are sorted and reported; an error-severity violation can block check or plan generation.
RESULT
A violation report that separates governance issues from the synchronization phase.
CODE EVIDENCE
evaluate_policies · NormalizedPolicySet
GovernanceModel
require_owner · requires ownership
require_description · requires a description
require_relationship · requires at least one relationship
report
Desired state and reviewable plan
Collibra appears at the end of the model, not at the beginning. Python translates the neutral GovernanceModel into a Collibra-specific representation and compares that desired state with the managed remote state before deciding which operations would be necessary.
WHAT GOES IN
GovernanceModel + Collibra mapping + managed remote state.
WHAT PYTHON DOES
Python transforms databases, schemas, tables, columns and FK relationships into CollibraAssetSpec and CollibraRelationshipSpec, including configured domain, type and attribute references. It then calculates a SyncPlan with CREATE, UPDATE, UNCHANGED and REMOTE_ONLY, and can save the actions in a .gplan together with identities for configuration, snapshot, policies, mapping, target context and remote state.
RESULT
A reviewable plan combining an ordered list of operations with an integrity contract. REMOTE_ONLY is reported but does not become a delete operation.
The .gplan is more than a description: it preserves the actions and identities required to detect whether the context has changed before applying.
CODE EVIDENCE
map_to_desired_state · build_sync_plan · build_saved_plan · SavedGovernancePlan
- 01
GovernanceModel
- 02
CollibraDesiredState
- 03
SyncPlan
- 04
.gplan
CREATE
UPDATE
UNCHANGED
REMOTE_ONLY
The .gplan is more than a description: it preserves the actions and identities required to detect whether the context has changed before applying.
Safe synchronization
Having a plan does not authorize a write. The workflow keeps dry-run as the default behaviour, requires explicit opt-in to apply and adds further checks before allowing operations against a live adapter.
WHAT GOES IN
A reviewed SyncPlan or .gplan, the current configuration and either a mock or live Collibra adapter.
WHAT PYTHON DOES
Python executes dry-run with no writes by default. Applying requires --apply and, when the adapter is live, an additional --confirm-live confirmation. Before writing, the .gplan flow checks that the relevant identities still match; if the plan is stale, it is blocked before the first write. During apply, write errors stop the sequence.
RESULT
CREATE and UPDATE can be executed when explicitly requested. The project does not implement DELETE, automatic deletion, rollback or a proven retry engine.
LiveCollibraAdapter is contract-tested against simulated HTTP. The repository does not demonstrate validation against a real commercial Collibra tenant.
CODE EVIDENCE
execute_sync_plan · plans/stale.py · MockCollibraAdapter · LiveCollibraAdapter
.gplan
DRY-RUN · DEFAULT
--apply ?
NO → 0 WRITES
YES
LIVE ADAPTER ?
NO → continue
YES → --confirm-live
STALE ?
YES → BLOCK · 0 WRITES
NO → CREATE / UPDATE
RESULT
In the local lifecycle using MockCollibraAdapter, an empty remote state produces creates and a second synchronization against the new state does not create the same assets again.
LiveCollibraAdapter is contract-tested against simulated HTTP. The repository does not demonstrate validation against a real commercial Collibra tenant.
Collibra does not have a single fixture that joins PostgreSQL, ODCS, dbt and OpenLineage in one end-to-end execution. The PostgreSQL → Collibra workflow and the ODCS/dbt/OpenLineage → impact workflow are shown with different real examples so the project is not credited with an integration that does not exist yet.
WORKFLOW B
Context, lineage and impact
This workflow answers a different question: if an asset changes, which dependencies, contracts and consumers fall within its blast radius. To answer it, Python composes declared, transformational and observed metadata coming from ODCS, dbt and OpenLineage.
RUNNING EXAMPLE · orders → downstream
Context sources
The three sources do not contribute the same information. ODCS describes contracts and declared context, dbt describes models and transformation dependencies, and OpenLineage contributes execution observations and lineage. The value comes from preserving those differences rather than collapsing them into one format without provenance.
WHAT GOES IN
ODCS v3.1.0 documents, a subset of dbt Manifest v12 and OpenLineage core 2-0-2 events.
WHAT PYTHON DOES
Python validates and maps each source specifically. ODCS creates contract, dataset and column nodes with governs and contains relationships. dbt turns models, sources, columns and parent_map into physical structure and depends_on relationships. OpenLineage composes RunEvent, JobEvent and DatasetEvent, input and output datasets and, when present, columnLineage.
RESULT
Three sets of nodes, relationships and provenance ready to be composed into a GovernanceGraph.
ODCS can express quality, SLA and servers in the standard, but this project does not currently map them into the GovernanceGraph.
CODE EVIDENCE
load_odcs_graph · load_dbt_graph · load_openlineage_graph
ODCS · contract and declared context
contract → governs → dataset → contains → column
dbt · models and declared dependencies
source/model → depends_on
OpenLineage · observed execution and column lineage
input → run/job → output
+ columnLineage
ODCS can express quality, SLA and servers in the standard, but this project does not currently map them into the GovernanceGraph.
Identity and provenance
To compose multiple sources, the project needs to decide when two observations may represent the same asset and preserve where each fact came from. That merge is not based on silently choosing a winning source.
WHAT GOES IN
Nodes and relationships produced by the ODCS, dbt and OpenLineage mappers.
WHAT PYTHON DOES
Python builds GraphNodeIdentity from namespace, kind, logical_id and, where applicable, parent. Observations with the same identity and the same material payload can merge their ProvenanceRecord. If two observations with the same identity contain incompatible material attributes, composition fails instead of choosing a source arbitrarily.
RESULT
Assets with canonical identity and declared, observed or derived provenance, ready to participate in the same graph when their identities are compatible.
dbt and OpenLineage can converge on the same physical identity when namespace, database, schema and table match.
ODCS only converges automatically with other assets when its logical IDs have been aligned. PostgreSQL does not participate in this identity scheme because it belongs to the GovernanceModel workflow.
The project preserves provenance, but it does not yet implement an authority or conflict-resolution engine. A material conflict is rejected.
CODE EVIDENCE
GraphNodeIdentity · ProvenanceRecord · GovernanceGraph.from_parts
dbt observation
OpenLineage observation
SAME IDENTITY
ONE NODE
CONFLICT · STOP
dbt and OpenLineage can converge on the same physical identity when namespace, database, schema and table match.
ODCS only converges automatically with other assets when its logical IDs have been aligned. PostgreSQL does not participate in this identity scheme because it belongs to the GovernanceModel workflow.
The project preserves provenance, but it does not yet implement an authority or conflict-resolution engine. A material conflict is rejected.
GovernanceGraph
Composition produces an in-memory governance graph built from immutable dataclasses. It does not use a graph database or NetworkX: the domain itself defines how nodes and relationships are normalized, validated and ordered.
WHAT GOES IN
Nodes, relationships and provenance information already normalized by each integration.
WHAT PYTHON DOES
Python merges compatible observations, checks that parents exist, rejects dangling edges, canonicalizes attributes and orders nodes and relationships. The graph uses contains for hierarchy, depends_on for dependencies and governs to associate contracts with the assets they govern.
RESULT
A deterministic GovernanceGraph with its own content identity and enough structure to build lineage and traverse dependencies.
depends_on is stored as derived → dependency. To calculate downstream impact, traversal uses that relationship in reverse.
CODE EVIDENCE
GovernanceGraph · GraphNode · GraphEdge · canonical_json_bytes
GovernanceGraph
contains
depends_on
governs
depends_on is stored as derived → dependency. To calculate downstream impact, traversal uses that relationship in reverse.
Lineage and contracts
Inside the graph, dependencies may come from different sources and preserve different levels of detail. dbt contributes declared dependencies between models and sources; OpenLineage contributes dependencies observed at runtime and can materialize column-to-column lineage; ODCS contributes contracts associated with datasets.
WHAT GOES IN
dbt parent_map, OpenLineage inputs/outputs and columnLineage, and ODCS governs relationships.
WHAT PYTHON DOES
Python materializes depends_on relationships with derived → dependency direction. In dbt those relationships are at table or transformation level. In OpenLineage they may also exist between columns through ColumnLineageAssertion. ODCS contracts remain contract nodes connected through governs and are included as context for impact analysis.
RESULT
A graph capable of representing dataset-level and column-level dependencies together with the available contract context.
dbt does not generate column-level lineage in this project. That capability is demonstrated through OpenLineage.
ODCS contracts are informational context inside impact analysis; they do not currently block changes or replace the require_* rules from the PostgreSQL workflow.
CODE EVIDENCE
materialize_column_lineage_edges · ColumnLineageAssertion · GraphEdge
- DATASETdownstreamorders
- COLUMNoutput columninput column
- CONTRACTcontractdataset
dbt does not generate column-level lineage in this project. That capability is demonstrated through OpenLineage.
ODCS contracts are informational context inside impact analysis; they do not currently block changes or replace the require_* rules from the PostgreSQL workflow.
Tests demonstrate that a physical identity built from OpenLineage can be compatible with the equivalent physical dbt asset and that column lineage is materialized as output column → depends_on → input column.
Impact analysis
When an asset is declared as changed, the project uses the GovernanceGraph to calculate which consumers sit downstream. The analysis is completely read-only and does not execute policies or perform remote writes.
WHAT GOES IN
GovernanceGraph + one or more changed roots defined through governance-impact-changes/v1.
WHAT PYTHON DOES
Python validates the roots, builds an effective adjacency, traverses depends_on in reverse and runs a multi-source BFS by distance. It distinguishes direct and transitive impact, avoids cycling back into roots, selects a canonical shortest path when equivalent routes exist and collects relationships, contracts and governance context inside the affected closure.
RESULT
A deterministic governance-impact-result/v1 containing affected assets, distances, paths and relevant context. The governance impact command reports an impacted state when affected consumers are found and performs no remote mutation.
The main CLI example is dataset-level. Column-level lineage is demonstrated by specific OpenLineage tests and must be shown as complementary technical evidence rather than as if it belonged to the same CLI fixture.
CODE EVIDENCE
analyze_downstream_impact · GovernanceImpactResult · _path_sort_key
orders
↓ reverse traversal over depends_on
downstream
TRANSITIVE · distance ≥2
In the real fixture used by governance impact, orders is the changed root and downstream appears as an affected consumer. That traversal is proven by both the CLI and the read-only GitHub Action.
The main CLI example is dataset-level. Column-level lineage is demonstrated by specific OpenLineage tests and must be shown as complementary technical evidence rather than as if it belonged to the same CLI fixture.
What Python actually automates
Python does not act here as a single chain of scripts. In the operational workflow it discovers PostgreSQL metadata, builds stable identities, evaluates checks, translates the model to Collibra and prepares reviewable plans with dry-run and stale-plan protection. In the impact workflow it ingests ODCS, dbt and OpenLineage, composes identities and provenance, materializes lineage and traverses the GovernanceGraph to calculate downstream impact deterministically.
- DISCOVERPostgresMetadataScanner.scan
- IDENTIFYGovernanceModel
- CHECKevaluate_policies
- MAPmap_to_desired_state
- PLANbuild_saved_plan
- PROTECTexecute_sync_plan
- COMPOSEGovernanceGraph.from_parts
- TRACE LINEAGEmaterialize_column_lineage_edges
- CALCULATE IMPACTanalyze_downstream_impact
- EVIDENCEContentIdentity
Identities, ordering, artifacts and impact paths are canonicalized so the same input produces a reviewable and reproducible representation.
Complete architecture
COLLIBRA GOVERNANCE AUTOMATION
TWO MODELS · NO CURRENT DIRECT CONVERSION
POSTGRESQL
GOVERNANCEMODEL
- CHECKSCOLLIBRA DESIRED STATE
SYNCPLAN / .GPLAN
DRY-RUN
APPLY
COLLIBRA
- ODCSDBTOPENLINEAGE
GOVERNANCEGRAPH
- PROVENANCELINEAGECONTRACTSIMPACT ANALYSIS
IMPACT RESULT
PR / REVIEW
Maturity and evolution
The polygon reflects documented functional state. The outer ring represents the full profile.
Most consolidated
Read-only PostgreSQL discovery, the GovernanceModel, ODCS/dbt/OpenLineage composition, the GovernanceGraph, OpenLineage-based column-level lineage, deterministic impact analysis and safe planning toward Collibra are capabilities demonstrated by code and tests.
Still evolving
The live Collibra integration is contract-tested but has not been validated against a real commercial tenant. PostgreSQL and the GovernanceGraph remain separate workflows, and the project does not yet implement an authority or source-conflict resolution engine.
Next areas
The repository documents Collibra integration hardening, authority, conflict and drift management, and greater provider extensibility as pending areas. They are presented as future evolution, not as implemented capabilities or as a closed versioned roadmap.
Maturity and evolution
- CURRENT · v1.2.0
- NEXT AREASCollibra hardeningFUTURE
- NEXT AREASAuthority / conflicts / driftFUTURE
- NEXT AREASProvider extensibilityFUTURE
What you can review on GitHub
The repository lets you inspect PostgreSQL discovery and Collibra synchronization separately from multi-source GovernanceGraph composition, lineage, impact analysis and plan safety controls.
Discovery and models
PostgreSQL scanner, GovernanceModel, GovernanceGraph and canonical identities.
Sources, provenance and lineage
ODCS, dbt and OpenLineage ingestion, ProvenanceRecord and column-level lineage materialization.
Impact, plans and safety
Downstream BFS, impact artifacts, .gplan, stale-plan checks, dry-run and no-delete behaviour.
Integration and tests
Collibra mapping, contract-tested mock/live adapters, local lifecycle and read-only GitHub Action.