# Data Governance Engine

*/Software/Data_Governance_Engine*

## Solution Overview

The Data Governance Engine connects to Snowflake and Databricks clusters to classify sensitive schemas, detect undocumented PII or PHI, and apply query-level data masking. It parses raw data streams and outputs sanitized tables paired with cryptographically signed compliance logs. The engine intercepts read requests via its API, dynamically redacting specific columns based on the caller's assigned access tier before the data leaves the warehouse.

Chief Data Officers and compliance engineering teams integrate this primitive to safely expose internal datasets to newly deployed enterprise LLMs and external developer portals. It eliminates the bottleneck of manual data-tagging queues, allowing internal builders to query production databases without risking regulatory breaches under HIPAA, CCPA, or GDPR.

Operating as a Headless SaaS layer, the engine sits between raw storage infrastructure below and autonomous analytical agents or BI platforms above. It ingests metadata and raw queries, passing strictly masked JSON payloads up to the consuming applications. Because regulatory enforcement carries strict financial penalties, the engine queues newly inferred privacy rules for a mandatory sign-off by a human compliance officer before activating the redaction policy in the production routing layer.

## Headless Saas Data Model

**Entities**:
- Name: MaskingPolicy · Description: Inferred or active rule for redacting data based on caller tier
- Name: Workspace · Description: Tenant boundary containing data sources and privacy policies
- Name: DataSource · Description: Connected Snowflake or Databricks cluster for data discovery
- Name: DataAsset · Description: Discovered schema, table, or column containing potentially sensitive data
- Name: AccessTier · Description: Assigned clearance level for callers reading the governed data
- Name: ComplianceLog · Description: Cryptographically signed audit trail of governed read requests
**Relations**:
- To: Workspace · From: MaskingPolicy · Label: belongs to · Cardinality: one-to-many
- To: DataAsset · From: MaskingPolicy · Label: applies to · Cardinality: one-to-many
- To: AccessTier · From: MaskingPolicy · Label: restricts · Cardinality: one-to-many
- To: Workspace · From: DataSource · Label: belongs to · Cardinality: one-to-many
- To: DataSource · From: DataAsset · Label: located in · Cardinality: one-to-many
- To: Workspace · From: AccessTier · Label: belongs to · Cardinality: one-to-many
- To: MaskingPolicy · From: ComplianceLog · Label: audits · Cardinality: one-to-many
**Tenant Anchor**: Workspace
**Primary Resource**: MaskingPolicy

## Api Definition

**Protocols**:
- REST
- SDK
- MCP
- Webhooks
**Consumed By**:
- [Enterprise Analytics Agent](/Agents/Enterprise_Analytics_Agent)
- [Compliance Audit Agent](/Agents/Compliance_Audit_Agent)
- [Developer Portal Agent](/Agents/Developer_Portal_Agent)
**Integrations**:
- [Snowflake](/Products/Snowflake)
- [Databricks](/Products/Databricks)
**Consumption Model**: An analytical agent mounts the MCP server to invoke execute_governed_query during data processing loops, dynamically receiving tier-masked JSON payloads while the engine generates cryptographic compliance logs.
**Workflow Wrappers**:
- Name: Discover Sensitive Assets · Wraps: Connects the data source, scans schemas, detects PII, and stages inferred masking policies.
- Name: Execute Governed Query · Wraps: Intercepts read requests, applies tier-based masking policies, returns sanitized data, and signs logs.
- Name: Activate Masking Policy · Wraps: Registers officer sign-off and activates the inferred redaction policy across production routing layers.

## Api Function Cascade

**Ai Role**: Operating as stateless software, the engine uses generative models to classify sensitive data and draft masking rules straight-through, relying on deterministic code for live query interception, redaction, and cryptographic logging, with a strict API gate awaiting asynchronous human approval for policy activation.
**Cascade**:
- Kind: Code · Note: Fetches raw schema definitions and data samples from connected Snowflake and Databricks instances. · Step: Ingest Schema Metadata · Verb: ingest · Realizes: Collect Database Metadata · Oversight: none
- Kind: Generative · Note: Scans schemas to detect PII and drafts tier-based masking policies. · Step: Infer Sensitivity Rules · Verb: classify · Realizes: Classify Data Assets · Oversight: none
- Kind: Code · Note: API suspends activation until a mandated compliance officer sign-off is registered via webhook. · Step: Register Masking Policy · Verb: stage · Realizes: Define Governance Policies · Oversight: approves
- Kind: Code · Note: Captures read requests executed by downstream agents via the MCP server. · Step: Intercept Data Request · Verb: intercept · Realizes: Monitor System Access · Oversight: none
- Kind: Code · Note: Applies the approved masking tier rules to sanitize the JSON payload payload dynamically. · Step: Apply Dynamic Redaction · Verb: redact · Realizes: Mask Sensitive Data · Oversight: none
- Kind: Code · Note: Generates and signs cryptographic compliance logs for the Audit Agent. · Step: Sign Audit Log · Verb: sign · Realizes: Generate Audit Logs · Oversight: none
**Optimizes**:
- [PII Detection Recall](/Metrics/PII_Detection_Recall)
- [Policy Enforcement Latency](/Metrics/Policy_Enforcement_Latency)
- [Unauthorized Exposure Rate](/Metrics/Unauthorized_Exposure_Rate)
- [Audit Log Completeness](/Metrics/Audit_Log_Completeness)

## Headless Saas Representative Offer

**Warranty**: Maintains 99.9% API uptime with sub-50ms query interception overhead, backing performance and availability with prorated service credits.
**Price Band**: ~$0.02 to $0.10 per GB of data processed or metered per 10,000 governed queries, depending on compute and masking complexity
**Pricing Kind**: UsageMeter
**Deliverables**:
- REST, SDK, and MCP interface credentials
- Dynamically masked JSON data payloads
- Automated PII detection and staging policies
- Cryptographically signed compliance audit logs
**Delivery Mode**: Instant self-serve provisioning via API keys and MCP server endpoints, enabling consuming agents to authenticate and stream governed data pipelines immediately.
**Business Function**: ProvideService
**Agent Checkout Support**:
- agentic-commerce-protocol
- stored-credential

## Headless Saas Crud Surface

**Auth Model**: Service Account
**Endpoints**:
- GET /masking-policies — list masking policies
- POST /masking-policies — create a new masking policy rule
- GET /masking-policies/{id} — retrieve details of a masking policy
- PATCH /masking-policies/{id} — update a masking policy redaction type or status
- POST /masking-policies/{id}/activate — mark a masking policy as active and enforce redactions
- GET /workspaces — list isolated tenant workspaces
- POST /workspaces — provision a new workspace boundary
- GET /workspaces/{id} — retrieve workspace configuration
- PATCH /workspaces/{id} — update workspace compliance framework settings
- GET /workspaces/{workspaceId}/data-sources — list data sources belonging to a workspace
- POST /workspaces/{workspaceId}/data-sources — register a data source cluster
- GET /data-sources/{id} — retrieve data source connection details
- PATCH /data-sources/{id} — update connection URI or status
- POST /data-sources/{id}/discover — trigger schema discovery for the data source
- GET /data-sources/{dataSourceId}/data-assets — list data assets located in a data source
- POST /data-sources/{dataSourceId}/data-assets — register a discovered data asset
- GET /data-assets/{id} — retrieve data asset path and classification
- PATCH /data-assets/{id} — update data asset classification label
- GET /workspaces/{workspaceId}/access-tiers — list access tiers for a workspace
- POST /workspaces/{workspaceId}/access-tiers — create a clearance level access tier
- GET /access-tiers/{id} — retrieve access tier clearance level
- PATCH /access-tiers/{id} — update access tier clearance level
- GET /masking-policies/{maskingPolicyId}/compliance-logs — list compliance logs auditing a masking policy
- POST /masking-policies/{maskingPolicyId}/compliance-logs — ingest a cryptographically signed read query event
- GET /compliance-logs/{id} — retrieve a compliance log cryptographic signature
**Multitenancy**: Row-level isolation
**Webhook Events**:
- data_source.connected
- data_asset.discovered
- masking_policy.activated
- compliance_log.signed

## Headless Saas Erd

```mermaid
erDiagram
    MaskingPolicy {
        UUID id PK
        UUID workspaceId FK
        UUID dataAssetId FK
        UUID accessTierId FK
        VARCHAR redactionType
        VARCHAR status
    }
    Workspace {
        UUID id PK "tenant key"
        VARCHAR name
        VARCHAR complianceFramework
    }
    DataSource {
        UUID id PK
        UUID workspaceId FK
        VARCHAR provider
        VARCHAR connectionUri
        VARCHAR status
    }
    DataAsset {
        UUID id PK
        UUID dataSourceId FK
        VARCHAR assetType
        VARCHAR path
        VARCHAR detectedClassification
    }
    AccessTier {
        UUID id PK
        UUID workspaceId FK
        VARCHAR name
        DECIMAL clearanceLevel
    }
    ComplianceLog {
        UUID id PK
        UUID maskingPolicyId FK
        VARCHAR queryHash
        VARCHAR cryptographicSignature
        TIMESTAMP executedAt
    }
    MaskingPolicy ||--o{ Workspace : "belongs to"
    MaskingPolicy ||--o{ DataAsset : "applies to"
    MaskingPolicy ||--o{ AccessTier : "restricts"
    DataSource ||--o{ Workspace : "belongs to"
    DataAsset ||--o{ DataSource : "located in"
    AccessTier ||--o{ Workspace : "belongs to"
    ComplianceLog ||--o{ MaskingPolicy : "audits"
```

## Neighborhood

### Related (exposed by)

- [Enforce Data Standard](/Action/Enforce_Data_Standard) — exposed by · Action
- [Enforce Data Policy](/Action/Enforce_Data_Policy) — exposed by · Action
- [Evaluate Data Transfer Policy](/Action/Evaluate_Data_Transfer_Policy) — exposed by · Action

### Composed into

- [Query Execution Engine](/Agents/Query_Execution_Engine) — composes · Agents

### Optimizes

- [Policy Enforcement Latency](/Metrics/Policy_Enforcement_Latency) — optimizes · Metrics
- [Unauthorized Exposure Rate](/Metrics/Unauthorized_Exposure_Rate) — optimizes · Metrics
- [Audit Log Completeness](/Metrics/Audit_Log_Completeness) — optimizes · Metrics
- [PII Detection Recall](/Metrics/PII_Detection_Recall) — optimizes · Metrics

### Who consumes this

- [Compliance Audit Agent](/Agents/Compliance_Audit_Agent) — consumed by · Agents
- [Developer Portal Agent](/Agents/Developer_Portal_Agent) — consumed by · Agents
- [Enterprise Analytics Agent](/Agents/Enterprise_Analytics_Agent) — consumed by · Agents

### What it uses

- [Snowflake](/Software/Snowflake) — uses · Software
- [Databricks](/Software/Databricks) — uses · Software

### Similar Software

- [Sanitization Rules Engine](/Software/Sanitization_Rules_Engine) — similar · Software
- [Identity Management Systems](/Metrics/Policy_Compliance_Rate/Software/Identity_Management_Systems) — similar · Software
- [Data Reliability Engine](/Software/Data_Reliability_Engine) — similar · Software
- [Compliance Validation Engine](/Software/Compliance_Validation_Engine) — similar · Software
- [Compliance Management Software](/Metrics/Time_To_Sign-Off/Software/Compliance_Management_Software) — similar · Software
- [Security Monitoring Systems](/Metrics/Policy_Compliance_Rate/Software/Security_Monitoring_Systems) — similar · Software
- [Policy Management Software](/Metrics/Policy_Compliance_Rate/Software/Policy_Management_Software) — similar · Software

### Similar Startups

- [Latticeforge](/Startups/Latticeforge) — similar · Startups
- [Sensept](/Startups/Sensept) — similar · Startups
- [Anirit](/Startups/Anirit) — similar · Startups
- [Sanode](/Startups/Sanode) — similar · Startups
- [Blendactable](/Startups/Blendactable) — similar · Startups
- [Logreg](/Startups/Logreg) — similar · Startups
- [Abantern](/Startups/Abantern) — similar · Startups
- [Prifig](/Startups/Prifig) — similar · Startups
- [Bedractable](/Startups/Bedractable) — similar · Startups
- [In-House Sanitization Scripts](/Startups/In-House_Sanitization_Scripts) — similar · Startups
- [Prifect](/Startups/Prifect) — similar · Startups

### Similar Agents

- [Pipeline Gateway API](/Agents/Pipeline_Gateway_API) — similar · Agents