# Edge Log Filter

*/Opportunities/Edge_Log_Filter*

## Opportunity Overview

**Wedge**: Target B2B SaaS companies running Kubernetes that spend over $20,000 monthly on Datadog or Splunk. This niche experiences acute, measurable budget pain and utilizes standardized logging pipelines for rapid proof-of-value. Expand by moving from simple drop-rules to intelligent log summarization, and eventually to edge-based anomaly detection and alerting.
**Timing**: Lightweight classification models now run efficiently on edge nodes with minimal CPU and memory overhead, enabling real-time log parsing at the source. Simultaneously, standard observability platforms charge high ingest fees, forcing engineering leaders to actively seek cost-reduction tools.
**Why This I C P**: DevOps and SRE leaders at Kubernetes-heavy organizations directly own the observability budget and possess the technical authority to deploy infrastructure agents, making purchasing decisions fast and ROI quantifiable.
**Size Of Prize**: ~40,000 mid-market and enterprise companies with significant cloud infrastructure × ~$30,000 annual spend on telemetry optimization software ≈ $1.2B addressable market.
**Gap Narrative**: Cloud engineering teams transmit terabytes of low-value telemetry to centralized observability platforms, inflating ingest and storage bills. They lack a mechanism to intelligently classify, summarize, and drop noisy logs directly at the compute node before transmission without losing critical audit trails.
**Defensibility**: Defensibility builds through infrastructure lock-in and localized data models. The system learns the specific log topology and noise patterns of a customer's environment, making the filtering logic highly accurate over time. Once embedded as a standard deployment component across thousands of compute nodes, the technical effort required to replace it creates high switching costs.
**Why This Thesis**: A Software thesis deployed as an edge agent or eBPF probe physically intercepts log streams before network transit. This structural approach prevents cloud egress costs and eliminates the need to route data to an intermediate processing cluster.

## Opportunity Linked Thesis

**Thesis**: [Software](/Theses/Software)

## Opportunity Linked I C P

**Icp**: [Content Delivery Network](/CompanyTypes/Content_Delivery_Network)

## Opportunity Market Sizing

_Illustrative — target and order-of-magnitude estimate figures, not an achieved track record (this Thing is concept-stage)._

**S A M**: ~$400M-600M targeting dedicated Content Delivery Networks and large edge compute providers
**S O M**: ~$15M-35M
**T A M**: ~10,000 global edge and cloud infrastructure operators x ~$150k/yr = ~$1.5B
**Growth Rate**: ~20-25%/yr, driven by expanding edge node footprints and rising cloud egress costs for raw telemetry data
**Paid Comparable Spend**: ~$300k-800k/yr spent on raw SIEM ingestion fees, unfiltered cloud storage, and data engineering labor to manage high-volume log pipelines

## Opportunity Incumbents

- [Cribl Stream](/Products/Cribl_Stream) — Tool
- [Datadog Vector](/Products/Datadog_Vector) — Open-Source
- [Fluent Bit](/Products/Fluent_Bit) — Open-Source
- [Mezmo Telemetry Pipeline](/Products/Mezmo_Telemetry_Pipeline) — Service
- [Custom Routing Scripts](/Products/Custom_Routing_Scripts) — DIY
- [Elastic Logstash](/Products/Elastic_Logstash) — Tool

## Opportunity Win Conditions

**Kill Thresholds**:
- Average agent CPU overhead > 2% during production load
- Log volume reduction < 20% during the proof of value phase
- Time to active filtering on 10+ nodes > 14 days
- Pilot conversion rate < 25% after 45 days
**Leading Metrics**:
- Percentage of redundant log volume dropped at the edge
- Filter agent CPU and memory consumption per edge node
- Time to first active dropping rule deployment
- Daily active edge nodes routing filtered telemetry
- SIEM ingestion volume reduction percentage
**What Proves Right**: Infrastructure operators deploy the filter on at least 100 edge nodes within 14 days of a pilot. Users configure rules that drop more than 40 percent of redundant log volume before SIEM ingestion, creating immediate and visible cost savings. Cohorts retain at 90 percent past the 90-day mark and convert to $50k+ annual contracts based on validated SIEM cost reductions.
**What Proves Wrong**: Edge compute environments experience greater than 5 percent CPU or memory overhead from the filter agent, triggering immediate uninstalls by infrastructure operators. Users misconfigure parsing rules and drop critical security telemetry, destroying trust and ending pilots early. The total volume reduction yields less than $20k in SIEM savings, making the product uncompetitive against free open-source tools like Fluent Bit.

## Opportunity Build Profile

**Hardest Part**: Processing tens of thousands of log events per second with sub-millisecond latency and zero data loss during unpredictable traffic spikes.
**Min Viable Scope**: A lightweight Kubernetes sidecar agent that executes static regex-based drop rules and simple percentage sampling. Deliberately exclude stateful log aggregation, automatic PII redaction, and centralized fleet management control planes.
**Cold Start Problem**: Engineering teams refuse to put unproven agents in the critical path of production telemetry. Break this by deploying in a read-only shadow mode that ingests a duplicated stream to prove exact cost savings without intercepting live traffic.
**Time To First Value**: Minutes after deployment; observability dashboards immediately reflect the drop in ingest volume and associated costs.
**Data Moat Available**: false
**Technical Difficulty**: Moderate

## Neighborhood

### Where the gap lives

- [Kiln Log Ingestion Agent](/Agents/Kiln_Log_Ingestion_Agent) — latent gap · Agents
- [Log Analysis](/Skills/Log_Analysis) — latent gap · Skills
- [Log Anomaly Triage Agent](/Agents/Log_Anomaly_Triage_Agent) — latent gap · Agents

### Incumbent in

- [Mezmo Telemetry Pipeline](/Products/Mezmo_Telemetry_Pipeline) — incumbent in · Products
- [Elastic Logstash](/Products/Elastic_Logstash) — incumbent in · Products
- [Fluent Bit](/Products/Fluent_Bit) — incumbent in · Products
- [Cribl Stream](/Products/Cribl_Stream) — incumbent in · Products
- [Custom Routing Scripts](/Products/Custom_Routing_Scripts) — incumbent in · Products
- [Datadog Vector](/Products/Datadog_Vector) — incumbent in · Products

### Applies thesis

- [Content Delivery Network](/CompanyTypes/Content_Delivery_Network) — applies thesis · CompanyTypes

### Embodies

- [Software](/Theses/Software) — embodies · Theses

### Similar Opportunities

- [Edge Telemetry Router](/Opportunities/Edge_Telemetry_Router) — similar · Opportunities
- [Semantic Telemetry Router](/Opportunities/Semantic_Telemetry_Router) — similar · Opportunities
- [Semantic Log Parsing for DevOps](/Opportunities/Semantic_Log_Parsing_for_DevOps) — similar · Opportunities
- [Automated Log Reconciliation](/Opportunities/Automated_Log_Reconciliation) — similar · Opportunities
- [Troubleshooting as a Service](/Opportunities/Troubleshooting_as_a_Service) — similar · Opportunities
- [SLA Degradation Triage](/Opportunities/SLA_Degradation_Triage) — similar · Opportunities
- [Incident Prevention API](/Opportunities/Incident_Prevention_API) — similar · Opportunities
- [Managed Log Compliance](/Opportunities/Managed_Log_Compliance) — similar · Opportunities
- [Cloud FinOps Automation](/Opportunities/Cloud_FinOps_Automation) — similar · Opportunities
- [Capacity Tuning Engine](/Skills/Systems_Evaluation/Opportunities/Capacity_Tuning_Engine) — similar · Opportunities
- [Incident Context Synthesizer](/Opportunities/Incident_Context_Synthesizer) — similar · Opportunities
- [Predictive Load Balancer](/Opportunities/Predictive_Load_Balancer) — similar · Opportunities
- [Predictive Telemetry Engine](/Opportunities/Predictive_Telemetry_Engine) — similar · Opportunities
- [Outage Detection Automation](/Opportunities/Outage_Detection_Automation) — similar · Opportunities
- [IoT Telemetry Filtering For Manufacturing](/Opportunities/IoT_Telemetry_Filtering_For_Manufacturing) — similar · Opportunities
- [Automated Fault Triage](/Opportunities/Automated_Fault_Triage) — similar · Opportunities
- [AI Alert Aggregation](/Opportunities/AI_Alert_Aggregation) — similar · Opportunities
- [Root Cause Analyst](/Opportunities/Root_Cause_Analyst) — similar · Opportunities
- [AI Systems Engineering](/Skills/Systems_Evaluation/Opportunities/AI_Systems_Engineering) — similar · Opportunities
- [Usage Telemetry Agent](/Opportunities/Usage_Telemetry_Agent) — similar · Opportunities