AI Coding Agent Platforms
We’re in security review—how does Factory export audit logs/telemetry to our SIEM via OpenTelemetry?
9 min read
Most security reviews boil down to two questions: what do you log, and how cleanly can you plug those logs into our existing observability and SIEM stack? Factory’s answer is simple: everything runs on OpenTelemetry, and you can export both audit logs and analytics telemetry to whatever OTLP-compatible collector you already trust.
This guide breaks down how Factory handles audit logging, how OpenTelemetry is wired into the platform, and what it looks like to get data into tools like Splunk, Datadog, Grafana, or New Relic during a security review.
Scope: This article focuses on log/telemetry export and integration patterns, not step‑by‑step UI instructions. It’s written for security, platform, and SRE teams running a formal review.
At-a-Glance Comparison
| Rank | Option | Best For | Primary Strength | Watch Out For |
|---|---|---|---|---|
| 1 | Direct OTLP export from Factory Analytics | Centralized SIEM/observability stacks (Splunk, Datadog, Grafana, New Relic) | Native OpenTelemetry pipeline with structured metrics and events | Requires OTLP endpoint configuration and network allowlisting |
| 2 | Audit log export + SIEM ingestion | Compliance, incident forensics, access oversight | Rich, human-readable audit trail aligned to security workflows | Volume considerations; you’ll want retention policies tuned in the SIEM |
| 3 | Hybrid: OTEL metrics + audit logs | Enterprises tying Droid usage to security + engineering outcomes | Combines telemetry (tokens, tools, usage) with user-level audit history | Slightly more setup: two feeds, two retention strategies |
Comparison Criteria
We evaluated these integration paths against three concerns most security teams raise in review:
- Control & Visibility: Can you see who did what, where, and when across Droids, tools, and code surfaces? Are the events structured enough for alerting and investigations?
- Integration Path: How cleanly does Factory plug into your existing SIEM and observability pipeline (no custom agents, no proprietary collectors)?
- Compliance & Governance: Does the logging model support SOC 2–grade controls, export to your SIEM, and alignment with standards like ISO 42001, GDPR, and CCPA?
Detailed Breakdown
1. Direct OTLP Export from Factory Analytics (Best overall for org-wide telemetry)
Direct OTLP export via Factory Analytics is the top fit for most security and platform teams because it uses OpenTelemetry end-to-end and pushes structured metrics and events into your existing observability backplane.
Factory Analytics is built entirely on OpenTelemetry. Every metric surfaced in the hosted dashboard is available via OTLP export to tools like Datadog, Grafana, New Relic, Splunk, or any other OTLP-compatible collector.
What it does well:
- Native OpenTelemetry pipeline:
- All agent usage and impact metrics (token consumption, tool calls, activity and adoption, productivity output, per-user breakdowns, agent readiness) are modeled as OTEL metrics/spans.
- You configure a collector endpoint (e.g., OTLP over gRPC/HTTP), and Factory streams telemetry into your existing ingestion layer—no sidecars, no proprietary protocol.
- Org-level observability:
- You can correlate Droid usage with incident timelines, deployment windows, and other infra signals inside a single observability stack.
- This is particularly useful in “war room” scenarios: you can see when Droids were invoked, what tools they used, and how that lined up with alerts, rollbacks, or changes.
Tradeoffs & Limitations:
- Requires OTLP endpoint coordination:
- Your team must expose an OTLP-compatible endpoint (often a gateway collector) that Factory can reach.
- Network controls, firewalls, and VPC peering rules need to be aligned so telemetry flows out while preserving your security posture.
Decision Trigger: Choose direct OTLP export as your primary path if you want structured, high-volume telemetry flowing into your existing observability graph and you already standardize on OpenTelemetry.
2. Audit Log Export + SIEM Ingestion (Best for compliance and forensic detail)
Audit log export is the strongest fit when your priority is compliance, access investigations, and “who did what when” reviews for Droids and users.
Factory maintains detailed audit logs that capture how Droids are operating, which tools they invoked, and who initiated or approved actions. These logs are designed to be exported to your SIEM for monitoring and alerting.
What it does well:
- Security-focused event detail:
- Logs include user-level context and action types (e.g., a Droid reading from a repo, generating edits, posting a PR, fetching from Slack/Teams, or working in a terminal).
- You can see source surfaces (IDE, web, CLI, Slack/Teams, project trackers) and tie every step in a delegated task back to a human user and a ticket or incident.
- SIEM-ready export:
- Factory supports exporting audit logs so you can ingest them using your existing SIEM pipeline rules.
- Once in your SIEM, you can set alerts on unusual behavior (e.g., unexpected repo access patterns, spike in Droids querying sensitive services, after-hours usage anomalies).
Tradeoffs & Limitations:
- Volume & retention management:
- Audit logs can be high volume in active orgs. You’ll want to define retention windows in your SIEM that balance forensic depth with cost.
- For aggregate trends and engineering-level reporting, you’ll still lean on Metrics/OTEL (see Option 1).
Decision Trigger: Choose audit log export as your primary integration path if your review is being run by security/compliance stakeholders and the core requirement is a defensible, SIEM-ingested audit trail for every Droid and user action.
3. Hybrid: OTEL Metrics + Audit Logs (Best when you care about both security and engineering outcomes)
The hybrid pattern combines OTLP telemetry export from Factory Analytics with audit-log export to your SIEM. This is where most mature enterprises land after the first security review.
It stands out because it gives security teams the detail they need while letting engineering leadership measure ROI, autonomy, and performance improvements from Droids—all using the same OpenTelemetry backbone.
What it does well:
- Security + performance in one view:
- Audit logs answer “who did what when” and feed incident response, investigations, and compliance.
- OTEL metrics answer “what impact did our Droids have?”—files edited, PRs created, usage distributions, autonomy ratio, and readiness signals.
- Together, you can correlate security events with engineering impact.
- Flexible reporting surfaces:
- Security teams work from the SIEM, chasing audit events.
- Engineering and platform teams use their observability stack and Factory’s hosted Analytics dashboards (with CSV export if needed) to see trends, adoption, and outcomes.
Tradeoffs & Limitations:
- Two integration flows to manage:
- You’ll configure OTLP export plus whatever pipeline you use to ingest log data.
- Policy decisions (e.g., retention, access, PII handling) need to be made for both metrics and logs.
Decision Trigger: Choose the hybrid approach if you want to satisfy security and governance requirements while giving engineering a clear measurement framework for Droids’ impact—without introducing shadow telemetry systems.
How Factory’s OpenTelemetry Export Works in Practice
From a security-review standpoint, the key questions are: what exactly is emitted, how is it transported, and how do we control it?
Data Model: Metrics and Events Built on OTEL
Factory Analytics tracks six areas of agent usage and impact:
- Token consumption
- Tool usage
- Activity and adoption
- Productivity output
- Per-user breakdowns
- Agent readiness
Each of these dimensions is modeled as OpenTelemetry metrics/events. That means:
- You get structured attributes (e.g., Droid ID, surface, workspace, project, user) that are suitable for SIEM correlation rules or observability dashboards.
- You can filter and aggregate in your downstream tools just as you do for application metrics today.
Transport: OTLP to Your Collector
Factory exports telemetry over OTLP (OpenTelemetry Protocol) to a collector you control. Common patterns:
- Datadog: OTEL collector → Datadog Agent or OTLP intake
- Splunk: OTEL collector → Splunk Observability Cloud or HEC via translation
- Grafana / Prometheus: OTEL collector → Prometheus / Tempo / Loki pipeline
- New Relic: OTEL collector → New Relic OTLP endpoint
You keep control of:
- Network boundaries: Factory sends telemetry out to your endpoint; there’s no pull from your network into Factory.
- Sampling and downsampling: Configure in your collector; you decide how much detail to keep.
- Routing: Direct specific metrics or log streams to distinct indexes, projects, or teams.
Security & Compliance Controls Around Logging and Telemetry
Audit logs and telemetry only matter if they sit inside a trustworthy system. Factory’s controls are designed to be inspected in security review:
-
Strict permissions enforcement:
Factory only shows a Droid (and the user driving it) the data they already have access to in the source system. Logs reflect the same permission boundaries—no “hidden context” that bypasses your existing ACLs. -
Encryption in transit and at rest:
- All data is encrypted in transit using TLS 1.2+.
- All data is encrypted at rest using AES-256.
- Controls are in place to limit data exfiltration risks; telemetry and audit export respect your network and policy configuration.
-
Single-tenant, sandboxed environments:
Enterprise deployments run in single-tenant, sandboxed environments with their own VPC, keeping your telemetry and logs isolated from other customers. -
Compliance posture:
- SOC 2–aligned with a completed SOC 2 Type I certification.
- Alignment with GDPR and CCPA for data protection and privacy.
- Early adoption of ISO 42001, the standard for AI management systems.
-
No training on your code without consent:
Factory does not use your code or private data as training data without prior written consent. Your telemetry and audit logs are for your governance and analytics, not for model training.
For security review, this means you can connect Factory to your SIEM and observability stack while preserving the same compliance posture you expect from other core engineering systems.
How Security and Platform Teams Typically Use These Feeds
Once audit logs and OTEL telemetry are integrated, teams usually implement:
-
Security use cases (via SIEM + audit logs):
- Alert on unexpected repo or system access patterns by Droids.
- Monitor after-hours or location-anomalous usage.
- Correlate incident timelines with Droid-powered investigations and fixes.
-
Engineering & leadership use cases (via OTEL metrics):
- Track adoption: how often teams trigger Droids in IDEs, terminals, Slack/Teams, and CI.
- Measure productivity: files edited, PRs created, tests generated, incident investigations completed.
- Monitor the “autonomy ratio” and agent readiness to understand where Droids can take on more end-to-end tasks safely.
Because everything is built on OpenTelemetry, these dashboards can live in the same place you track deploys, incidents, and SLOs.
Final Verdict
For a security review, the cleanest approach is:
- Use Factory’s OTLP export from Analytics as the default path for metrics and telemetry, integrating with your existing OTEL collector and observability tools.
- Enable audit log export into your SIEM so your security team has full visibility into user and Droid actions with SOC 2–grade traceability.
- For mature organizations, adopt the hybrid pattern to align security governance with engineering outcomes in a single, observable system.
You get a verifiable audit trail, exportable telemetry, strict permissions, and encryption—without forcing new agents or proprietary logging pipelines into your stack.