Redpanda BYOC requirements checklist for security review (IAM, networking, private connectivity, logging)

Security teams are right to be opinionated about BYOC. You’re asking Redpanda to orchestrate a high‑throughput streaming engine inside your cloud, against your data, under your controls. This checklist is designed to make a security review concrete: what Redpanda BYOC needs, what you keep, and which controls wrap the whole deployment (IAM, networking, private connectivity, logging, and more).

Use this as a line‑by‑line artifact with your security, networking, and platform teams during design and review.

---

The Quick Overview

• What It Is: Redpanda BYOC (Bring Your Own Cloud) is a deployment model where Redpanda runs and manages the streaming control plane, while the data plane (brokers, storage, network paths) lives entirely inside your cloud accounts.
• Who It Is For: Security‑sensitive teams that require data sovereignty, need Kafka‑compatible streaming, and want managed operations without giving up control of infrastructure or network boundaries.
• Core Problem Solved: You get a high‑performance Kafka alternative plus the emerging Agentic Data Plane—without shipping your event data into a vendor’s VPC.

---

How BYOC Works (Control Plane vs. Data Plane)

At a high level, BYOC splits responsibilities:

• You own: Cloud accounts/projects, VPCs, subnets, security groups, routing, peering, KMS, and local compliance posture.
• Redpanda manages: The Redpanda cluster software, upgrades, configuration, health, and support—using tightly scoped access into your environment.

The flow looks like this:

1. Provision & Connect

   • You create the required cloud resources (VPC, subnets, IAM roles, KMS, logging destinations).
   • You grant Redpanda’s automation minimal IAM roles and bootstrap access (usually via cloud‑native roles and policies, not long‑lived keys).
   • A secure channel is established between Redpanda’s control plane and your cloud environment.

2. Deploy & Configure

   • Redpanda installs the data plane inside your VPC: brokers, tiered storage config, metrics and log shipping.
   • TLS, ACLs/RBAC, and private connectivity options (peering, PrivateLink, VPN, etc.) are enabled according to your requirements.
   • Monitoring hooks (Prometheus/Grafana telemetry) and system logs are wired for operations and debugging.

3. Operate & Govern

   • Redpanda continuously monitors cluster health and performance (CPU, disk, throughput).
   • You enforce access via IAM, OIDC, RBAC, and network controls.
   • Audit trails, security logs, and (for Agentic Data Plane use cases) replayable agent sessions give you a permanent record of what happened.

Absolutely. You keep control; Redpanda makes it reliable and fast.

---

IAM Requirements Checklist

You’ll want a clean picture of “who can do what, where.” This section focuses on cloud IAM plus in‑cluster identity and authorization.

1. Cloud Provider IAM

Goal: Grant Redpanda just enough access to deploy and operate BYOC resources in your environment.

Checklist:

• Dedicated IAM role/service account
  • Create a dedicated role (per environment: dev/stage/prod).
  • Avoid sharing with other vendors or internal automations.
• Scoped permissions for infrastructure
  • Allow only the specific actions needed for:
    • Compute (instances / node groups / autoscaling).
    • Storage (disks/volumes, buckets/objects for tiered storage).
    • Networking (ENIs, load balancers, route updates where required).
  • No blanket *:* policies; attach least‑privilege policies.
• Secrets & KMS integration
  • Enable use of your KMS for:
    • Disk/storage encryption keys.
    • Application secrets (if stored via cloud provider secret managers).
  • Ensure Redpanda BYOC role can use—but not administer—KMS keys.
• Auditable access
  • Ensure all role actions are visible in your cloud audit log (CloudTrail, Cloud Audit Logs, etc.).
  • Confirm your security team knows where these entries appear and how to filter by Redpanda role.

2. Identity for Clients and Operators

Redpanda delivers enterprise features like ACLs, TLS, and RBAC. Tie them into your identity fabric.

Checklist:

• Authentication mechanism chosen
  • mTLS (client certs) for services.
  • OIDC/Kerberos where applicable for human access and admin tools.
• Role‑based access control (RBAC) configured
  • Map Redpanda roles to groups in your IdP (e.g., “streaming‑admins”, “agent‑platform”, “read‑only‑analytics”).
  • Use RBAC to gate:
    • Topic creation/deletion.
    • Schema changes (if using Schema Registry).
    • Administrative API access.
• Fine‑grained ACLs
  • Define topic‑level ACLs for:
    • Production apps.
    • Agentic workloads.
    • External/partner integrations.
  • Verify ACLs are enforced before data is produced/consumed—critical for agents that act on operational data.
• Credential lifecycle
  • Rotate keys/certs and tokens on your standard cadence.
  • Use short‑lived credentials for interactive access.

---

Networking Requirements Checklist

Your security review will zero in on network blast radius and egress patterns. BYOC is designed so the data plane stays in your VPC.

1. VPC and Subnet Layout

Checklist:

• Dedicated VPC or well‑segmented shared VPC
  • Prefer isolated VPC per environment for strict segmentation.
  • If shared, use network policies to isolate Redpanda subnets.
• Private subnets for brokers
  • Brokers, controller nodes, and observability components live in private subnets.
  • No direct public IPs on brokers.
• Controlled egress
  • Outbound traffic restricted to:
    • Redpanda control plane endpoints (documented allowlist).
    • Your observability/logging endpoints (Prometheus, SIEM, etc.).
  • Use NAT gateways or egress firewalls with explicit rules.

2. Inbound Access Patterns

You choose how apps, agents, and admins reach the cluster.

Checklist:

• Internal access only (recommended for production)
  • Access via:
    • Peered VPCs.
    • Transit gateway.
    • Service mesh / internal load balancers.
• Per‑environment entry points
  • Separate LBs or endpoints for dev, staging, and prod.
  • Use firewall rules/security groups to restrict source CIDRs and security group IDs.
• Administrative access controls
  • Restrict admin API endpoints to:
    • Bastion hosts.
    • Privileged management subnets.
    • Specific admin security groups.

---

Private Connectivity Requirements Checklist

For most regulated environments, “no public hops” is non‑negotiable. BYOC supports classic private connectivity patterns.

1. VPC Peering / Transit Gateways

Checklist:

• Peering strategy decided
  • Direct VPC peering between app VPCs and Redpanda VPC.
  • Or central transit gateway / hub‑and‑spoke architecture.
• Route tables configured
  • Only necessary CIDR ranges routed.
  • No broad “0.0.0.0/0” back to Redpanda subnets.
• Security group alignment
  • Allow necessary ports:
    • Kafka/Redpanda client ports.
    • Admin/HTTP ports (restricted).
  • Restrict to specific security group IDs, not just CIDRs, where possible.

2. PrivateLink / Private Service Connect (if supported in your target cloud)

Checklist:

• Endpoint services configured
  • Private endpoints created per environment.
  • DNS override for Redpanda bootstrap hosts routed via private endpoints.
• Zero public internet path
  • Confirm traffic from apps to Redpanda remains on provider backbone.
  • Validate flows with VPC flow logs.

3. Hybrid & On‑Prem Connectivity

For hybrid workloads:

• VPN or Direct Connect/ExpressRoute
  • Encrypted tunnel or dedicated connectivity into the Redpanda VPC.
  • Routes scoped to only necessary subnets.
• Latency SLOs
  • Measure RTT between on‑prem and Redpanda brokers.
  • Ensure your design supports required throughput (e.g., >=100K tx/sec, or 100GB/min for gaming‑like workloads).

---

Logging, Telemetry, and Observability Checklist

Your security review will ask: “What does Redpanda see, and what do we see?”

Redpanda BYOC is built with built‑in observability via Prometheus and a clear separation between operational telemetry and your actual data.

1. Data You Provide to Redpanda

Per BYOC service terms, your environment sends Redpanda:

• Grafana/Prometheus telemetry: CPU, disk, usage metrics for health and capacity planning.
• System logs: e.g., syslog / journalctl for debugging.
• HTTP Admin: For command and control of the cluster.

Checklist:

• Metrics scope understood
  • Confirm metrics contain operational details only, not message payloads.
• System logs configuration
  • Ensure system logs do not contain sensitive application payloads.
  • If needed, configure log scrubbing/anonymization on your side before shipping.
• Admin API protection
  • HTTP Admin endpoints are:
    • TLS‑protected.
    • Authenticated and authorized via RBAC/IAM.
    • Logged and auditable.

2. Your Security and Audit Logs

Redpanda also supports security audit logs and cluster‑level auditing.

Checklist:

• Centralized log shipping
  • Forward Redpanda audit logs, system logs, and access logs to:
    • Your SIEM (Splunk, Datadog, Elastic, etc.).
    • Your cloud‑native logging service.
• Audit log coverage
  • Confirm logging of:
    • Authentication successes/failures.
    • ACL/RBAC changes.
    • Topic creation/deletion.
    • Cluster configuration changes.
• Retention and immutability
  • Store logs in an immutable or WORM‑compliant target if required for regulatory reasons.
• Agentic use cases
  • For Agentic Data Plane deployments:
    • Ensure “every agent interaction is captured.”
    • Capture and retain session transcripts and tool calls for replay/debugging.
    • Align retention with internal AI and compliance policies.

---

Encryption and Data Protection Checklist

You’re responsible for the data; BYOC needs to align with your encryption standards.

1. Data in Transit

Checklist:

• TLS everywhere
  • Broker <-> client connections encrypted with TLS.
  • Admin/API endpoints exposed only over HTTPS.
• Certificate management
  • Use your CA or trusted provider for cert issuance.
  • Automate rotation; avoid expired‑cert outages.

2. Data at Rest

Checklist:

• Disk/volume encryption
  • Use cloud‑provider disk encryption with your KMS keys.
• Tiered storage encryption
  • Buckets/objects for tiered storage encrypted with your keys.
• Key management
  • KMS keys governed by your own policies (rotation, access, separation of duties).
  • Redpanda’s BYOC role has use permissions but not administrative control.

---

Security Controls & Governance Checklist

This is where your security team asks: “What are the guardrails if something goes wrong?”

1. Enterprise Security Posture

Per BYOC terms, both you and Redpanda commit to:

• Reasonable, industry‑standard security measures.
• Audited cloud environment, WebAssembly‑based encryption, and security audit logs.

Checklist:

• Security baselines applied
  • CIS or internal hardening baselines applied to:
    • OS images.
    • Network policies.
    • IAM policies.
• Vulnerability management
  • Define how patches, upgrades, and CVE remediation happen within BYOC.
  • Align with your change‑management process.
• Pen‑test and third‑party risk
  • Incorporate BYOC into your vendor risk program.
  • Ensure security reviews include Redpanda’s BYOC architecture and controls.

2. Agent‑First Governance (Optional but Recommended)

If you’re using Redpanda as an Agentic Data Plane, extend your checklist to agent behavior:

• Identity for agents
  • Agents authenticate via OIDC or short‑lived service credentials.
• On‑behalf‑of (OBO) authorization
  • Agents access data under the human or app identity they represent.
  • Policies enforced before tools run or data is changed.
• Tool‑level policies
  • Filter/redact/restrict tools agents can use (e.g., write‑capable vs read‑only topics).
• Kill switch
  • Ability to disable agent access at once (revoke tokens, disable roles, cut network paths).
• Replayability
  • Full replay of agent sessions and tool calls for forensic analysis.

---

Compliance and Data Sovereignty Checklist

BYOC is built for teams that can’t move data into a vendor VPC.

Checklist:

• Region and residency alignment
  • Deploy BYOC clusters in required regions (per data residency and localization policies).
• Data never leaves your cloud
  • Confirm architecture: event data lives in your VPC and your cloud accounts.
  • Redpanda receives only operational telemetry and logs as described above.
• BYOC across AWS/GCP/Azure
  • For multi‑cloud, ensure:
    • Separate accounts/projects per provider, if mandated.
    • Consistent IAM/network/logging patterns across clouds.
• Audit attestations
  • Map BYOC controls to your frameworks:
    • SOC 2, ISO 27001, PCI, HIPAA, etc., as applicable.
  • Document the shared responsibility model for internal auditors.

---

Putting It All Together: Pre‑Review Checklist Snapshot

Here’s a compressed view to carry into your security review:

• IAM
  • Dedicated Redpanda BYOC roles with least privilege.
  • RBAC + ACLs configured within Redpanda.
  • Short‑lived creds, OIDC/Kerberos where applicable.
• Networking
  • Private subnets, no public IPs on brokers.
  • Controlled egress, peering/Transit/PrivateLink configured.
  • Admin endpoints locked down by SGs/firewalls.
• Private Connectivity
  • Clear app‑to‑cluster paths (VPC peering, TGW, VPN, or Direct Connect).
  • Validated latency and throughput for your workloads.
• Logging & Telemetry
  • Metrics + system logs shared with Redpanda as defined.
  • Security/audit logs centralized in your SIEM.
  • Clear retention and immutability for compliance.
• Encryption
  • TLS in transit, KMS‑backed encryption at rest.
  • Well‑governed key lifecycle.
• Security & Governance
  • Baseline hardening (CIS or equivalent).
  • Vulnerability and change‑management processes defined.
  • Agent governance (identity, OBO, tool‑level policies, kill switch, replay) if applicable.
• Compliance & Sovereignty
  • Region selection, data residency verified.
  • Documented shared responsibility model.

---

Summary

Redpanda BYOC is built for teams that want the performance and simplicity of a Kafka‑compatible engine—plus an Agentic Data Plane for AI—without ever surrendering control of their data or cloud perimeter. Your security review should focus on concrete levers: IAM roles, VPC boundaries, private connectivity, TLS/KMS, telemetry scope, and audit logging. With those dialed in, you get the streaming backbone for real‑time and agentic workloads, inside your own accounts, with a permanent record and a kill switch when you need it.

Next Step

Get Started