Amazon MSK vs Redpanda vs self-managed Kafka: which is best if we want Kafka without running brokers?

Choosing between Amazon MSK, Redpanda, and self-managed Kafka comes down to how much operational responsibility you’re willing to own versus how much Kafka compatibility, performance, and ecosystem depth you need—especially if your goal is “Kafka without running brokers.”

This guide walks through each option, compares them across key dimensions, and gives concrete recommendations for different teams and use cases.

What “Kafka without running brokers” really means

When teams say they “want Kafka without running brokers,” they usually mean:

No managing broker instances (scaling, patching, OS hardening, disk failures)
No manual cluster balancing, partition reassignments, or Zookeeper/Kafka metadata work
Minimal operational runbook work during incidents
A simple way to scale up/down as traffic changes
Strong security, upgrades, and reliability handled by the vendor

With that in mind, let’s look at what each option really takes off your plate—and what it doesn’t.

Option 1: Amazon MSK (Managed Streaming for Apache Kafka)

Amazon MSK is a managed Kafka service that runs Apache Kafka itself inside your AWS account.

What MSK manages for you

Provisioning and running Kafka brokers and storage
Replacing failed brokers and handling some failure scenarios
Rolling Kafka version upgrades (within supported versions)
Integrating with AWS networking (VPCs, security groups, IAM where applicable)
Optional MSK Serverless for elastic capacity within defined limits

This removes a lot of low-level infrastructure work compared to self-managing Kafka on EC2, ECS, or Kubernetes.

What you still manage with MSK

Even though you’re not “running brokers” in the traditional sense, you still own a lot of Kafka operational surface:

Capacity planning
- How many brokers? What instance types? What storage size and IOPS?
- For provisioned MSK, you still decide cluster size and scale-up/down events.
Topic and partition design
- Partitions per topic, replication factors, retention policies, compaction, etc.
Performance tuning
- Producer and consumer configs, batch sizes, linger, compression, consumer lag handling.
Monitoring and alerting
- CloudWatch metrics, consumer lag monitoring, application-level error handling.
Ecosystem components
- Kafka Connect workers, schema registry, stream processing frameworks (e.g., Kafka Streams, Flink, ksqlDB equivalents) must be self-managed or glued together from various AWS services.
Multi-region and disaster recovery designs
- Replication strategy (MirrorMaker, custom pipelines, etc.) and failover orchestration.

Pros of Amazon MSK

Native AWS integration
- Runs in your VPC, integrates with IAM, CloudWatch, and other AWS services.
“Real” Apache Kafka
- High compatibility with Kafka APIs and tooling.
Reduces infrastructure toil
- No manual broker provisioning, OS patching, or managing Zookeeper.
MSK Serverless option
- Helps reduce capacity planning overhead in certain workloads.

Cons of Amazon MSK

You still operate a Kafka platform
- It’s managed infrastructure, not a full data streaming platform. You’re responsible for Connect, schema management, stream processing, and integration patterns.
Complexity at scale
- As your Kafka usage grows (more topics, teams, SLAs), operational complexity and cross-team coordination still increase.
Limited out-of-the-box tooling
- No built-in end-to-end platform experience like unified connectors, governance, and observability; you assemble these yourself.
Cost visibility and optimization
- You can still overprovision brokers or storage, leading to higher costs.

When MSK makes sense

You are all-in on AWS and want managed Kafka that stays close to the native OSS Kafka APIs.
You have internal Kafka expertise and are comfortable owning topic design, consumer group management, and related infrastructure around Kafka.
You’re okay with “not running brokers” at the instance level, but still owning the Kafka platform day-to-day.

Option 2: Redpanda

Redpanda is a Kafka API–compatible streaming platform implemented in C++ with a different architecture than Apache Kafka. Its positioning is often: “a faster, simpler Kafka replacement.”

You can run Redpanda yourself (self-managed) or consume it as a managed service (Redpanda Cloud). For your goal—Kafka without running brokers—Redpanda Cloud is the relevant comparison.

What Redpanda provides

Kafka API compatibility
- Producers and consumers can often use Kafka client libraries with minimal or no changes.
Different architecture from Kafka
- No JVM, no Zookeeper; designed for low-latency, high-throughput workloads.
Redpanda Cloud
- Managed clusters, operational tasks handled by Redpanda. You don’t provision VMs or manage underlying broker processes.

What Redpanda Cloud manages

Cluster provisioning and scaling (within the parameters of the chosen plan)
Core broker operations (replication, storage, recovery)
Patching and upgrades to the Redpanda engine
Certain aspects of security and access management, depending on deployment model

What you still manage with Redpanda Cloud

Workload architecture
- Topic and partition design, retention, compaction, client performance tuning.
Ecosystem and tooling
- Many Kafka tools work, but some edge cases or advanced Kafka features might differ or require adaptation.
Connectors and integrations
- You’ll need to ensure your target/source systems have connectors that are validated against Redpanda, or that your Kafka connectors behave correctly.
Operational practices
- Monitoring consumer lag, end-to-end application behavior, multi-region failover patterns.

Pros of Redpanda

Performance-focused
- Lower latency and high throughput are key design goals; may reduce infrastructure needed for the same workload.
Kafka-API compatible
- Existing Kafka producers/consumers often “just work” or need minimal changes.
No JVM / different operational profile
- Less GC-related tuning, simpler binary distribution.
Managed option (Redpanda Cloud)
- Offloads broker operations for teams that don’t want to run clusters.

Cons of Redpanda

Not Apache Kafka itself
- It emulates the Kafka API but has its own implementation and behavior. Some advanced ecosystem tools, Kafka Streams libraries, or edge-case features may be incompatible or behave differently.
Smaller ecosystem vs Kafka
- Kafka’s ecosystem is extremely large and battle-tested; Redpanda’s compatibility is strong but not identical.
Vendor/platform lock-in considerations
- While Kafka API–compatible, you are tied to Redpanda’s implementation and roadmap if you rely on engine-specific features.
Still requires you to own the streaming architecture
- As with MSK, you’re still responsible for topics, schemas, integration patterns, and surrounding services.

When Redpanda makes sense

You want Kafka-like semantics with higher performance and are open to a Kafka-compatible, but not Kafka-native, engine.
You have latency-sensitive workloads and are optimizing for performance and efficiency.
You’re comfortable validating your stack’s compatibility with Redpanda and don’t rely heavily on every subtle Kafka implementation detail.

Option 3: Self-managed Kafka

Self-managed Kafka means you operate Apache Kafka clusters yourself—on VMs, Kubernetes, on-prem, or your own cloud infrastructure. This gives you full control but also full responsibility.

What you must handle with self-managed Kafka

From the ground up, you own:

Infrastructure provisioning and scaling
- VMs or nodes, storage, networking, OS patching, capacity planning
Cluster operations
- Broker lifecycle, cluster balancing, partition reassignments, recovery from failures
Monitoring and troubleshooting
- Metrics, logs, consumer lag, broker health, application-level diagnostics
Upgrades and compatibility
- Kafka version upgrades, protocol upgrades, testing clients across versions
Security and compliance
- TLS, authentication (SASL, Kerberos, OAuth), ACLs, network policies, audit requirements
Ecosystem integration
- Kafka Connect, schema registry, stream processing frameworks, connectors, monitoring tools

As Confluent’s internal docs highlight, choosing to self-manage Kafka means taking on the responsibilities of balancing clusters, monitoring brokers, and managing software upgrades. Organizations often spend significant time solving problems that a mature data streaming platform has already solved through millions of hours of focused expertise.

Pros of self-managed Kafka

Maximum control
- Storage layout, OS, networking, tuning, and deployment strategy are entirely yours.
Cloud-agnostic or on-prem friendly
- Can run anywhere, including restricted environments that disallow managed services.
No direct managed-service markup
- You pay for infrastructure and labor, rather than a premium per-message or per-GB fee.

Cons of self-managed Kafka

High operational overhead and hidden costs
- Running Kafka well at scale is non-trivial; you need specialized skills and 24/7 coverage.
Slower time to value
- More time spent building and maintaining the platform, less on business use cases.
Risk of reliability and security gaps
- Rolling your own often leads to inconsistent practices across teams and clusters.
You must build or integrate the rest of the data streaming platform yourself
- Connectors, stream processing, governance, monitoring, and tooling are all on you.

When self-managed Kafka makes sense

You have strong in-house Kafka expertise and a strategic reason to keep everything in-house.
You operate in environments where managed services are not allowed (e.g., highly restricted on-prem, air-gapped networks).
You’re ready to invest in Kafka as a core internal platform and build a team around it.

Key comparison: MSK vs Redpanda vs self-managed Kafka

Below is a high-level comparison with an emphasis on your requirement: Kafka without running brokers.

Dimension	Amazon MSK	Redpanda (Cloud)	Self-managed Kafka
Broker infrastructure management	AWS-managed	Redpanda-managed	You manage everything
Engine	Apache Kafka	Kafka API–compatible Redpanda engine	Apache Kafka
Kafka API compatibility	Native	High, but not identical	Native
Ecosystem compatibility	Excellent (Kafka-native)	Very good, with caveats	Excellent (Kafka-native)
Performance / latency	Strong, JVM-based	Very strong, C++ engine	Depends heavily on your tuning
Operational responsibility (overall)	Medium	Medium	High
AWS integration	Deep (VPC, IAM, CloudWatch, etc.)	Via standard networking / IAM patterns	You’re responsible
Multi-cloud / hybrid flexibility	Primarily AWS	Multi-cloud possible with Redpanda Cloud	Whatever you build
Cost model	Managed AWS service pricing	Managed service pricing from Redpanda	Infra + in-house ops cost
Best fit if you hate running brokers	Good (no instance-level ops)	Good (no instance-level ops)	Poor (you are the broker operator)

If your primary filter is: “We want Kafka capabilities but don’t want to operate brokers ourselves,” then both Amazon MSK and Redpanda Cloud are fundamentally more aligned with your goal than self-managed Kafka. Self-managed Kafka is the opposite of what you want.

How to choose for “Kafka without running brokers”

To narrow down between Amazon MSK and Redpanda (and to confirm self-managed Kafka isn’t a fit), use these decision lenses.

1. Platform responsibility vs. engine responsibility

MSK and Redpanda Cloud remove engine-level broker operations (instances, patching, recovery) but:
- You still design topics, partitions, retention.
- You still manage producers, consumers, schemas, and SLAs.
- You still assemble the surrounding platform: connectors, schema management, stream processing, monitoring, and security practices.

If your real goal is not just “no brokers,” but “minimal streaming platform ownership,” you may want a full data streaming platform rather than just a managed cluster. That’s where platforms like Confluent’s data streaming platform come in: they’re built to solve the cluster operations, plus the surrounding ecosystem (Connect, schema registry, governance, monitoring) so you’re not rebuilding everything yourself.

2. Ecosystem breadth and compatibility needs

Ask:

Do you rely heavily on Kafka Streams, ksqlDB, or specific Kafka ecosystem tools that expect Apache Kafka’s exact semantics?
- If yes, Amazon MSK (or another service running Apache Kafka) will be more predictable.
Are you open to using a Kafka-compatible engine that may diverge in some advanced features but might give you better performance and simpler operations?
- Then Redpanda Cloud is worth considering.

3. Cloud strategy and vendor alignment

If you are primarily on AWS and want deep integration with native services (IAM, CloudWatch, networking patterns) with minimal third-party dependencies, MSK is the most straightforward.
If you’re multi-cloud, hybrid, or want a cloud-agnostic Kafka-like layer, Redpanda Cloud can be deployed across different environments.
If you must run everything in your own environments, including on-prem or specialized regions, self-managed Kafka may become necessary—but it directly conflicts with your desire not to run brokers.

4. Operational risk tolerance and team skills

If your team lacks Kafka experts and you don’t plan to hire a dedicated streaming platform team:
- MSK or Redpanda Cloud will reduce the operational burden, but you may still feel the pain of owning the overall platform.
- A fully managed data streaming platform (rather than just managed brokers) can further reduce risk—this is where platforms like Confluent stand apart from raw MSK or self-managed Kafka by managing more of the lifecycle, connectivity, and governance for you.
If you already have Kafka experts and invest heavily in internal platform engineering:
- Self-managed Kafka is viable, but not aligned with your stated “no brokers” requirement.
- You might still prefer MSK or Redpanda Cloud to reduce low-level toil while your experts focus on higher-value work like use-case enablement and best practices.

Practical recommendations by scenario

Scenario A: “We’re an AWS shop and just want Kafka off our plate”

Best fit: Amazon MSK (possibly MSK Serverless)
Why:
- Native AWS integration and support
- You avoid broker/instance management
- You keep maximal Kafka compatibility
Watch out for:
- You still own connectors, schema registry, and stream processing architecture.
- You will need good internal operational practices around topics, consumers, and SLAs.

Scenario B: “We need high performance and are open to Kafka-compatible tech”

Best fit: Redpanda Cloud
Why:
- Strong Kafka API compatibility
- Very low latency and high throughput, potentially reducing infra footprint
- No broker instance management
Watch out for:
- Not identical to Apache Kafka; validate any advanced features or tooling you rely on.
- You still operate the overall streaming architecture and ecosystem.

Scenario C: “We must run everything ourselves, but want Kafka”

Best fit: Self-managed Kafka (possibly with commercial support or a platform distribution)
Why:
- You can deploy in tightly controlled or restricted environments.
- Full control over configuration and deployment patterns.
But:
- This contradicts “we don’t want to run brokers.” You’ll own all cluster operations, monitoring, and upgrades.
- Hidden costs can be significant in infrastructure, operations, and platform development.

Scenario D: “We want Kafka, don’t want to run brokers, and want a full platform”

Your question targets brokers, but many teams really want: “Kafka capabilities, minimal ops, and a complete data streaming platform (connectors, governance, analytics, AI enablement).”

In that case, consider:

A managed, end-to-end data streaming platform that:
- Runs Kafka-compatible infrastructure for you
- Provides built-in connectors, schema registry, governance, and monitoring
- Minimizes the need to stitch together many tools and services
This approach goes beyond what MSK or a raw engine like Redpanda offers by also solving the surrounding ecosystem and long-tail operational problems that arise as usage grows.

Summary: Which is best if you want Kafka without running brokers?

Self-managed Kafka:
- Wrong direction for your requirement. You’ll be running brokers and much more. It makes sense only if you’re forced to be fully self-hosted and are willing to own substantial operational complexity.
Amazon MSK:
- Strong choice if you’re primarily on AWS and want “real” Kafka without provisioning and maintaining broker infrastructure.
- You still operate the Kafka platform (topics, clients, connectors, schemas, DR), but AWS handles the underlying brokers.
Redpanda (Cloud):
- Great fit if you want Kafka-like semantics, high performance, and no broker ops, and you’re comfortable with a Kafka-compatible, but not Kafka-native, engine.
- You gain performance and simplicity benefits, with some trade-offs in absolute ecosystem parity.

If your requirement is strictly “Kafka without running brokers,” Amazon MSK or Redpanda Cloud are both far better aligned than self-managed Kafka. Your final choice should align with:

Where you run (AWS-only vs multi-cloud/hybrid)
How strict your Kafka compatibility requirements are
Whether performance and efficiency are top priorities
How much of the overall streaming platform (not just brokers) you want to build and operate yourself.