Application Observability
How can I figure out which service is actually causing a latency spike in Kubernetes when everything looks healthy at the node level?
9 min read
When Kubernetes latency spikes but node metrics look perfectly healthy, you’re usually dealing with a service-level problem that traditional infrastructure monitoring can’t see. CPU, memory, and node-level disk or network can all be “green,” while a single microservice, dependency, or deployment change quietly drags your entire user journey.
In modern clusters, you won’t find that root cause by eyeballing dashboards or chasing correlated graphs. You need three things working together:
- End-to-end trace visibility for every request
- Real-time topology that understands service dependencies
- Causation-based AI that can identify the one service that is actually responsible
Below, I’ll rank three practical approaches to figuring out which service is causing a Kubernetes latency spike when node-level health looks fine.
Quick Answer: The best overall choice for fast, reliable root-cause detection in Kubernetes latency spikes is Dynatrace full-stack with OneAgent and Davis® AI. If your priority is staying within an existing open-source stack and you have strong SRE capacity, OpenTelemetry + Prometheus/Grafana is often a stronger fit. For teams that want tracing but are comfortable with manual RCA and sampling-based trade-offs, consider Jaeger or Zipkin-based tracing.
At-a-Glance Comparison
| Rank | Option | Best For | Primary Strength | Watch Out For |
|---|---|---|---|---|
| 1 | Dynatrace full-stack (OneAgent + Davis® AI) | Enterprises that need precise root-cause answers across Kubernetes, services, and deployments | Automatic discovery, end-to-end topology, and causation-based AI that pinpoints the true culprit | Commercial platform; requires adopting the unified approach rather than isolated tools |
| 2 | OpenTelemetry + Prometheus/Grafana stack | Teams with strong observability engineering that want open-source flexibility | Rich metrics and traces with full control over pipelines and instrumentation | Manual correlation, custom dashboards, and high operational overhead for RCA |
| 3 | Jaeger / Zipkin tracing alone | Teams focused primarily on request tracing within a single or limited set of services | Deep visibility into request paths and latency per span | No unified topology, limited infra context, correlation and root cause stay manual |
Comparison Criteria
We evaluated each option against three criteria that matter when you’re trying to isolate a misbehaving service in a Kubernetes latency spike:
- Root-cause precision: How reliably can the approach identify the actual service and change (deployment, config, dependency) that caused the spike, not just correlated symptoms?
- Coverage and context: How completely does it see the environment—metrics, logs, traces, user experience, Kubernetes entities, and external dependencies—and link them via real-time topology?
- Operational overhead: How much manual work does your team need to invest—instrumentation, dashboard building, correlation during incidents—to get actionable answers instead of just more data?
Detailed Breakdown
1. Dynatrace full-stack (Best overall for precise, low-effort root-cause answers)
Dynatrace full-stack observability with OneAgent and Davis® AI ranks as the top choice because it automatically discovers every service in your Kubernetes environment, builds a real-time topology of their dependencies, and uses causation-based AI to tell you which service—and often which deployment or change—actually caused the latency spike.
In modern, highly dynamic microservice environments, infrastructure and services spin up and disappear in milliseconds. A disappearing pod can be perfectly healthy autoscaling—or the first sign of a cascading failure. Human operators simply can’t track this in real time across millions of dependencies. Dynatrace is designed to do exactly that.
What it does well:
-
Causation-based root cause, not just correlation:
Davis® AI doesn’t just see that several services are slow; it analyzes all transactions and entity interdependencies to determine what is broken (technical root cause) and why it is broken (foundational root cause).
When a latency spike occurs, Davis follows every transaction through the real-time topology, ranks all contributing anomalies, and identifies the service with the most negative impact. It can directly link that spike to a specific deployment, configuration change, or downstream dependency. -
Full-stack visibility with automatic coverage:
OneAgent automatically discovers and instruments your Kubernetes nodes, pods, containers, and services—no manual code changes or per-service setup.
You see:- Service-level latency, error rates, and throughput
- Distributed traces across microservices
- Kubernetes entities (nodes, pods, namespaces, workloads)
- Infrastructure metrics plus logs, user experience, and security events
Dynatrace builds a live topology graph that connects all of these entities in context, from the end user through services down into the underlying infrastructure and cloud platforms.
-
Answers that drive automated action:
Because the platform can pinpoint the true root cause, it doesn’t just raise alerts; it gives you answers you can act on automatically.- Trigger Workflows to roll back a specific deployment when it’s identified as the foundational root cause
- Open tickets in ITSM systems with the precise failing service and change already attached
- Protect SLOs by alerting only on root problems instead of triggering an alert storm across every affected service
Tradeoffs & Limitations:
- Unified platform adoption vs. point tools:
To get the full value—automatic discovery, topology, causation-based AI, and workflows—you adopt Dynatrace as a unified observability and security platform. It’s not a single-purpose, logs-only or metrics-only tool. For some teams, that means evolving beyond a patchwork of separate open-source components and dashboards.
Decision Trigger: Choose Dynatrace full-stack with OneAgent and Davis® AI if you want fast, explainable answers to “which service is actually causing this Kubernetes latency spike?” and you prioritize precise, automated root-cause detection over manually correlating metrics and traces.
2. OpenTelemetry + Prometheus/Grafana (Best for open-source flexibility and DIY observability)
An OpenTelemetry + Prometheus/Grafana stack is the strongest fit if you want open-source control and already have (or plan to build) a capable observability engineering function. You can absolutely figure out which service is causing a latency spike—but you’ll do more of the work yourself.
What it does well:
-
Rich, flexible metrics and traces:
OpenTelemetry (OTel) gives you vendor-neutral instrumentation for traces, metrics, and logs, while Prometheus is a battle-tested metrics backend and Grafana provides powerful visualizations. Properly configured, this stack can show:- Per-service latency, error rates, and saturation
- Request paths through microservices using distributed traces
- Time-correlated Kubernetes resource metrics (CPU, memory, etc.)
You can create dashboards that highlight which service latency is increasing first, and traces to see the slow span in the call chain.
-
Custom pipelines and governance:
You can tailor data retention, sampling rates, label cardinality, and routing rules. For organizations that treat observability as a first-class engineering discipline, this allows fine-grained control over cost, data quality, and internal standards.
Tradeoffs & Limitations:
-
Manual correlation and RCA effort:
The tools show you data; they don’t natively provide causation-based root cause. During a latency spike, you typically:- Notice an SLO or dashboard breach
- Drill into request traces
- Manually compare which services show latency first
- Cross-check deployment histories, config changes, and pod events
This can work well, but it relies on human pattern recognition and practice. In large microservice estates, this can quickly become a war room exercise—exactly what many teams are trying to move beyond.
-
Operational overhead and complexity:
Maintaining Prometheus at scale, managing OTel collectors, controlling cardinality, and keeping dashboards aligned with rapidly changing services becomes a significant engineering effort. As Kubernetes environments grow more dynamic, the overhead of keeping everything in sync increases.
Decision Trigger: Choose OpenTelemetry + Prometheus/Grafana if you want maximum openness and customization, your team is comfortable owning observability as an internal product, and you’re prepared to do the heavy lifting to track down the service causing latency spikes.
3. Jaeger / Zipkin-based tracing (Best for trace-centric debugging in focused environments)
Jaeger or Zipkin-based tracing stands out when your primary need is to understand request flows and span-level latency across a (relatively) constrained set of services. These tools can show exactly which span in a trace is slow, which is essential to figuring out where time is being spent.
What it does well:
-
Detailed request-path visibility:
For each user or API request, you can:- See the full span tree as it traverses services
- Identify which service or operation introduced the bulk of latency
- Compare traces before and during an incident
This is particularly useful in debugging complex call graphs, e.g., a gateway calling multiple downstream services, where one dependency becomes slow.
-
Developer-friendly diagnostics:
Engineers can use traces to understand the internal behavior of their services, optimize code paths, and confirm whether a new feature or integration is adding overhead.
Tradeoffs & Limitations:
-
Limited topology and context:
Tracing alone doesn’t give you:- A full real-time topology across all Kubernetes entities and external dependencies
- A unified view of metrics, logs, and user experience
- An AI that ranks contributors and identifies a single root cause
You’ll spend time pivoting between the tracing UI, Kubernetes dashboards, deployment histories, and logs to form a complete picture.
-
Sampling and blind spots:
To keep costs manageable, many teams sample traces. During a short-lived latency spike, the critical traces you need may not be captured. Without continuous, full-coverage instrumentation and a unified data lakehouse, important signals can be lost or become hard to correlate.
Decision Trigger: Choose Jaeger or Zipkin-based tracing if you’re primarily focused on trace-centric debugging in a known set of services, you’re comfortable handling the rest of the context manually, and you accept that root-cause detection will remain a hands-on, human-driven task.
Final Verdict
When Kubernetes latency spikes while node-level health still looks normal, the culprit almost always sits higher in the stack: a specific service, a deployment, a changed dependency, or a noisy neighbor pattern that manifests at the service layer before it saturates the node. Traditional infrastructure metrics won’t tell you which service is responsible, or why it started misbehaving.
All three approaches above can help:
-
Dynatrace full-stack with OneAgent and Davis® AI is the best fit if you want precise, real-time answers to “which service is actually causing this?” without assembling them manually from charts and traces. It automatically discovers your services, maps their dependencies, monitors metrics, logs, traces, UX, and security in context, and uses causation-based AI to identify the technical and foundational root cause—exactly what you need to move from reactive war rooms to preventive, autonomous operations.
-
OpenTelemetry + Prometheus/Grafana is a strong option if you value open-source tooling, have the engineering capacity to maintain a complex observability stack, and are comfortable that incident resolution will hinge on expert humans correlating signals.
-
Jaeger/Zipkin-based tracing is best suited to environments where tracing is the primary need, and where you accept the trade-off of manual root-cause analysis and limited holistic context.
If your goal is to consistently and quickly figure out which service is actually causing a latency spike in Kubernetes—even when everything looks fine at the node level—the most robust and scalable path is to unify observability and let deterministic, causation-based AI do the heavy lifting for you.