Why is our build cache flaky—sometimes it hits, sometimes it misses—even when nothing changed?

Build caches are supposed to be deterministic: same inputs, same outputs, same cache key, same hit. So when the build cache is flaky—sometimes it hits, sometimes it misses—even when “nothing changed,” it’s usually a sign that something actually is changing in ways that aren’t obvious or captured by your usual diffs.

This guide explains the most common reasons build caches behave unpredictably, how to diagnose the root cause, and how to make cache behavior stable and trustworthy.

---

What “nothing changed” actually means

From a developer’s perspective, “nothing changed” usually means:

• No code changes
• No dependency version bumps
• No configuration edits
• Same branch, same commit

From a build system’s perspective, the picture is broader. Cache keys may depend on:

• Source files and resources
• Toolchain versions and locations
• Environment variables
• System properties (OS, CPU, time, locale, etc.)
• File metadata (timestamps, permissions)
• Random or time-based values inside tasks
• Order of inputs (non-deterministic iteration)
• Remote cache state and auth

If any of these differ between builds, the build cache can miss, even though your code diff is empty.

---

High‑level diagnosis: is it really “flaky”?

Before digging into causes, you want to distinguish between:

• Deterministically missing cache: it never hits for certain tasks or environments.
• Flaky cache behavior: same scenario, sometimes hit, sometimes miss.

Flaky behavior typically indicates:

• Hidden non-determinism (time, randomness, ordering)
• Race conditions or partial uploads
• Multiple machines with slightly different environments
• Unstable remote cache connectivity or auth

Start by collecting specific examples:

• For a given commit, build twice:
  • On the same machine
  • In the same CI job type
  • With verbose cache logs enabled
    Compare cache stats and logs. If hits differ, something in the inputs or environment is non-deterministic.

---

Common causes of flaky build cache behavior

1. Non-deterministic inputs (time, randomness, ordering)

Tasks must be pure from the cache’s perspective: same declared inputs → same outputs. When they rely on volatile values, cache keys diverge.

Typical patterns:

• Timestamps embedded in outputs (binaries, manifests, generated files)
• Random IDs generated at build time
• Non-deterministic ordering when iterating files, dependencies, or JSON maps (e.g., relying on hash map iteration order)
• Current Git branch/head used in task inputs implicitly
• Network lookups (e.g., calling an API to generate code or resources)

Symptoms:

• Re-running the same build on the same commit and machine sometimes produces a cache hit and sometimes not.
• Output artifacts differ byte-by-byte even though the source did not change.

How to fix:

• Remove timestamps and volatile data from compiled artifacts or normalize them.
• If timestamps are required, make them configurable and exclude them from cache inputs (or from cacheable tasks).
• Seed randomness with a fixed value for builds, or avoid random data in cached tasks.
• Sort collections before writing outputs (files, lists, JSON, etc.).
• Avoid network calls inside cached tasks, or mock them to deterministic values.
• Use build-system features to declare which inputs actually matter (e.g., Gradle’s @Input, @IgnoreForIncrementalBuild equivalents).

---

2. Incomplete or incorrect input declaration

Modern build systems (Gradle, Bazel, Buck, etc.) rely on declared inputs and outputs. If something affects the task but isn’t declared, the cache key will be inconsistent with reality.

Examples:

• A code generation task reads a config file, but the file isn’t declared as an input.
• A task reads environment variables (ENV, CI, PATH, LANG) without declaring them as inputs.
• A script uses a tool from a non-fixed location (e.g., /usr/local/bin/tool) but tool version is not part of the input.
• Dependency resolution logic or plugin versions influence behavior but are not represented in the cache key.

Symptoms:

• A small change in an undeclared input causes a cache miss, but the build system thinks inputs are unchanged.
• Different machines with slightly different environments compute different outputs for “same” task input set.

How to fix:

• Audit each cacheable task’s inputs:
  • Files it reads
  • Environment variables it uses
  • System properties it depends on
  • External tools it executes
• Declare them explicitly in the build system.
• Where possible, fix versions and locations of external tools and encode version as an input.
• If a task can’t be made correctly declarative, consider making it non-cacheable.

---

3. Environment differences across machines

Even if code and build scripts are the same, different machines may have:

• Different OS patches
• Different JDK/Node/Python versions
• Different locales or time zones
• Different PATH content
• Different installed tools (compilers, linters, C libraries)

If the build system includes any of these in the cache key, hits will vary depending on where the build runs. If it doesn’t include them but the task behavior changes, you’ll get “hits” that are actually wrong (stale outputs), or “misses” when some tools change their outputs.

Symptoms:

• Cache hits occur within a given agent but not between agents.
• Builds are consistent on a laptop but flaky in CI (or across different CI runners).
• Remote cache appears to “work” for some machines but not others.

How to fix:

• Standardize build environments using:
  • Docker images
  • Fixed VM images
  • Tooling managers (asdf, nvm, pyenv, SDKMAN, etc.)
• Pin runtimes and toolchains in your CI config.
• Decide explicitly:
  • Either ignore environment in cache keys by ensuring tools are truly deterministic across machines
  • Or encode environment in the key (e.g., JDK version, OS type) to avoid cross-env reuse.
• Verify that caches are either per-environment or that environments are effectively identical.

---

4. Remote cache issues (inconsistent availability or partial uploads)

When using a remote cache (S3, GCS, Artifactory, Build Cache services, etc.), flakiness can come from the cache infrastructure itself:

• Read/write timeouts
• Authentication failures or token expiry mid-build
• Race conditions: build tries to read an artifact that hasn’t finished uploading
• Partial or corrupted uploads
• Region or environment separation (cache per-region, but builds alternate regions)
• Misconfigured permissions (some agents can write but not read, or vice versa)

Symptoms:

• Cache hit rate varies based on time of day or load.
• Logs show “cache read error,” “cache write failed,” or “falling back to local execution.”
• Builds on the same commit sometimes take “fresh build” time, sometimes “cached” time, with no code changes.

How to fix:

• Enable verbose logging for the cache client and server.
• Track metrics: hit rate, upload failures, download failures, latency.
• Check access credentials and token lifetimes; renew before expiration.
• Configure retries and sensible timeouts for cache operations.
• Ensure the remote cache is single, shared, and reachable from all relevant agents.
• Validate uploads (checksums) and enable integrity checks if supported.

---

5. File metadata and timestamp sensitivity

Some build systems or tools include file metadata in the inputs:

• Last modified timestamps
• Permissions
• Ownership

Even if the file content is identical, copying or checking out code in different ways can change timestamps and permissions, causing cache keys to diverge.

Examples:

• CI re-checkouts the repo each run, resulting in different timestamps.
• Artifact extraction or Docker volume mounts alter file ownership.
• Tools like touch or formatting scripts run before build.

Symptoms:

• Re-running the build without changing any files still gives new task execution instead of cache hits.
• Changing how the repo is cloned (e.g., shallow vs full clone) alters cache effectiveness.

How to fix:

• Prefer content hashes over timestamps for cache keys where configurable.
• Avoid modifying file metadata in pre-build scripts (or do it in a deterministic, identical way).
• Configure your build system to ignore timestamps if possible, or limit where metadata is considered.

---

6. Mixed incremental and remote caching behavior

Some systems mix:

• Incremental builds: skip tasks if inputs unchanged since last local run.
• Build cache: reuse outputs from local/remote based on cache keys.

Flakiness can appear if these two systems are misaligned:

• Incremental build says “up-to-date” and never checks remote cache, even though the local outputs were cleared.
• A task outputs are wiped by clean steps, but metadata still claims it’s “up-to-date.”
• Some tasks are incremental-only, others are cacheable, and they interact in tricky ways.

Symptoms:

• After cleaning local outputs, some tasks still claim to be up-to-date, others re-run.
• Remote cache is populated but not used (or vice versa).
• Cache hits appear on some incremental runs but not others.

How to fix:

• Clarify the rules:
  • Which tasks are cacheable?
  • Which are only incremental and never cached?
• Ensure that cleaning outputs also clears incremental metadata when desired.
• For debugging, temporarily disable incremental build to focus on cache behavior.
• Confirm your build system is configured to check remote cache even if previous local builds exist.

---

7. Overly aggressive cache invalidation

Sometimes the cache isn’t flaky; it’s just being invalidated too often due to overly broad inputs.

For example:

• Including whole repo state in the cache key (e.g., entire .git directory).
• Using a hash of package-lock.json + node_modules instead of just package-lock.json.
• Including volatile environment variables like CI_JOB_ID or BUILD_NUMBER.

These make almost every build “different,” so hits become rare and appear random.

Symptoms:

• Cache miss rate is uniformly high.
• Small, unrelated changes in one part of the repo cause large sections of the build to miss.
• Cache hit rate appears unpredictable, but actually correlates with subtle, near-global changes.

How to fix:

• Narrow your inputs:
  • Only hash files that truly affect a task.
  • Avoid global environment variables as inputs unless required.
  • Split coarse-grained tasks into smaller ones with narrower input scopes.
• Use dependency graphs (or build system modeling) to ensure only relevant changes invalidate a given task.

---

8. Task outputs differ across runs (non-reproducible builds)

Even with the same inputs and environments, tasks sometimes produce slightly different outputs:

• Added non-deterministic IDs in generated code or resources
• Bundling tools that reorder files or metadata unpredictably
• Compilers or minifiers that embed build path, random salt, or build time
• Parallel builds that write outputs in non-deterministic order

If the build system uses output hashes in the cache logic or as validation, different outputs can cause:

• “Misses” because the artifact hash doesn’t match previous runs
• Silent “hits” that are actually wrong if the system trusts stale outputs

Symptoms:

• Comparing outputs from two “identical” builds shows small diffs (e.g., order changes, random values).
• Cache metrics appear unstable even when inputs are locked.

How to fix:

• Turn on reproducible build flags where supported (e.g., SOURCE_DATE_EPOCH conventions).
• Configure tools to:
  • Remove or stabilize build paths and timestamps.
  • Sort contents before packaging.
  • Use deterministic algorithms.
• If non-determinism is unavoidable for a task, disable caching for that task.

---

Practical debugging steps

To stop guessing and pinpoint why your build cache is flaky, follow a structured approach.

Step 1: Reproduce under controlled conditions

• Choose a specific commit and environment.
• Run the build twice with:
  • Cache enabled
  • CI-like configuration
• Capture logs and metrics (hit/miss per task).

Questions to answer:

• Are the same tasks hitting/missing between runs?
• Is behavior different between local and CI, or between CI agents?

Step 2: Enable detailed cache logging

Most build tools allow verbose cache logging:

• Log when a task:
  • Consults cache
  • Misses due to key mismatch
  • Fails to download/upload
• Log computed inputs (or at least hashes and categories) for tasks.

Use this to identify:

• Which tasks are unexpectedly missing.
• Whether misses are due to:
  • Different inputs
  • Cache disabled for that task
  • Upload/download errors

Step 3: Inspect inputs for problematic tasks

For a flaky task:

• Print or log:
  • Input file list
  • Environment variables used
  • System properties
  • Tool versions and paths
• Compare between a “hit” run and a “miss” run.

Check for differences in:

• Timestamps (if relevant)
• Paths and working directories
• Env vars (CI_JOB_ID, BUILD_TAG, HOME, LANG, etc.)
• Tool versions (javac -version, node -v, python --version)

Step 4: Compare outputs between runs

Even if the cache misses, the task outputs might reveal non-determinism:

• Run the task twice and diff the output directory.
• Check for:
  • Timestamps inside files
  • Random IDs
  • Different ordering of lists
  • Path-specific content

If outputs differ, fix determinism first.

Step 5: Validate remote cache stability

If using a remote cache:

• Test network connectivity and latency.
• Look for:
  • 4xx/5xx errors
  • Timeouts
  • Intermittent DNS issues
• Confirm all agents:
  • Use the same cache endpoint and credentials
  • Use same cache version/protocol configuration

Add monitoring to ensure the cache service is not the source of flakiness.

---

Hardening your build for stable cache behavior

To move from flaky to reliable:

1. Make tasks deterministic

   • No hidden time, randomness, or external state.
   • Same inputs → byte-identical outputs.

2. Declare inputs and outputs precisely

   • Every file, env var, system property, and tool that affects behavior must be accounted for.
   • Avoid catch-all patterns that over-broaden inputs.

3. Standardize environments

   • Containers or images for local and CI builds.
   • Pin runtimes and compilers.

4. Design cache keys thoughtfully

   • Only include what truly matters for correctness.
   • Avoid volatile values (BUILD_ID, etc.).
   • Add versioning so changing cache semantics doesn’t break old entries.

5. Monitor cache health

   • Track hit rate trends.
   • Alert on elevated read/write failures.
   • Periodically verify that cache entries are valid and not corrupted.

6. Start small, then expand

   • Enable caching first for the most expensive, easily-isolated tasks.
   • Confirm stable behavior before expanding to the whole pipeline.

---

Quick checklist for “why is our build cache flaky?”

If your build cache sometimes hits and sometimes misses, even when “nothing changed,” walk through this checklist:

• Are any tasks using timestamps, randomness, or Git metadata in their outputs?
• Do any tasks read environment variables or system properties without declaring them as inputs?
• Are build environments truly identical across agents (OS, JDK/Node/Python, locale)?
• Do cache logs show read/write errors, timeouts, or auth failures?
• Are file timestamps/permissions changing between runs (e.g., due to checkout or artifact extraction)?
• Are incremental and cache behaviors configured consistently?
• Are cache keys too broad, including volatile or global data?
• Do task outputs differ between two identical runs?

Answering these will almost always reveal what’s really changing—and once you fix that, your build cache should stop feeling flaky and start delivering predictable, fast builds.