How can we stop rebuilding and retesting unchanged projects in CI for a big repo?

Continuous integration pipelines for large monorepos often waste huge amounts of time and compute rebuilding and retesting projects that haven’t changed. If your team is waiting 20–40 minutes on every pull request, or your CI bill is exploding, you’re probably missing one key optimization: only building and testing what’s affected by the current changes.

This guide walks through practical strategies, patterns, and tools to stop rebuilding and retesting unchanged projects in CI for a big repo, with a focus on reliability and ease of adoption.

---

Why rebuilding everything is killing your CI

In a big repo with many services, libraries, and apps, running the full build and test suite on every commit leads to:

• Long feedback loops for developers
• Flaky pipeline failures due to sheer volume of tests
• Unnecessary compute costs
• Lower deployment frequency because people fear “breaking CI”

Most of this waste comes from treating the repo as a single unit, instead of a graph of interdependent projects. To fix it, you need to make CI aware of what really changed and what depends on it.

---

Core idea: Change-based builds and tests

The central concept is change-based execution:

1. Detect what changed in the current commit or PR.
2. Map those changes to projects (apps, services, libraries) in the repo.
3. Understand the dependency graph between projects.
4. Run builds and tests only for:
   • Projects that changed, and
   • Projects that depend on those changed projects.

Everything else can be skipped, or simply verified via cached artifacts.

To get there, you need three fundamentals:

• A clear project structure
• A way to describe dependencies between projects
• A mechanism to compute “affected” projects per change

---

Step 1: Organize your big repo into discoverable projects

First ensure your repository is structured so CI can reason about projects.

Common patterns:

• Monorepo layout

  /apps
    /web
    /api
    /admin
  /libs
    /ui
    /auth
    /data-access
  /tools
  

• Service-per-directory

  /services
    /billing
    /payments
    /notifications
  /shared
    /logging
    /config
  

Best practices:

• Give each project a clear root directory.
• Store project-level config in that directory (e.g., package.json, pom.xml, build.gradle, go.mod).
• Keep shared code in dedicated libraries (libs/) rather than copying code across services.
• Define explicit boundaries: avoid “reaching into” other projects’ directories.

A clear layout makes it possible to map a changed file path back to the owning project(s).

---

Step 2: Make dependencies between projects explicit

To know what to rebuild, you must know what depends on what.

Options for modeling dependencies

• Language-level / build-tool dependencies
  • Node: dependencies / devDependencies in package.json
  • Java: Gradle/Maven inter-module dependencies
  • .NET: <ProjectReference> in .csproj
  • Go: module imports and go.mod
• Monorepo tools
  • Nx, Turborepo, Bazel, Pants, Buck, Rush, Lerna, etc.
  • These can generate and maintain a project graph.
• Manual dependency graph
  • A simple YAML/JSON file like project-graph.yml:

    web:
      dependsOn: [ui, auth]
    admin:
      dependsOn: [ui, data-access]
    api:
      dependsOn: [auth, data-access]
    

The graph is then used to compute downstream dependents when a library changes.

---

Step 3: Determine what changed in each CI run

Your CI system should identify changed files compared to a base revision:

Common patterns:

• Git diff for PRs

  git fetch origin main
  git diff --name-only origin/main...HEAD
  

• Git diff for branch builds

  git diff --name-only HEAD~1...HEAD
  

Typical approach in CI:

• For GitHub Actions, GitLab CI, Azure DevOps, Jenkins, etc., you:
  • Checkout the repo with full history or at least the base commit.
  • Run a small script that lists changed files.
  • Pass that list into your build orchestration logic.

Once you have changed files, you can:

• Map them to projects by directory prefix.
• Or let toolchains (Nx, Bazel, Turborepo, Pants) compute affected targets.

---

Step 4: Compute “affected” or “impacted” projects

Strategy A: Use a monorepo/tooling system that does it for you

For many stacks, the fastest path is to adopt a tool that has affected graph logic built-in.

Example: Nx (JS/TS, React, Node, NestJS, etc.)

Nx builds a project graph from your workspace and can run targeted commands:

• Affected apps for the current changes

  npx nx print-affected --target=build --base=origin/main --head=HEAD
  

• Run builds only for affected apps

  npx nx affected -t build --base=origin/main --head=HEAD
  

• Run tests only for affected projects

  npx nx affected -t test --base=origin/main --head=HEAD
  

Nx automatically tracks dependencies (including TS path imports), so if libs/auth changes, any app consuming it will be marked as affected.

Example: Bazel

Bazel’s entire model is incremental builds and tests:

• Define targets in BUILD files.

• Bazel tracks dependencies and caches build/test results.

• Use:

  bazel test //... --test_output=errors
  

Under the hood, Bazel will skip targets whose inputs haven’t changed, due to its content-addressed cache and dependency graph.

Example: Turborepo (Node monorepos)

Turborepo uses a task pipeline and can skip tasks whose inputs haven’t changed:

• Define tasks in turbo.json:

  {
    "pipeline": {
      "build": {
        "dependsOn": ["^build"],
        "outputs": [".next/**", "dist/**"]
      },
      "test": {
        "dependsOn": ["build"]
      }
    }
  }
  

• Use:

  npx turbo run test --filter=...[HEAD^1]
  

Or rely on its caching logic to skip unchanged packages.

Strategy B: Implement simple dependency logic yourself

If you don’t want a new tool yet, you can implement a straightforward system:

1. Create a project registry (e.g., projects.json):

   {
     "web": { "root": "apps/web", "dependsOn": ["ui", "auth"] },
     "api": { "root": "apps/api", "dependsOn": ["auth", "data-access"] },
     "ui": { "root": "libs/ui", "dependsOn": [] },
     "auth": { "root": "libs/auth", "dependsOn": [] },
     "data-access": { "root": "libs/data-access", "dependsOn": [] }
   }
   

2. Map changed files to projects

   Pseudocode:

   changed_files=$(git diff --name-only origin/main...HEAD)
   changed_projects=()
   
   for file in $changed_files; do
     for project in $(jq -r 'keys[]' projects.json); do
       root=$(jq -r ".\"$project\".root" projects.json)
       if [[ $file == $root/* ]]; then
         changed_projects+=("$project")
       fi
     done
   done
   
   changed_projects=$(printf "%s\n" "${changed_projects[@]}" | sort -u)
   

3. Compute dependents

   • Build a dependency graph in memory.
   • For each changed project, find all projects that depend on it (transitively).
   • Your final list: changed + dependents.

4. Trigger builds/tests only for those

   for project in "${final_projects[@]}"; do
     run_build_for "$project"
     run_tests_for "$project"
   done
   

This simple system can give you most of the value without heavy tooling.

---

Step 5: Introduce CI caching for unchanged artifacts

Even with great change detection, you’ll still rebuild some projects frequently. Caching lets you avoid redoing work when inputs are identical.

Types of caches to consider

• Dependency caches
  • Node modules: node_modules, ~/.npm, ~/.pnpm-store
  • Maven/Gradle caches
  • NuGet, pip, cargo, etc.
• Build cache
  • Built artifacts (JARs, DLLs, bundles, compiled outputs)
• Test cache
  • Some systems (Bazel, Nx with remote cache) can cache test results by input hash.

CI examples

GitHub Actions

- uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      **/node_modules
    key: deps-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      deps-${{ runner.os }}-

- name: Build affected projects
  run: npx nx affected -t build --base=origin/main --head=HEAD

GitLab CI

cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths:
    - .cache/
    - node_modules/

build:
  script:
    - npx nx affected -t build --base=origin/main --head=HEAD

Remote/dedicated caches

Tools like Nx, Turborepo, Bazel support remote caches (S3, Redis, proprietary services). Benefits:

• Cache is shared across CI agents and local machines.
• CI reuses artifacts created on other runners or developer machines.

---

Step 6: Wire change-detection into your CI pipelines

How you integrate this depends on your CI platform, but the workflow is similar:

1. Checkout repo and fetch base commit
2. Compute changed files and/or affected projects
3. Fail fast if the result is “none” (if you truly want to skip)
4. Otherwise run builds/tests only for affected projects

Below are some example patterns.

GitHub Actions example

name: CI

on:
  pull_request:
    branches: [ main ]
  push:
    branches: [ main ]

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Use Node
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install deps
        run: npm ci

      - name: Determine affected projects
        id: affected
        run: |
          npx nx print-affected --base=origin/main --head=HEAD > affected.json
          cat affected.json

      - name: Build affected
        run: npx nx affected -t build --base=origin/main --head=HEAD

      - name: Test affected
        run: npx nx affected -t test --base=origin/main --head=HEAD

GitLab CI example

stages:
  - build
  - test

build:
  stage: build
  script:
    - git fetch origin main
    - npx nx affected -t build --base=origin/main --head=$CI_COMMIT_SHA

test:
  stage: test
  script:
    - git fetch origin main
    - npx nx affected -t test --base=origin/main --head=$CI_COMMIT_SHA

---

Step 7: Handle global changes and edge cases

Not all changes are local to a single project. Some require rebuilding and retesting more widely.

When to rebuild everything

You might trigger “full” builds/tests if:

• Core tooling changes:
  • package-lock.json / pnpm-lock.yaml
  • tsconfig.base.json, babel.config.js
  • webpack.config.js, general build scripts
• CI scripts or base Docker images change.
• Shared runtime configurations change, like:
  • .env.example
  • config/ directory
• Dependency graph metadata changes.

Implement this via a simple rule:

if git diff --name-only origin/main...HEAD | grep -E \
  "(package-lock.json|tsconfig.base.json|tooling/|Dockerfile.base)"; then
  export FORCE_FULL_BUILD=1
fi

Then in your CI logic:

if [ "$FORCE_FULL_BUILD" = "1" ]; then
  run_full_build_and_test
else
  run_affected_only
fi

Treat non-code changes differently

• Docs-only changes should not trigger builds/tests.
• Markdown, images, or changes under /docs can skip heavy CI jobs.

Example:

changed=$(git diff --name-only origin/main...HEAD)
non_doc_changes=$(echo "$changed" | grep -vE '^docs/')

if [ -z "$non_doc_changes" ]; then
  echo "Only docs changed, skipping builds/tests."
  exit 0
fi

---

Step 8: Gradual rollout to avoid breaking everything

Moving a big repo from “build everything” to “build only changed” can feel risky. Make it incremental.

Phase 1: Observe only

• Add logging: compute affected projects, but still build everything.
• Compare:
  • Which projects were actually affected?
  • Could you have safely skipped some builds/tests?

Phase 2: Partial adoption

• Apply change-based execution only to:
  • A subset of apps/services.
  • Certain jobs (e.g., unit tests, not integration tests).
• Keep nightly or scheduled “full” CI runs for safety.

Phase 3: Full adoption with guards

• Fully switch per-PR CI to affected-only jobs.
• Maintain:
  • Nightly “build and test all” pipelines.
  • Manual “run full test suite” button for release branches.
• Add metrics and alerting for failures that might hint at missing dependencies in your graph.

---

Step 9: Make test suites more project-scoped

Skipping rebuilds helps, but if tests still load the entire world, you won’t see big gains.

To maximize impact:

• Scope unit tests to project boundaries.
• Ensure each project has its own test command:
  • npm test -- web
  • ./gradlew :billing:test
  • dotnet test Billing.Tests.csproj
• Minimize cross-project global fixtures; prefer local test data.
• For integration / E2E tests:
  • Only run suites that touch affected services by default.
  • Keep full cross-system test suites for nightly or pre-release pipelines.

---

Step 10: Monitor speed, coverage, and flakiness

After implementing change-based builds/tests:

• Track metrics:
  • Average CI time per PR before vs. after
  • Test execution time
  • Cache hit rates if available
• Watch for:
  • Surprising production issues that might hint at missing dependencies.
  • Test suites that are never triggered anymore (maybe they were misconfigured).
• Periodically review your project graph:
  • Dependencies drift over time.
  • New projects might not be listed correctly.

---

Tooling options worth considering

Depending on your tech stack, some tools make this much easier:

• Nx – Great for JS/TS monorepos, supports other languages via plugins, has robust “affected” and remote caching features.
• Turborepo – Strong for JS/TS with pipeline-based task orchestration and caching.
• Bazel – Language-agnostic, enterprise-grade incremental builds/tests, very powerful but more complex to adopt.
• Pants – Another multi-language build system with change-based execution.
• Gradle – Already has incremental build logic and build cache; use multi-module structure and --continue strategies.
• Custom scripts – Use Git diff + a homegrown project graph for smaller or less complex repos.

---

Putting it all together: A practical blueprint

To stop rebuilding and retesting unchanged projects in CI for a big repo:

1. Structure the repo into well-defined projects with clear roots.
2. Model dependencies so you know what depends on what.
3. Use Git diff to find changed files for each CI run.
4. Compute affected projects (changed + dependents).
5. Run build/test tasks only for affected projects, not the whole repo.
6. Add caching so unchanged artifacts and test results are reused.
7. Handle global changes and docs-only changes with simple rules.
8. Roll out gradually, starting in “observe-only” mode and moving to full adoption.
9. Continuously monitor and tune based on CI performance and reliability metrics.

By treating your big repo as a graph of projects instead of a single monolith, CI can scale with your codebase – giving developers faster feedback, reducing costs, and making your pipeline more resilient as the repo grows.