How can we run builds/tests in parallel across many packages without stepping on each other (race conditions, shared outputs)?

Running builds and tests in parallel across many packages is one of the fastest ways to speed up your CI/CD pipeline—but it’s also one of the easiest ways to introduce flaky failures, race conditions, and corrupted outputs if you’re not careful. The core challenge is simple: how do you safely share compute without unintentionally sharing state?

This guide walks through patterns, tools, and architecture decisions that let you run high-concurrency builds/tests across many packages without stepping on each other, while staying maintainable as your monorepo or multi-package setup grows.

The core problem: parallelism vs. shared state

When you run builds/tests in parallel across many packages, you’re usually fighting a few classes of issues:

Shared output directories
- Multiple jobs writing to the same dist/, build/, .next/, or out/ directory
- Concurrent writes to the same compiled artifact or bundle
Global mutable state
- Shared temp directories (/tmp, C:\Temp), shared config files
- Tools writing logs or caches to fixed, global locations
Non-isolated test environments
- Shared in-memory services (e.g., global Redis, DB, or Kafka used by all tests at once)
- Ports reused across tests
Tooling that assumes serial execution
- Legacy build scripts that “clean & build” in-place
- Test harnesses that write to fixed filenames (coverage.out, report.xml)

You can run builds/tests in parallel safely if you enforce one simple design principle:

Every parallel unit of work must treat its environment as exclusive: no shared output paths and no shared mutable resources unless they are designed for concurrency.

The rest of this article is about how to engineer that principle into your pipeline.

Strategy 1: Make builds and tests package-local and deterministic

The safest baseline is to ensure that each package can be built and tested fully within its own sandbox.

1.1 Use per-package output directories

Instead of writing artifacts into shared folders, ensure each package writes within its own directory tree:

packages/app-1/dist/
packages/app-2/dist/
packages/lib-1/build/

Avoid patterns like:

Shared dist/ at the repo root
Shared generated code directories (e.g., generated/ used by all packages)

Practical approaches:

Configure your build tools to write outputs relative to package root:
- TypeScript: set outDir per package in tsconfig.json
- Webpack/Rollup: output paths under the package directory
- Go: use per-package -o paths
If a tool insists on using a global output directory, namespace the paths by package name, e.g. dist/{package-name}.

1.2 Avoid global clean steps

A common anti-pattern:

rm -rf dist
# then run parallel builds for each package that writes into dist

This creates non-deterministic failures depending on which job runs the rm -rf at what time.

Instead:

Clean per package, e.g.:
```
rm -rf packages/app-1/dist
```
Or use incremental builds with content hashes so cleaning is rarely needed.

1.3 Make test artifacts package-scoped

Test artifacts also need to be per-package, including:

Coverage reports
JUnit/XML reports
Snapshots (e.g., Jest snapshots)
Generated golden files

Avoid a single coverage/ or reports/ directory that all packages write into. Instead:

packages/app-1/coverage/
packages/app-1/junit/

This allows:

Parallel test runs for multiple packages
Aggregation later in a deterministic way (merge coverage, aggregate reports)

Strategy 2: Use process-level isolation (per-job sandboxes)

Even if your code and build config are designed to be safe, some tools or third-party components may still assume a global environment. Process-level isolation helps mitigate that.

2.1 Use separate working directories per job

In CI, avoid having all parallel jobs operate in the same checkout and writing into the same workspace.

Options:

Matrix jobs with fresh checkouts
Each job checks out the repo into its own directory.
Clone-on-write or git worktree
If disk space is a concern, use lightweight copies, but maintain logically separate workspaces per job.

Benefits:

No contention for file writes
No need to coordinate npm install or yarn install globally
Each job can cache dependencies independently

2.2 Use containers for test/build isolation

Containers make it easy to guarantee a clean, isolated environment per parallel unit.

Define a base build/test image with tools and dependencies baked in.
In your CI pipeline, run each package’s build/test in a separate container.
Mount only the package workspace (or the whole repo) into the container but keep ephemeral state inside the container.

This minimizes:

Cross-job interference via global tool configuration
Conflicts in global caches under user home directories (~/.npm, ~/.cache, etc.)

2.3 Control “global” caches carefully

Parallel jobs often share caches for performance (e.g., npm, Maven, Gradle, pnpm). This is usually safe if:

The cache is read-only during the job, or
The tooling is designed for concurrent access

To minimize risk:

Use versioned cache keys, so incompatible versions aren’t mixed.
Prefer tools known to be concurrency-safe (e.g., pnpm works better with shared store than some ad-hoc caching).
When in doubt, use per-job cache directories and aggregate them in the CI caching layer, not in the file system.

Strategy 3: Use workspace/monorepo tools built for parallel execution

If you’re working with a monorepo or many packages, dedicated tooling can handle this complexity for you.

3.1 Task runners and build orchestrators

Tools like:

Nx
Bazel
Buck
Pants
Turborepo
Lage
Rush

are designed for:

Declaring dependencies between packages
Running tasks (build/test/lint) in parallel where safe
Caching outputs and inputs
Ensuring hermetic builds (Bazel/Pants focus heavily on this)

Key features to leverage:

Per-target output directories: each task writes to its own output tree.
Input/output hashing: prevents unnecessary rebuilds and ensures repeatability.
Task graph: guarantees that dependencies build first and consumers build second, but still parallelizes independent branches.

Using one of these systems can shift the majority of the “don’t step on each other” burden from your scripts into the orchestration tool.

3.2 Workspaces in language-specific ecosystems

For JavaScript/TypeScript:

Yarn workspaces, pnpm workspaces, or npm workspaces plus a task runner like Nx or Turborepo.
These tools:
- Understand package dependency graphs
- Can run build or test across many packages with controlled parallelism
- Respect per-package package.json scripts and outputs

For other ecosystems:

Go modules: support separate modules; use go test ./... with -run or -parallel flags carefully, ensuring no shared outputs.
Maven/Gradle: multi-module builds have built-in support for parallel execution with per-module target directories.

Strategy 4: Manage concurrency in tests that use shared services

Parallel tests that hit shared external services are a major source of flakiness.

4.1 Prefer ephemeral, per-test or per-suite resources

Whenever possible, provision dedicated resources per test suite:

Databases:
- Use a per-test database schema or a separate DB instance (via Docker, Testcontainers).
- Reset state between tests instead of sharing a single long-lived DB.
Message queues / brokers:
- Use per-test topics/queues named with unique IDs.
File systems:
- Use temporary directories created via standard libraries (mkdtemp, etc.), not shared fixed paths.

4.2 Avoid hard-coded ports and shared listeners

If many parallel tests try to bind to the same port, you’ll see intermittent failures.

Best practices:

Let OS assign a free port (port 0 in many languages) and pass it to the app.
Use test harness utilities that manage free-port discovery.
If a shared service must run on a fixed port, run it once per CI job, not per test, but make sure tests themselves don’t attempt to bind to that port.

4.3 Synchronize access only where necessary

If you must share a resource (e.g., a shared integration environment):

Introduce a thin synchronization layer (mutex/lock) around operations that cannot run concurrently.
But keep these shared sections small and minimal; otherwise, you destroy parallelism.

For CI:

Prefer test partitioning: split integration tests into shards that run in completely isolated environments, rather than locking around a globally shared environment.

Strategy 5: Make scripts and tools parallel-safe

Even with good architecture, your scripting layer can reintroduce races.

5.1 Ensure scripts are re-entrant and idempotent

Each script (e.g., build, test) should:

Not assume it is the only thing running
Be safe to run multiple times in a row
Not remove or modify files outside its own output tree

Examples of what to avoid:

rm -rf node_modules in a package script that might run concurrently with npm install.
Writing logs to a global logs.txt instead of a per-package or per-run file.

5.2 Use unique filenames for temp and log files

When scripts create temp files or logs:

Incorporate a unique identifier in filenames:
- Package name
- Process ID
- A random or time-based suffix
Or use system APIs that guarantee uniqueness (mktemp, tempfile, etc.)

Examples:

LOG_FILE="logs/build-$(date +%s)-$$.log"

or within tests:

const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'my-app-test-'));

5.3 Make coverage/reporting tools aware of parallelism

Coverage tools often expect one run writing to one file. In parallel:

Write separate coverage files per test process or per package.
Use a merge step:
- nyc merge for JavaScript
- go tool covdata merge for Go
- lcov/gcov aggregation for C/C++
Configure your CI to upload merged coverage, not individual partial files only.

Strategy 6: Coordinate parallelism at the CI level

Your CI system is usually the orchestration layer for parallel builds/tests across many packages.

6.1 Use matrix jobs to shard the work

Instead of one giant job doing everything in parallel internally, use a matrix:

Each matrix entry handles a subset of packages or tests.
Example: split by package name prefix, or by dependency graph partitions.
This isolates state, reduces job-level complexity, and avoids collisions.

6.2 Limit concurrency to avoid shared resource overload

Even if everything is technically safe, you can still overload:

CPU/memory on build agents
Shared databases or queues
External APIs with rate limits

Use CI-level controls:

max-parallel or equivalent in GitHub Actions, GitLab CI, CircleCI, etc.
Separate concurrency “groups” for builds and heavier integration tests.

6.3 Use caching smartly

Caching reduces build time but can be another shared state vector.

Safer patterns:

Use content-hash-based keys:
- cache-key: build-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
Avoid using mutable directories as both cache source and runtime output simultaneously.
Clearly separate:
- Input caches: dependencies, toolchains
- Output caches: compiled artifacts (ideally managed by your build orchestrator)

Strategy 7: Design for reproducibility and determinism

When you run builds/tests in parallel across many packages, nondeterminism becomes painful to debug. Design for determinism:

Deterministic builds:
- Use pinned versions (lockfiles).
- Avoid using wall-clock time or random numbers in compiled outputs.
Deterministic tests:
- Avoid reliance on execution order.
- Ensure that seeding random number generators leads to reproducible sequences.
Stable task graphs:
- Ensure that your monorepo or workspace tool generates the same dependency graph regardless of environment or run.

Determinism makes it easier to reason about whether a failure is due to a race condition or a real bug.

Concrete patterns to avoid stepping on each other

To summarize, here are specific patterns you can adopt to run builds/tests in parallel without conflicts:

Per-package isolation
- Each package has:
  - Its own build output directory (e.g., dist/, build/ under the package).
  - Its own test artifacts directory (coverage, reports, snapshots).
Per-job workspaces
- Each CI job works in its own workspace or container, not a shared checkout.
Parallel-aware test design
- Use ephemeral resources (DBs, queues), random or OS-assigned ports, and namespaced test data.
Parallel-safe scripts
- No global rm -rf in shared directories.
- No shared log or temp filenames.
- Coverage and reports written per process, then merged.
Monorepo/task tools built for concurrency
- Leverage Nx, Bazel, Pants, Turborepo, or similar tooling that:
  - Encodes dependency graphs.
  - Manages target outputs.
  - Can run tasks in parallel safely.
CI-level control of parallelism and caching
- Use matrix builds to shard packages.
- Limit concurrency when necessary.
- Manage caches with versioned, content-hashed keys.

Example: Safe parallel builds/tests in a JavaScript monorepo

A concrete pattern for a JS/TS monorepo with many packages:

Use pnpm or yarn workspaces.
Adopt Nx or Turborepo as the task runner.
Per package:
- dist/ in packages/{name}/dist
- coverage/ and reports/ in packages/{name}/coverage|reports
CI pipeline:
1. Checkout repo in each parallel job (matrix over package subsets).
2. Restore dependency cache keyed by lockfile hash.
3. Run pnpm install or yarn install.
4. Run nx run-many --target=build --projects=${PACKAGE_SET} --parallel.
5. Run nx run-many --target=test --projects=${PACKAGE_SET} --parallel.
6. Aggregate coverage and reports.
Ensure all scripts are local to packages and do not touch shared directories.

This gives you highly parallelized builds/tests and avoids stepping on each other by construction.

Running builds and tests in parallel across many packages without stepping on each other is mostly a matter of discipline around state: make outputs and resources local, design tests and scripts to be parallel-aware, and let modern monorepo and CI tools orchestrate safely. Once those foundations are in place, you can scale concurrency aggressively without paying the price in flaky, hard-to-debug failures.

Answers you can trust, from Codeables