How do I create a working CircleCI .circleci/config.yml for a Docker-based service with tests, build, and deploy?

Most teams running a Docker-based service want one thing from their CircleCI config: a clean, repeatable pipeline that runs tests, builds an image, and deploys only when everything is green. The good news is you can get there with a single .circleci/config.yml that stays readable and easy to evolve as your stack grows.

Quick Answer: A working CircleCI .circleci/config.yml for a Docker-based service wires together three core jobs—tests, Docker build/push, and deploy—into a workflow triggered from Git. You define each job with the right executor, use workspaces or registries to pass artifacts, and gate deploy behind successful builds, approvals, and branch or tag filters.

The Quick Overview

What It Is: A CircleCI configuration file (.circleci/config.yml) that defines jobs and workflows to test, build, and deploy a Docker-based service from your Git repository.
Who It Is For: Dev, DevOps, and platform teams running containers (on Kubernetes, ECS, Nomad, or simple VM hosts) who want reliable CI/CD without hand-rolling every pipeline.
Core Problem Solved: It turns ad-hoc scripts and manual deploys into a trusted pipeline: every commit is tested, images are built and tagged consistently, and deploys are automated but controlled.

How It Works

CircleCI reads .circleci/config.yml on each push, PR, or tag, then runs the jobs and workflows you define. For a Docker-based service, you typically:

Spin up a build executor with Docker (or remote Docker) available.
Run your test suite against the service (unit, integration, or both).
Build and push a Docker image to your registry.
Deploy that image to your environment (staging, prod) behind filters and approvals.

At an operational level, a “working” config for this use case usually includes:

A test job that checks out code, restores caches, installs dependencies, and runs tests.
A build-and-push job that builds a Docker image and pushes it to Docker Hub, ECR, GCR, or another registry.
A deploy job that consumes the image tag and updates your runtime (e.g., kubectl, helm, ecs deploy, Terraform, or a custom script).
A workflows section that orders these jobs and adds controls like branch filters and manual approvals.

A Complete Example: Tests, Build, and Deploy

Below is a concrete config.yml you can adapt. This example assumes:

You have a Docker-based service (e.g., Node.js, Python, Go) with tests runnable via make test (replace as needed).
You use Docker Hub as your registry.
You deploy with a simple script (e.g., ./scripts/deploy.sh) that reads IMAGE_TAG.

Make sure your project has:

.circleci/config.yml in the repo root.
A Dockerfile at the repo root (or update the path).
A CIRCLECI_DOCKERHUB_USERNAME and CIRCLECI_DOCKERHUB_PASSWORD stored as project or org-level environment variables.
Any deployment credentials (Kubernetes kubeconfig, cloud creds) stored in CircleCI contexts or project env vars.

version: 2.1

orbs:
  # Optional: use CircleCI orbs for Docker and Kubernetes if desired
  # docker: circleci/docker@2.4.0
  # kubernetes: circleci/kubernetes@1.3.0

# Reusable executor for jobs that need Docker
executors:
  docker-executor:
    docker:
      - image: cimg/base:stable
    resource_class: medium

# Reusable environment settings
commands:
  setup-environment:
    description: "Common environment setup"
    steps:
      - checkout
      - run:
          name: Show environment info
          command: |
            echo "Branch: ${CIRCLE_BRANCH}"
            echo "Commit: ${CIRCLE_SHA1}"
            echo "Build Num: ${CIRCLE_BUILD_NUM}"

jobs:
  test:
    executor: docker-executor
    steps:
      - setup-environment
      - restore_cache:
          keys:
            # Adjust the cache key to match your language ecosystem
            - v1-deps-{{ arch }}-{{ .Branch }}-{{ checksum "package-lock.json" }}
            - v1-deps-{{ arch }}-{{ .Branch }}
            - v1-deps-
      - run:
          name: Install dependencies
          command: |
            # Example for Node.js; swap for pip, go mod, etc.
            if [ -f package-lock.json ]; then
              npm ci
            elif [ -f package.json ]; then
              npm install
            fi
      - save_cache:
          key: v1-deps-{{ arch }}-{{ .Branch }}-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
      - run:
          name: Run tests
          command: |
            # Replace with your test command
            if [ -f Makefile ]; then
              make test
            else
              npm test
            fi
      - store_test_results:
          path: test-results
          when: always
      - store_artifacts:
          path: test-results
          when: always
      - persist_to_workspace:
          root: .
          paths:
            - .

  build-and-push:
    docker:
      - image: cimg/base:stable
    environment:
      DOCKER_IMAGE_NAME: your-dockerhub-username/your-service
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Enable remote Docker
          command: |
            # For Docker-in-Docker builds; for machine executors, this is not required
            echo "Using remote Docker layer cache"
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Log in to Docker Hub
          command: |
            echo "${CIRCLECI_DOCKERHUB_PASSWORD}" | docker login -u "${CIRCLECI_DOCKERHUB_USERNAME}" --password-stdin
      - run:
          name: Build Docker image
          command: |
            IMAGE_TAG="${CIRCLE_SHA1}"
            docker build -t "${DOCKER_IMAGE_NAME}:${IMAGE_TAG}" .
            docker tag "${DOCKER_IMAGE_NAME}:${IMAGE_TAG}" "${DOCKER_IMAGE_NAME}:latest"
            echo "IMAGE_TAG=${IMAGE_TAG}" >> image.env
      - run:
          name: Push Docker image
          command: |
            source image.env
            docker push "${DOCKER_IMAGE_NAME}:${IMAGE_TAG}"
            docker push "${DOCKER_IMAGE_NAME}:latest"
      - persist_to_workspace:
          root: .
          paths:
            - image.env

  deploy:
    docker:
      - image: cimg/base:stable
    environment:
      DOCKER_IMAGE_NAME: your-dockerhub-username/your-service
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Load image tag
          command: |
            if [ -f image.env ]; then
              source image.env
              echo "Deploying image: ${DOCKER_IMAGE_NAME}:${IMAGE_TAG}"
            else
              echo "image.env not found. Failing deploy."
              exit 1
            fi
      - run:
          name: Configure deploy environment
          command: |
            # Example: configure kubectl, cloud CLI, etc.
            # export KUBECONFIG=/home/circleci/.kube/config
            echo "Configuring deployment environment..."
      - run:
          name: Deploy to environment
          command: |
            # Replace with your actual deploy commands
            # Example for Kubernetes:
            # kubectl set image deployment/your-deployment your-container=${DOCKER_IMAGE_NAME}:${IMAGE_TAG} --record
            # kubectl rollout status deployment/your-deployment
            if [ -x ./scripts/deploy.sh ]; then
              source image.env
              IMAGE="${DOCKER_IMAGE_NAME}:${IMAGE_TAG}" ./scripts/deploy.sh
            else
              echo "No deploy script found. Implement your deploy logic here."
            fi

workflows:
  version: 2
  build_test_deploy:
    jobs:
      - test:
          filters:
            branches:
              ignore:
                - main # e.g., PRs and feature branches; adjust as needed

      - test:
          name: test-main
          filters:
            branches:
              only:
                - main

      - build-and-push:
          requires:
            - test-main
          filters:
            branches:
              only:
                - main

      - deploy:
          type: approval
          name: approve-deploy-to-prod
          requires:
            - build-and-push
          filters:
            branches:
              only:
                - main

      - deploy:
          name: deploy-to-prod
          requires:
            - approve-deploy-to-prod
          filters:
            branches:
              only:
                - main

This config gives you:

Fast feedback on feature branches (test runs, but no build or deploy).
A controlled mainline pipeline:
- test-main → build-and-push → manual approval → deploy-to-prod.
An explicit image tag derived from CIRCLE_SHA1, so you can correlate code and runtime.

How It Works: Step by Step

1. Test job: keep your signal clean

The test job:

Uses a Docker executor (cimg/base:stable) to run tests in a clean environment.
Restores and saves dependency caches to avoid rebuilding everything on every run.
Runs your test command (swap make test / npm test for your stack).
Stores test results and artifacts so you can inspect failures.
Persists the workspace so the build job can reuse the checked-out code without another clone.

You can extend this with:

Service containers for dependencies (e.g., Postgres, Redis) by adding more images in the executor definition.
Parallelism and smarter test splitting with CircleCI’s Chunk/Smarter Testing to move up to 97% faster on large suites.

2. Build-and-push job: create a trusted Docker image

The build-and-push job:

Attaches the workspace, so it gets the exact code revision that passed tests.
Uses setup_remote_docker to build and push Docker images securely.
Logs into Docker Hub using environment variables stored in CircleCI.
Builds an image tagged with both the commit SHA and latest, then pushes both tags.
Writes IMAGE_TAG to image.env and persists it to the workspace for the deploy job.

You can swap Docker Hub for ECR, GCR, or another registry by changing the login and image name:

ECR/GCR: configure cloud CLIs and use docker login with those registries.
Self-hosted registry: set the DOCKER_IMAGE_NAME to your registry hostname.

3. Deploy job: ship with control, not luck

The deploy job:

Reads image.env to retrieve the exact image tag produced by build-and-push.
Configures your deploy environment (kubeconfig, cloud credentials, etc.).
Executes your deployment logic, typically via:
- kubectl / helm for Kubernetes
- aws ecs update-service for ECS
- Terraform apply
- A custom deploy.sh script that updates your environment

In the workflow, deploy is split into two jobs:

An approval job (approve-deploy-to-prod) that requires a human to click “Approve” in CircleCI. This keeps control in the loop.
The actual deployment job (deploy-to-prod) that runs once you approve.

This pattern matches what you see in teams that need enterprise-grade confidence: automated pipelines up to the edge of production, then explicit approvals and policy checks.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Separate test/build/deploy jobs	Splits the pipeline into three focused jobs in a workflow	Faster diagnosis and targeted retries when things break
Workspace + image tagging	Shares code and image metadata between jobs, tags images with commit SHA	Traceable releases: you always know what’s running where
Branch filters & approvals	Limits build/push/deploy to main (or tags) and gates prod with approvals	AI-speed delivery with guardrails and human oversight
Remote Docker with caching	Uses `setup_remote_docker` and layer caching for Docker builds	Faster, more reliable image builds at scale

Ideal Use Cases

Best for Docker microservices with PR-based flow: Because it lets you run tests on every branch, build images only from trusted branches, and deploy behind approvals.
Best for teams standardizing CI/CD across repos: Because this pattern becomes a golden path you can copy-paste and tweak (test commands, registry, deploy script) without rethinking the workflow each time.

Limitations & Considerations

Not yet using contexts or policy checks: This example uses simple env vars; for multi-team or regulated orgs, you’ll want CircleCI contexts and policy-as-code to standardize credentials, approvals, and checks before execution.
Deploy logic is project-specific: The deploy step is intentionally generic. You must plug in your actual runtime (Kubernetes, ECS, Nomad, VMs) and handle rollbacks—ideally with dedicated rollback pipelines so you can recover quickly when a release misbehaves.

Pricing & Plans

CircleCI offers a mix of free and paid plans, so you can start simple and scale up as your Docker estate grows:

Free / Starter: Best for small teams or individual services needing CI/CD with moderate concurrency and basic Docker builds.
Performance / Enterprise: Best for larger engineering orgs that need higher concurrency, standardized golden paths, org-wide policy checks, and governance features like audit logs and SSO.

For current details, limits, and enterprise options, check CircleCI’s pricing page directly from the product site.

Frequently Asked Questions

Do I need a machine executor for Docker, or is the Docker executor enough?

Short Answer: You can use the Docker executor with setup_remote_docker for most build-and-push pipelines; switch to a machine executor only for low-level Docker or privileged operations.

Details:
The config above uses:

docker:
  - image: cimg/base:stable
...
setup_remote_docker:
  docker_layer_caching: true

This works for typical docker build + docker push flows. Use a machine executor if you need:

Privileged operations (e.g., building custom kernel modules).
Running Docker daemons directly without remote Docker.
Access to specific host-level capabilities.

For most app images, the Docker executor + remote Docker is the simpler, faster choice.

How do I run integration tests that need a database or other services?

Short Answer: Add service containers to your executor and point your app/tests at them via environment variables.

Details:
In CircleCI, you can spin up service containers alongside your primary container. For example, if your Docker-based service needs Postgres:

executors:
  docker-with-postgres:
    docker:
      - image: cimg/base:stable
      - image: cimg/postgres:14.3
        environment:
          POSTGRES_USER: user
          POSTGRES_PASSWORD: pass
          POSTGRES_DB: app_test

Then in your test job, use docker-with-postgres as the executor and configure your tests to connect to localhost (or the service name, depending on how you wire it). This keeps integration tests close to production while still fitting neatly into the same config.yml.

Summary

A working CircleCI .circleci/config.yml for a Docker-based service doesn’t need to be complicated. You define three clear jobs—test, build-and-push, deploy—and wire them into a workflow that:

Runs tests on every change to protect your main branch.
Builds and pushes images from trusted branches only.
Deploys behind explicit approvals and environment filters.

From there, you can layer on smarter testing, rollback pipelines, policy checks, and AI-era debugging (via logs and job metadata) to keep shipping at AI speed without sacrificing control.

Next Step

Get Started