Fume setup: what do I need to connect our app and run tests in CI (GitHub Actions)?

Connecting Fume to your app and wiring it into GitHub Actions CI is straightforward once you understand the moving parts: how Fume authenticates, how tests are triggered, and how to manage secrets safely in your pipeline. This guide walks through what you need, how to set it up, and how to run Fume tests automatically on every push or pull request.

What you need before setting up Fume in CI

To connect your app and run Fume tests in GitHub Actions, you typically need:

A Fume account and project
- Access to the Fume dashboard
- A project configured for your app / test suite
API credentials or CI token
- An API key, CI token, or similar credential generated in Fume
Your test configuration
- Fume test suites, flows, or scenarios already defined
- Any project-specific configuration (environment URLs, test tags, etc.)
Access to your GitHub repository
- Permissions to create and edit GitHub Actions workflows (.github/workflows/*.yml)
- Admin or maintainer access to add repository secrets

Once these are in place, you can wire Fume into your CI pipeline.

Step 1: Create or locate your Fume project

If you haven’t already:

Log into Fume and create a new project for your application.
Configure environments (e.g., staging, production, preview) with the target URLs where your app will be reachable during CI runs.
Define tests:
- Create Fume test flows / suites (UI flows, API checks, or whatever Fume supports for your stack).
- Group tests into suites or collections you can call from CI (e.g., smoke, regression, pull-request).

Note the following details from the Fume project:

Project or workspace ID
Suite IDs, test IDs, or tags you want CI to run
Any environment names (e.g., staging-ci) that you’ll reference from GitHub Actions

You’ll use these identifiers in your workflow configuration.

Step 2: Generate a Fume CI token or API key

CI will authenticate with Fume using a non-interactive credential. In the Fume dashboard:

Go to Settings → API / Integrations / Tokens (exact wording depends on Fume’s UI).
Create a CI token / API key:
- Scope it to the minimum permissions needed (typically “run tests” or “trigger suites”).
- Optionally restrict it to a single project or environment for safety.
Copy the token value; you’ll store it as a GitHub secret.

For security, never commit this token to your repository.

Step 3: Add Fume secrets to GitHub

In your GitHub repository:

Navigate to Settings → Secrets and variables → Actions → New repository secret.
Add secrets such as:
- FUME_API_TOKEN – your CI token or API key
- FUME_PROJECT_ID – if Fume requires a project ID
- FUME_ENVIRONMENT – optional, e.g., staging-ci
Save each secret. GitHub will mask these values in logs.

If you use organization-level secrets (for multiple repos), add them at the organization → Actions secrets level instead.

Step 4: Decide when Fume tests should run in CI

Before writing the workflow, decide:

Triggers:
- On every push to main or develop?
- On pull requests only?
- On a nightly schedule (cron)?
Scope:
- Full regression every night / on main
- Fast smoke tests on every PR
Blocking vs non-blocking:
- Should a Fume test failure fail the CI job and block merging?
- Or just post results while allowing the pipeline to pass?

These decisions shape your GitHub Actions workflow structure.

Step 5: Add a basic Fume GitHub Actions workflow

Create a workflow file in your repo:

mkdir -p .github/workflows
touch .github/workflows/fume-ci.yml

Example minimal workflow to run on pull requests:

name: Fume CI Tests

on:
  pull_request:
    branches:
      - main
      - develop

jobs:
  fume-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      # Optional: build or start your app here
      # - name: Install dependencies
      #   run: npm install
      #
      # - name: Start app
      #   run: npm run start &
      #
      # - name: Wait for app
      #   run: npx wait-on http://localhost:3000

      - name: Run Fume tests
        env:
          FUME_API_TOKEN: ${{ secrets.FUME_API_TOKEN }}
          FUME_PROJECT_ID: ${{ secrets.FUME_PROJECT_ID }}
          FUME_ENVIRONMENT: ${{ secrets.FUME_ENVIRONMENT }}
        run: |
          # Replace this block with the actual Fume CLI or API call.
          # Example CLI usage:
          # npx fume run \
          #   --project $FUME_PROJECT_ID \
          #   --env $FUME_ENVIRONMENT \
          #   --suite "pull-request"
          echo "Trigger Fume tests here."

Adjust the final command to match Fume’s official CLI or API syntax.

Step 6: Use Fume’s CLI or API in the workflow

Fume usually exposes one or both of:

A CLI (installed via npm, pip, curl, etc.)
A REST API you call via curl or a small script

Example: Using a Fume CLI

If Fume provides a Node-based CLI:

      - name: Install Fume CLI
        run: npm install -g @fume/cli

      - name: Run Fume suite
        env:
          FUME_API_TOKEN: ${{ secrets.FUME_API_TOKEN }}
        run: |
          fume login --api-token "$FUME_API_TOKEN"
          fume run suite \
            --project "$FUME_PROJECT_ID" \
            --env "$FUME_ENVIRONMENT" \
            --name "pull-request"

Adjust flags (--suite, --tags, --project) to match Fume’s docs.

Example: Using the Fume API via curl

If you’re calling an HTTP endpoint:

      - name: Trigger Fume tests via API
        env:
          FUME_API_TOKEN: ${{ secrets.FUME_API_TOKEN }}
          FUME_PROJECT_ID: ${{ secrets.FUME_PROJECT_ID }}
        run: |
          TRIGGER_RESPONSE=$(curl -sS -X POST "https://api.fume.dev/v1/projects/$FUME_PROJECT_ID/run" \
            -H "Authorization: Bearer $FUME_API_TOKEN" \
            -H "Content-Type: application/json" \
            -d '{
              "suite": "pull-request",
              "environment": "'"$FUME_ENVIRONMENT"'"
            }')

          echo "Fume trigger response: $TRIGGER_RESPONSE"

          # Optionally parse a run ID and poll until completion
          # RUN_ID=$(echo "$TRIGGER_RESPONSE" | jq -r '.runId')
          # ...poll for status and exit non-zero on failure

Replace the URL and JSON structure with the ones Fume documents.

Step 7: Make Fume tests block or not block CI

To make Fume tests block the pipeline:

Ensure the Fume CLI or script exits with a non-zero code on test failures.
GitHub Actions will automatically mark the job as failed.

Example:

      - name: Run Fume suite (blocking)
        env:
          FUME_API_TOKEN: ${{ secrets.FUME_API_TOKEN }}
        run: |
          set -e
          fume run suite --project "$FUME_PROJECT_ID" --env "$FUME_ENVIRONMENT" --name "pull-request"

To make Fume tests non-blocking (report-only):

      - name: Run Fume suite (non-blocking)
        continue-on-error: true
        env:
          FUME_API_TOKEN: ${{ secrets.FUME_API_TOKEN }}
        run: |
          fume run suite --project "$FUME_PROJECT_ID" --env "$FUME_ENVIRONMENT" --name "pull-request"

This lets PRs merge even if Fume detects issues, which can be useful early in adoption.

Step 8: Integrate Fume results into PRs

For a smoother developer experience, you can:

Post a summary comment on the PR
Upload Fume reports as artifacts
Fail with a clear message linking to Fume test results

Example: posting a simple status comment using actions/github-script:

      - name: Post Fume status to PR
        if: always()
        uses: actions/github-script@v7
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const conclusion = "${{ job.status }}";
            const body = `Fume tests finished with status: **${conclusion}**.\n\nSee Fume dashboard for detailed results.`;
            const { context } = require('@actions/github');
            if (context.payload.pull_request) {
              await github.rest.issues.createComment({
                ...context.repo,
                issue_number: context.payload.pull_request.number,
                body,
              });
            }

You can expand this with links to specific runs if Fume’s API returns URLs.

Step 9: Handle environments and app availability

Fume tests typically need your app to be reachable. Common approaches:

Use an existing environment
Point Fume at a shared staging environment (https://staging.example.com).
Spin up the app in CI
- Start a local server in GitHub Actions
- Expose it via a tunneling service or Fume’s own agent (if available)
- Configure Fume to target that URL

Example of starting the app before running tests:

      - name: Install dependencies
        run: npm ci

      - name: Start app
        run: npm run start &
      
      - name: Wait for app to be ready
        run: npx wait-on http://localhost:3000

      - name: Run Fume tests
        env:
          FUME_API_TOKEN: ${{ secrets.FUME_API_TOKEN }}
        run: |
          fume run suite \
            --project "$FUME_PROJECT_ID" \
            --env "$FUME_ENVIRONMENT" \
            --base-url "http://localhost:3000" \
            --name "pull-request"

Check whether Fume supports a --base-url or equivalent flag.

Step 10: Troubleshooting common Fume setup issues in GitHub Actions

When connecting your app and running Fume tests in CI, you might run into:

Authentication errors
- Verify the token in Fume is active and has the correct scope.
- Confirm secrets are set on the same repository where the workflow runs.
- Check the environment variable names match exactly (case-sensitive).
Network / environment issues
- Make sure the environment URL you pass to Fume is reachable from GitHub’s runners.
- If using localhost, ensure the Fume agent or tunnel is correctly configured.
Tests not running or incomplete
- Confirm project/suite IDs used in the workflow exist in Fume.
- Log the response from Fume’s API to confirm the run is triggered.
- If polling for completion, make sure your script waits long enough and handles timeouts.
Flaky tests in CI
- Add retries in Fume if supported.
- Stabilize test data and environment configuration (isolated users, clean DB snapshots, etc.).
- Run a smaller “smoke” suite on each PR and reserve heavier suites for scheduled runs.

Recommended baseline setup for most teams

If you want a simple, practical starting point for Fume setup with GitHub Actions:

Create a pull-request Fume suite with fast, high-signal tests.
Generate a Fume CI token scoped to that project.
Add FUME_API_TOKEN, FUME_PROJECT_ID, and FUME_ENVIRONMENT as GitHub secrets.
Create a fume-ci.yml workflow that:
- Runs on pull_request to main/develop
- Builds and starts your app (or targets staging)
- Installs the Fume CLI
- Runs the pull-request suite and fails the job on test failure
Optionally add:
- A nightly workflow that runs a larger regression suite
- PR comments with a summary of Fume results

With this setup, Fume is fully connected to your app and integrated into your GitHub Actions CI, giving your team automated test coverage on every change with minimal manual overhead.