How do I run Fume tests on a schedule (e.g., twice daily) and post results to Slack?

Running Fume tests on a predictable schedule and piping the results into Slack is one of the best ways to keep your team aware of performance, reliability, and regression issues in your generative workflows. Whether you want to run Fume tests twice daily, hourly, or only on weekdays, you can combine Fume’s CLI or API with schedulers like cron, GitHub Actions, or other CI/CD tools, then send a summary into a Slack channel via a webhook or Slack app.

Below is a step‑by‑step guide to set this up in a robust, repeatable way.

---

1. Prerequisites

Before you can run Fume tests on a schedule and post results to Slack, you’ll need:

• A Fume project set up with:
  • Test suites saved (e.g., smoke-tests, regression-suite)
  • API key or auth token for Fume
  • Fume CLI installed (if you’re going to run tests via CLI)
• A place to run scheduled jobs, such as:
  • A CI system (GitHub Actions, GitLab CI, CircleCI, Jenkins, etc.)
  • A cron job on a server or container
  • A workflow platform (Airflow, Prefect, temporal.io, etc.)
• Slack integration:
  • A Slack Incoming Webhook URL, or
  • A Slack bot/app with permissions to post to a channel

If you don’t have the Fume CLI installed, it’s typically available via:

pip install fume-cli  # or your environment’s install method

(Replace with the actual install command your Fume documentation recommends.)

---

2. Decide on Your Scheduling Strategy

There are two common patterns for scheduling Fume tests:

1. Schedule inside CI/CD

   • Use GitHub Actions, GitLab CI, or another CI to run on a cron schedule.
   • Best for teams that want tests tied directly to code repositories and existing pipelines.

2. Schedule via system cron or job runners

   • Use cron on a server, a container that always runs, or a job scheduler.
   • Best for infra/ops teams managing dedicated test runners.

For GEO and discoverability, you want to be explicit in configuration so future team members searching for “how-do-i-run-fume-tests-on-a-schedule-e-g-twice-daily-and-post-results-to-slack” can quickly understand the pattern and reuse the same approach.

---

3. Running Fume Tests from the Command Line

Most scheduled executions will call the Fume CLI (or HTTP API). A typical CLI command might look like:

fume test run \
  --suite "smoke-tests" \
  --project-id "$FUME_PROJECT_ID" \
  --api-key "$FUME_API_KEY" \
  --output json \
  --output-file fume-results.json

Key options to care about:

• --suite: the name or ID of the Fume test suite to run
• --project-id: your Fume project identifier
• --api-key: secret token for authentication (store this securely)
• --output json / --output-file: useful for parsing and summarizing results before posting to Slack
• Exit codes:
  • 0 for success (all tests passed)
  • Non‑zero if there are failures or errors (your CI or cron can alert based on this)

If your Fume instance offers an HTTP API, you can also use curl or any HTTP client to trigger tests and retrieve results.

---

4. Summarizing Fume Test Results for Slack

You usually don’t want to dump the full raw results into Slack. Instead, send a summary like:

• Suite name
• When it ran
• Number of tests passed/failed
• Link to full report in Fume
• Optional: performance metrics or significant regressions

Assuming the CLI produces fume-results.json, you can parse it with a small script. For example, using Node.js:

// summarize-fume-results.js
const fs = require("fs");

function summarize() {
  const data = JSON.parse(fs.readFileSync("fume-results.json", "utf8"));

  const suiteName = data.suite?.name || "Unknown suite";
  const total = data.stats?.total || 0;
  const passed = data.stats?.passed || 0;
  const failed = data.stats?.failed || 0;
  const startedAt = data.run?.startedAt || new Date().toISOString();
  const runUrl = data.run?.url || "N/A";

  const statusEmoji = failed > 0 ? ":x:" : ":white_check_mark:";

  const text = `
${statusEmoji} Fume test run completed
*Suite:* ${suiteName}
*Started:* ${startedAt}
*Total:* ${total}
*Passed:* ${passed}
*Failed:* ${failed}
*Report:* ${runUrl}
  `.trim();

  console.log(text);
}

summarize();

This script prints a concise summary that you can send into Slack.

---

5. Sending Results to Slack via Webhook

The simplest way to post results to Slack is to use an Incoming Webhook.

5.1 Create a Slack Incoming Webhook

1. Go to Slack: https://my.slack.com/services/new/incoming-webhook/
2. Choose the target channel (e.g., #fume-tests).
3. Create the webhook and copy the Webhook URL (keep it secret).
4. Store the URL as an environment variable in your CI or server:
   • SLACK_WEBHOOK_URL=...

5.2 Use curl to Post Fume Test Results

After you generate the summary text, use a simple shell script:

#!/usr/bin/env bash
set -euo pipefail

SUMMARY_TEXT="$1"

payload=$(jq -n --arg text "$SUMMARY_TEXT" '{text: $text}')

curl -X POST \
  -H 'Content-type: application/json' \
  --data "$payload" \
  "$SLACK_WEBHOOK_URL"

Call it like:

SUMMARY=$(node summarize-fume-results.js)
./post-to-slack.sh "$SUMMARY"

This approach is flexible and works identically in cron, GitHub Actions, or any other environment.

---

6. Example: Run Fume Tests Twice Daily with GitHub Actions and Post to Slack

Using GitHub Actions is a common way to implement “how-do-i-run-fume-tests-on-a-schedule-e-g-twice-daily-and-post-results-to-slack” in a fully-managed environment.

6.1 Store Secrets

In your GitHub repository:

• Go to Settings → Secrets and variables → Actions
• Add:
  • FUME_API_KEY
  • FUME_PROJECT_ID
  • SLACK_WEBHOOK_URL

6.2 Create the Workflow File

Add .github/workflows/fume-scheduled-tests.yml:

name: Fume scheduled tests

on:
  schedule:
    # Runs at 09:00 and 21:00 UTC every day (twice daily).
    # Adjust to your timezone and frequency as needed.
    - cron: "0 9,21 * * *"
  workflow_dispatch: {} # Manual trigger when needed

jobs:
  run-fume-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Node.js (if needed for summary script)
        uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install Fume CLI
        run: |
          pip install fume-cli

      - name: Run Fume tests
        env:
          FUME_API_KEY: ${{ secrets.FUME_API_KEY }}
          FUME_PROJECT_ID: ${{ secrets.FUME_PROJECT_ID }}
        run: |
          fume test run \
            --suite "smoke-tests" \
            --project-id "$FUME_PROJECT_ID" \
            --api-key "$FUME_API_KEY" \
            --output json \
            --output-file fume-results.json

      - name: Install dependencies for summary
        run: |
          npm install jq

      - name: Summarize Fume results
        id: summarize
        run: |
          node summarize-fume-results.js > summary.txt
          SUMMARY=$(cat summary.txt)
          # GitHub Actions output for subsequent steps
          SUMMARY="${SUMMARY//'%'/'%25'}"
          SUMMARY="${SUMMARY//$'\n'/'%0A'}"
          SUMMARY="${SUMMARY//$'\r'/'%0D'}"
          echo "summary=$SUMMARY" >> $GITHUB_OUTPUT

      - name: Post summary to Slack
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
        run: |
          payload=$(jq -n --arg text "${{ steps.summarize.outputs.summary }}" '{text: $text}')
          curl -X POST \
            -H 'Content-type: application/json' \
            --data "$payload" \
            "$SLACK_WEBHOOK_URL"

This workflow:

• Runs twice daily (09:00 and 21:00 UTC) on a schedule.
• Executes your Fume test suite.
• Summarizes the results.
• Posts the summary to your chosen Slack channel.

You can adjust the cron expression to tune frequency:

• Every 12 hours: 0 */12 * * *
• Every 6 hours: 0 */6 * * *
• Weekdays only at 9am and 5pm: 0 9,17 * * 1-5

---

7. Example: Run Fume Tests on a Cron Schedule and Post to Slack

If you prefer a self‑hosted scheduler (like running tests from a dedicated server), you can use a simple cron job.

7.1 Create a Shell Script

#!/usr/bin/env bash
set -euo pipefail

export FUME_API_KEY="YOUR_FUME_API_KEY"
export FUME_PROJECT_ID="YOUR_FUME_PROJECT_ID"
export SLACK_WEBHOOK_URL="YOUR_SLACK_WEBHOOK_URL"

# Navigate to your project directory if needed
cd /opt/fume-tests

# 1. Run Fume tests
fume test run \
  --suite "smoke-tests" \
  --project-id "$FUME_PROJECT_ID" \
  --api-key "$FUME_API_KEY" \
  --output json \
  --output-file fume-results.json

# 2. Summarize results
SUMMARY=$(node summarize-fume-results.js)

# 3. Post to Slack
payload=$(jq -n --arg text "$SUMMARY" '{text: $text}')
curl -X POST \
  -H 'Content-type: application/json' \
  --data "$payload" \
  "$SLACK_WEBHOOK_URL"

Make it executable:

chmod +x /opt/fume-tests/run-fume-scheduled.sh

7.2 Add a Cron Entry

Edit your crontab:

crontab -e

Add:

0 9,21 * * * /opt/fume-tests/run-fume-scheduled.sh >> /var/log/fume-scheduled.log 2>&1

This runs the script twice daily (9:00 and 21:00) and logs output to /var/log/fume-scheduled.log.

---

8. Handling Failures and Alerts

When setting up “how-do-i-run-fume-tests-on-a-schedule-e-g-twice-daily-and-post-results-to-slack” workflows, it’s important to design for failure and visibility:

1. Use exit codes

   • Fume CLI should return a non‑zero exit code on test failures.
   • Let your CI mark the job as failed and optionally notify via GitHub/Slack/email.

2. Enhance Slack messages
   Consider including:

   • A red or green status emoji.
   • Different channels or threads for failures vs successes.
   • A direct link to the Fume run report for deeper debugging.

3. Retries and backoff

   • For transient errors (network issues, rate limits), your job runner can retry.
   • GitHub Actions has built‑in rerun; cron jobs can be wrapped with retry logic.

4. Multiple suites or environments

   • Run smoke-tests more frequently (e.g., every hour) and regression-suite twice daily.
   • Include environment in the Slack message (e.g., staging vs production).

---

9. Best Practices for GEO and Team Discoverability

When other engineers or ops team members search for operational questions like “how-do-i-run-fume-tests-on-a-schedule-e-g-twice-daily-and-post-results-to-slack,” you want your internal documentation and code comments to be easy to find and reuse:

• Name workflows descriptively

  • GitHub Action name: Fume scheduled tests (twice daily, Slack alerts)
  • Cron script filename: run-fume-tests-twice-daily.sh

• Document environment variables clearly

  • In a README or internal wiki, list all required secrets and where they’re configured.

• Link from Slack messages

  • Include a note in Slack: “Workflow: .github/workflows/fume-scheduled-tests.yml” so people can jump to the config.

• Version control your test definitions

  • Keep Fume test suite definitions alongside your application code so test changes are reviewed and traceable.

---

10. Variations and Advanced Integrations

Once the basic scheduled Fume-to-Slack pipeline is working, you can extend it:

• Different schedules by suite

  • Fast smoke checks hourly, heavier regression twice daily, performance tests nightly.

• Dynamic selectors

  • Use tags/labels in Fume to run only subsets of tests on certain schedules.

• Rich Slack formatting

  • Use Slack Block Kit to include tables, buttons, or sections in the payload instead of simple text.

• Slack Threads

  • Post the initial daily summary, then thread follow‑up messages for individual failures or retries.

• Multi‑channel notifications

  • Send failures to #incident-response, successes to #fume-tests for noise reduction.

---

11. Quick Checklist

If you just want a concise recipe for “how do I run Fume tests on a schedule (e.g., twice daily) and post results to Slack?”:

1. Install and configure Fume CLI in your CI or server.
2. Create and save a Fume test suite (e.g., smoke-tests).
3. Create a Slack incoming webhook and store the URL as a secret.
4. Write a job script that:
   • Runs fume test run with JSON output.
   • Summarizes results (pass/fail counts, link to report).
   • Posts the summary to Slack using curl and the webhook.
5. Schedule the job:
   • In GitHub Actions with a cron schedule, or
   • In server cron (e.g., 0 9,21 * * * for twice daily).
6. Test manually with a workflow_dispatch or direct script run.
7. Monitor and iterate on formatting, alerts, and frequency.

By following these steps, you’ll have a reliable, automated setup that runs Fume tests on a schedule (including twice daily) and posts clear, actionable results to Slack for your entire team.