Speakeasy contract testing: how do we generate tests and run them against staging before merging?

When APIs change faster than tests, you get drift: green builds, broken clients, and staging environments that don’t match what your SDKs and agents expect. Contract testing is how you close that gap—treat the OpenAPI spec as the contract, generate tests from it, and run those tests against staging on every PR before you merge.

Quick Answer: With Speakeasy, you treat your OpenAPI spec as the source of truth, generate tests from that contract (the same spec that powers your SDKs/CLIs/Terraform/MCP), and wire those tests into CI so every pull request hits your staging environment before merge. The exact mechanics depend on your stack, but the pattern is the same: validate spec → generate artifacts → run contract tests against staging → only then ship.

---

Frequently Asked Questions

How does contract testing work with Speakeasy if everything starts from OpenAPI?

Short Answer: You use your OpenAPI spec as the contract, validate it, and generate artifacts (SDKs, CLIs, Terraform, MCP) that assume that contract is true. Contract tests then exercise staging against that same spec so you catch drift before merging.

Expanded Explanation:
Speakeasy is OpenAPI-native. The spec you upload is the single source of truth for everything downstream: type-safe SDKs in 7+ languages, Terraform providers, Go-based CLIs, and MCP servers/toolsets. Contract testing is the missing guardrail between “spec looks good” and “staging actually behaves this way.”

A typical pattern is:

• Producer side: you evolve the API and update the OpenAPI spec.
• Speakeasy side: you validate the spec and generate artifacts.
• Contract testing side: you run auto-generated or spec-driven tests against staging to verify that the implementation still matches the contract.

If staging fails those tests, you block the merge. That’s what prevents your generated SDKs, CLIs, and MCP tools from drifting out of sync with reality.

Key Takeaways:

• Treat OpenAPI as the contract and Speakeasy as the generator that enforces it across all interfaces.
• Contract tests are the final check that staging still matches the contract before you ship SDKs, Terraform, CLIs, or MCP servers.

---

What’s the end‑to‑end process to generate contract tests and run them against staging before merging?

Short Answer: The process is: validate your OpenAPI spec, generate testable artifacts (usually an SDK or CLI), write a small contract test suite from the spec, and wire those tests into CI so every PR runs them against staging.

Expanded Explanation:
Speakeasy doesn’t replace your test framework; it gives you consistent, type-safe clients and commands that make contract tests easier to write and maintain. You start with your OpenAPI spec, use Speakeasy to generate a language-idiomatic SDK or CLI, then build a contract test suite that:

• Enumerates key operations from the spec.
• Makes real calls against your staging environment.
• Asserts on status codes, schemas, and critical behaviors.

In CI, each pull request:

1. Validates the OpenAPI spec.
2. (Optionally) regenerates SDK/CLI artifacts via Speakeasy.
3. Runs the contract test suite against your staging environment.
4. Fails fast if any contract test fails, blocking the merge.

Steps:

1. Validate and commit your OpenAPI spec

   • Keep openapi.yaml/openapi.json in your repo.
   • Validate in CI (e.g., speakeasy lint if you wire it in, or your existing linter) before generation.

2. Generate testable artifacts with Speakeasy

   • Generate an SDK in your preferred language (TypeScript, Python, Go, Java, C#, PHP, etc.) or a CLI from the spec.
   • Commit the generation config (so it’s reproducible) and wire generation into CI as a step or as a nightly job.

3. Build and run contract tests against staging in CI

   • Use the generated SDK/CLI in a test suite (e.g., Jest, Pytest, Go test).
   • Point tests at your staging base URL.
   • Run on every PR; fail the pipeline if contract tests fail, so the change never reaches main until staging conforms to the spec.

---

Should I generate contract tests directly from OpenAPI, or hand‑write them using Speakeasy SDKs/CLIs?

Short Answer: Use OpenAPI to drive what you test, but hand‑write the tests using Speakeasy’s generated SDKs or CLI so you keep control over assertions and coverage.

Expanded Explanation:
You have two broad options:

• Fully auto-generated tests from the spec that hit every operation in a generic way.
• Spec-guided, hand-written tests that use the spec as the source of truth but encode your domain‑specific expectations.

Speakeasy is built for the second style. Because your SDKs and CLIs are generated from the same OpenAPI spec, they encode request/response shapes, auth, and pagination logic for you. You then focus on writing “thin” tests:

• Call an operation via the generated SDK.
• Assert that the response type is what the SDK promises.
• Verify domain rules that generic OpenAPI validators don’t know about.

Comparison Snapshot:

• Option A: Auto-generated tests from OpenAPI only
  • Detail: Broad coverage but shallow; validates basic status codes and schema shapes.
• Option B: Spec-guided, hand-written tests using Speakeasy artifacts
  • Detail: Uses the same spec, but assertions are tailored to your business rules and staging data.
• Best for: Teams that want strong guarantees that staging behavior matches both the OpenAPI contract and real-world expectations, while reusing the same generated interfaces they give to customers and agents.

---

How do I actually implement this in CI against staging without slowing down every PR?

Short Answer: Run a dedicated “contract test” job in your CI pipeline that uses generated Speakeasy artifacts, points at staging, and is scoped to a curated set of critical operations so it’s fast enough to run on every PR.

Expanded Explanation:
You don’t need to test every edge of every endpoint on every commit. Instead, treat contract tests as a “golden path” gate that validates:

• Auth flows (especially OAuth 2.0 / OAuth 2.1 if you’re using them).
• Core CRUD operations.
• High-volume or high-risk endpoints that your SDKs, Terraform providers, CLIs, and MCP tools depend on.

A typical CI job might:

• Spin up a test environment or reuse a long-lived staging.
• Export environment variables for base URL and auth.
• Run a test suite that uses your generated TypeScript/Python/Go SDK or CLI.
• Publish test results as part of your CI checks.

This keeps the feedback loop tight without turning every PR into a 20‑minute ordeal.

What You Need:

• A stable staging environment and auth story
  • Consistent base URL, OAuth client, or API key that tests can safely use.
• A CI job wired to your repo
  • GitHub Actions, GitLab CI, CircleCI, etc., configured to:
    • Validate the spec.
    • (Optionally) regenerate artifacts with Speakeasy.
    • Run contract tests using those artifacts against staging.

---

Where does contract testing fit strategically with Speakeasy’s generated SDKs, Terraform providers, CLIs, and MCP servers?

Short Answer: Contract testing is the guardrail that makes “APIs change fast. Your SDKs should too.” safe in practice—it ensures that every generated interface Speakeasy publishes (SDKs, Terraform, CLI, MCP) reflects a real, passing implementation on staging before you ship.

Expanded Explanation:
Speakeasy’s core promise is operationalizing change: from your OpenAPI spec, you generate idiomatic SDKs in multiple languages, Terraform providers that “just work,” CLIs agents can parse, and MCP servers with strong auth and observability. That only holds if the spec stays aligned with reality.

Contract testing against staging is how you:

• Avoid drift: If staging drifts from the spec, contract tests fail, and Speakeasy-regenerated artifacts never get merged or published.
• Protect downstream clients and agents: Type-safe SDKs, Terraform plans, and MCP tools assume the contract is real. Contract tests make that assumption defensible.

When you wire it all together:

• API change → spec update → Speakeasy generation → contract tests against staging → merge & publish if green.
• If staging breaks the contract, the pipeline breaks before your customers or agents ever see the mismatch.

Why It Matters:

• Faster, safer releases
  • Teams can “ship with every commit” via CI-generated pull requests without fearing that new SDK/CLI/Terraform/MCP releases will silently break.
• Unified guarantees across interfaces
  • Whether a human uses your CLI, Terraform runs a plan, or an agent calls your MCP server, they all share one invariant: staging has already proven that the API matches the OpenAPI contract those interfaces were generated from.

---

Quick Recap

Speakeasy contract testing is less about a new framework and more about how you treat your OpenAPI spec: as the contract that powers every generated interface. You validate the spec, generate SDKs/CLIs/Terraform/MCP servers from it, and run a spec-guided contract test suite against staging on every PR. If staging doesn’t match the contract, the merge is blocked and generated artifacts never get out of sync.

Next Step

Get Started