TinyFish SSE streaming: how do I stream progress events and capture the final result in my backend?

TinyFish streams like an operator, not a black box: your Web Agents run in the cloud, and you get a live event feed plus a final structured result over Server-Sent Events (SSE). If you wire your backend correctly, you can watch navigation/auth/ extraction/transact steps in real time and still capture the final JSON payload cleanly for storage, enrichment, or downstream jobs.

Quick Answer: Use TinyFish’s /v1/automation/run-sse endpoint to open a long‑lived SSE connection from your backend, parse event: / data: lines as they arrive, and treat the final completed (or terminal) event as the canonical result to persist in your database or queue.

---

Quick Answer: TinyFish SSE streaming gives you real‑time progress updates and a final structured result through a single HTTP connection.

Frequently Asked Questions

How does TinyFish SSE streaming work for Web Agent runs?

Short Answer: TinyFish uses SSE to stream real-time progress events and the final result over a single HTTP response, so you can observe every step and still capture a clean, structured output at the end.

Expanded Explanation: When you call the TinyFish SSE endpoint, TinyFish spins up one or more Web Agents in the cloud to navigate, authenticate, extract, and transact on your target sites. As those agents run, TinyFish emits a sequence of SSE messages—each tagged with an event type and a JSON payload. You see high-level milestones (started, navigation, form_submit, captcha_handled), debug context (logs, screenshots), and eventually a terminal event containing the final result object.

Because this all happens over SSE, you don’t poll or manage websockets. Your backend opens one HTTP connection, reads event-stream lines as they arrive, and closes the connection when the run finishes or errors. At that point, you typically store the final result in your database, emit it to Kafka/SQS, or use it to drive downstream pricing, availability, or eligibility logic.

Key Takeaways:

• One SSE connection streams both progress events and the final structured result.
• You don’t poll: SSE pushes status changes from TinyFish to your backend in real time.

---

How do I stream TinyFish SSE progress events and capture the final result in my backend?

Short Answer: Open an SSE request to https://agent.tinyfish.ai/v1/automation/run-sse, parse each event: / data: block as it arrives, and treat the terminal event (e.g., completed or failed) as the point where you persist the final payload.

Expanded Explanation: TinyFish’s SSE API behaves like a standard text/event-stream endpoint. Your backend sends a JSON payload describing the run—URL, goal, optional proxy/session config—and TinyFish responds with a streaming body. Each chunk is an SSE event with an event: name and a data: line containing JSON.

The pattern is:

1. Open the SSE connection.
2. While the connection is open, parse progress events for logs, monitoring, or UI.
3. When you receive the terminal event, parse its data, persist the result, and close the connection (or let the server close it).

Here’s a minimal Node.js example (using node-fetch style APIs):

import fetch from 'node-fetch';

async function runTinyFishJob() {
  const resp = await fetch(
    'https://agent.tinyfish.ai/v1/automation/run-sse',
    {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-API-Key': process.env.TINYFISH_API_KEY,
        'Accept': 'text/event-stream',
      },
      body: JSON.stringify({
        url: 'https://amazon.com',
        goal: 'Find me the price of airpods pro 3',
        proxy_config: { enabled: false },
      }),
    }
  );

  if (!resp.ok || !resp.body) {
    throw new Error(`TinyFish SSE request failed: ${resp.status}`);
  }

  const reader = resp.body.getReader();
  const decoder = new TextDecoder('utf-8');
  let buffer = '';
  let finalResult = null;

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });

    // SSE events are separated by blank lines
    let boundary;
    while ((boundary = buffer.indexOf('\n\n')) >= 0) {
      const rawEvent = buffer.slice(0, boundary);
      buffer = buffer.slice(boundary + 2);

      const lines = rawEvent.split('\n');
      let eventName = 'message';
      let data = '';

      for (const line of lines) {
        if (line.startsWith('event:')) {
          eventName = line.slice(6).trim();
        } else if (line.startsWith('data:')) {
          data += line.slice(5).trim();
        }
      }

      if (!data) continue;
      let parsed;
      try {
        parsed = JSON.parse(data);
      } catch (e) {
        console.error('Failed to parse TinyFish event data', e, data);
        continue;
      }

      // Handle progress vs terminal events
      if (eventName === 'step' || eventName === 'log') {
        console.log('[TinyFish progress]', eventName, parsed);
      } else if (eventName === 'completed') {
        finalResult = parsed;
        console.log('[TinyFish completed]', parsed);
      } else if (eventName === 'error' || eventName === 'failed') {
        console.error('[TinyFish error]', parsed);
      }
    }
  }

  if (!finalResult) {
    throw new Error('TinyFish run finished without a completed event');
  }

  // Persist or forward the final result
  await saveResultToDB(finalResult);
  return finalResult;
}

async function saveResultToDB(result) {
  // Your persistence logic here
}

Steps:

1. Call the TinyFish SSE endpoint with your API key and JSON body (url, goal, and config).
2. Stream and parse SSE events in your backend, using event: and data: lines to separate progress from terminal events.
3. Persist the final result when you receive the terminal event, then close or let the connection close.

---

What’s the difference between TinyFish SSE streaming and a normal (non‑SSE) run?

Short Answer: SSE gives you live, step‑by‑step progress and a final result in one stream; non‑SSE runs usually hide the intermediate steps and only expose the final result after completion.

Expanded Explanation: With a traditional “fire-and-forget” API, you issue a request and either block until it finishes or poll a status endpoint. You see the world in two states: not done vs done. That’s painful when agents are navigating 20+ portals, handling CAPTCHAs, and filling 50‑step forms.

TinyFish’s SSE endpoint surfaces the execution timeline: start, navigation, auth, form fillings, retries, anti-bot challenges, and completion. You can wire these events into your monitoring stack, show real‑time agent traces in your dashboard, and debug failures without replaying runs blindly.

You still end up with the same structured result you’d expect from a non-streaming run, but SSE adds observability and removes the need for polling. For long or concurrent runs (hundreds of agents at once), that’s the difference between guessing and knowing where your workflows are stuck.

Comparison Snapshot:

• SSE streaming: Live progress events + terminal result over one streaming response.
• Non‑SSE / polling pattern: Fire request → poll status → fetch final result, with no granular steps in between.
• Best for: Production workflows where you care about real-time visibility, debugging, and user-facing status (e.g., “Gathering quotes from 23 carriers…”).

---

How do I implement TinyFish SSE streaming in a production backend?

Short Answer: Treat the TinyFish SSE run as a long‑lived job: open the connection from a worker or job runner, stream events for observability, and write the terminal result into your system of record or message bus.

Expanded Explanation: In production, you don’t want web requests blocking while agents run through 50+ steps. Instead, use a job queue or worker system (Sidekiq, Celery, BullMQ, custom) to manage TinyFish runs. Your API layer enqueues work; a worker process opens the SSE connection, consumes events, and persists the final output.

You’ll usually:

• Log or forward progress events to your monitoring/telemetry system.
• Attach run IDs to correlate TinyFish Workbench screenshots and logs with your internal entities.
• Handle retries for transient failures (network timeouts, anti-bot anomalies) at the job level, not at the HTTP controller.

Because TinyFish runs unattended in the cloud with production speed (often sub‑minute even across 50 portals), this pattern scales: you can fan out 1 → 1,000 Web Agents, each streaming back progress, without managing browsers, proxies, or custom headless stacks yourself.

What You Need:

• A background job runner (workers) that can hold long‑lived HTTP connections and parse SSE.
• A persistence path (database, queue, or data warehouse) where your final structured results are stored or forwarded.

---

How should I design my SSE streaming strategy for long‑running, concurrent TinyFish jobs?

Short Answer: Use SSE streams for observability and final result capture, but architect around concurrency: fan out jobs to workers, correlate events by run ID, and forward final payloads to durable storage or a message bus for downstream systems.

Expanded Explanation: Once you push beyond 1–2 agents and into hundreds, streaming design becomes a systems question. You’re essentially running a “live web data fabric” where TinyFish Web Agents handle the hard parts—navigation, authentication, CAPTCHAs—while your backend orchestrates what happens with the outputs.

A resilient pattern:

• Concurrency: Spin up many workers; each worker owns an SSE connection for one TinyFish run. TinyFish is designed for parallel execution at scale (1,000 simultaneous ops), so your bottleneck becomes your worker pool and event handling.
• Run correlation: Include your own job_id or correlation token in the TinyFish metadata or goal payload. When events stream back, attach that ID so logs and results land under the right customer, portal, or request.
• Observability: Push progress events into your logging stack (e.g., Datadog, Grafana, OpenTelemetry). Use event types to alert on anti-bot hits, repeated failures, or elongating runtimes.
• Durability: As soon as you see the terminal completed event, write the result into a durable system (database, S3, data warehouse, or Kafka/SQS). Don’t treat SSE as storage; treat it as transport.

The payoff: you move from “3–5 days of manual portal work” or brittle Playwright/Selenium stacks to “sub‑minute live data, 98.7% success rate, 1,000 simultaneous runs,” with clear traces for every job.

Why It Matters:

• Streaming-first design lets you scale TinyFish from one-off experiments to production web data operations without changing patterns.
• Your team gets live execution, not cached pages, with the governance and reliability you expect from enterprise infrastructure.

---

Quick Recap

TinyFish SSE streaming gives you a single, long‑lived HTTP connection that delivers both real‑time progress events and a final structured result from your Web Agents. In your backend, you open the /v1/automation/run-sse endpoint, parse event: / data: blocks as they arrive, use progress events for observability, and persist the terminal completed payload as your source of truth. Wrap that in a worker/queue architecture, and you can run hundreds or thousands of concurrent agents at production speed, with clear traces and durable outputs.

Next Step

Get Started