AI Agent Automation Platforms
How do teams automate data collection from portals that require login and sometimes MFA?
7 min read
Most teams hit the same wall: the data they need lives inside portals that were never designed for automation—logins, MFA, rotating forms, CAPTCHAs, changing HTML. They try to bolt Selenium on top, wire in residential proxies, paste recovery codes into secrets managers…and then watch everything break the next time the portal adds a new step or tweaks the DOM.
Quick Answer: Teams automate data collection from login- and MFA-gated portals by running Web Agents that can authenticate like a real user, handle MFA through pre-agreed flows, navigate multi-step forms, and return structured data via API—at scale and in real time—rather than relying on brittle scripts or manual workflows.
Frequently Asked Questions
How do teams reliably automate data collection from portals that require login and sometimes MFA?
Short Answer: They use Web Agents or similar infrastructure that can authenticate, handle MFA flows predictably, and execute multi-step workflows server-side, returning structured outputs via API instead of fragile screenshots or HTML dumps.
Expanded Explanation:
The core shift is moving away from “just log in with Selenium” thinking and towards live workflow execution. For high-value portals—insurance carriers, banking dashboards, marketplace consoles, vendor portals—you define the workflow (where to log in, what to click, what to extract), then delegate the execution to an agent platform that can navigate like a user, survive anti-bot measures, and normalize outputs.
MFA is treated as a repeatable pattern, not a one-off exception. Teams either 1) route MFA challenges into a controlled channel (e.g., shared email inbox, dedicated SMS/OTP service, or hardware-token-backed accounts), or 2) pre-register app-specific passwords or long-lived device trust, so the automation path is stable and auditable. The result: agents can run unattended, across hundreds of accounts and portals, without a human typing codes all day.
Key Takeaways:
- “Automating login + MFA” means building a reliable authentication pattern, not just solving for a single portal screen.
- Web Agents that authenticate, navigate, and extract in one flow are more stable than homegrown Selenium + proxy stacks.
What is the typical process to set up automation for a login- and MFA-protected portal?
Short Answer: You map the workflow, stabilize authentication (including MFA), codify the navigation steps, then run agents in parallel and monitor them via logs, screenshots, and structured outputs.
Expanded Explanation:
Teams that get this right treat portal automation like any other production integration. First, they inventory which portals matter, which accounts/credentials they’ll use, and what “done” means (e.g., “get final quote premium + fees across 20 carriers”). Then they define the exact interaction path: login page → MFA prompt → navigation steps → extraction points → expected outputs.
On a platform like TinyFish, this turns into a declarative workflow: “authenticate with these credentials → answer these MFA challenges → navigate to this report → extract these fields → return JSON.” The system handles the messy parts—DOM changes, CAPTCHAs, transient failures—while you watch real-time execution and examine run history if something drifts.
Steps:
- Define the workflow: Identify portals, credentials, MFA type, and the specific data you need (fields, formats, frequency).
- Stabilize authentication: Choose and configure an MFA pattern you can automate or consistently respond to (app passwords, trusted devices, OTP routing).
- Codify navigation and extraction: Use Web Agents to log in, navigate multi-step flows, and return structured data via API; iterate based on observability (logs, screenshots) until it runs reliably at scale.
How is using Web Agents different from traditional automation (Selenium) or hiring manual ops?
Short Answer: Web Agents execute live workflows like automation, but with search-like speed and scale—avoiding the slowness and fragility of Selenium stacks and the cost/delay of manual teams.
Expanded Explanation:
Traditional automation (Selenium/Playwright) can reach logged-in and MFA-protected portals, but it doesn’t scale cleanly. Each new portal means another brittle script, more proxy management, more CAPTCHAs, and weekly breakage. Manual ops are accurate but slow and expensive, especially when pricing or eligibility changes hourly. Search/indexed data is fast but stale and often can’t see behind logins at all.
Web Agents like TinyFish introduce a fourth path: “One API. Any website. Live data back.” You define your workflow once, then deploy agents against 10, 100, or 1,000 accounts or portals concurrently. They authenticate, handle anti-bot systems, navigate changing UIs, and deliver normalized, structured outputs—without your team ever touching a browser, proxy, or captcha solver directly.
Comparison Snapshot:
- Option A: Traditional automation (Selenium/Playwright): Reaches auth + MFA but is fragile, expensive to maintain, and slow to scale across many portals and accounts.
- Option B: Web Agents (e.g., TinyFish): Serverless, concurrent, and resilient; handle auth/MFA, anti-bot, and navigation, returning structured results via API.
- Best for: Teams that need live, authenticated data at production speed—across many portals, steps, and countries—without growing a platform-maintenance team.
How do you actually implement something like TinyFish for login- and MFA-gated portals?
Short Answer: You specify your workflows and portals, connect credentials and MFA strategy, then let TinyFish deploy Web Agents that execute those workflows in parallel and send results to your systems via API.
Expanded Explanation:
Implementation is less “build a bot” and more “describe the job.” On TinyFish, you start by defining the workflow: which portals, how often to run, what credentials to use per portal/account, and what data to extract or transact on. TinyFish then runs Web Agents server-side to authenticate, navigate, and extract—handling CAPTCHAs and bot detection autonomously.
You monitor runs in the TinyFish Workbench: real-time streaming logs via SSE, screenshots, run history, and success metrics. For teams with strong governance needs, SSO, permissions, and audit trails anchor who can use which credentials and why. Outputs are returned as structured JSON over API straight into data warehouses, pricing engines, or internal tools—no HTML parsing layer you have to maintain.
What You Need:
- Clear workflows and credentials: A mapped process per portal (login → MFA → pages → fields) and a credential/MFA pattern you can support operationally.
- An execution platform: A Web Agent / Search Agent API like TinyFish that runs workflows concurrently, handles auth and anti-bot, and returns structured data with observability and enterprise controls.
How should teams think strategically about MFA when automating data collection from secure portals?
Short Answer: Treat MFA as part of the system design—standardize MFA types, centralize control, and choose patterns that are automatable or at least predictable, balancing security with operational throughput.
Expanded Explanation:
MFA is non-negotiable for most sensitive portals—insurance carriers, banks, vendor consoles, admin dashboards. The goal isn’t to “get around” MFA; it’s to choose MFA patterns that fit automation. That often means shifting from SMS codes to app-based tokens, hardware keys, or app passwords that can be referenced by your automation platform under strict access controls.
Teams that succeed create a clear policy: which accounts are automation-only, which MFA methods are allowed, how MFA secrets are stored and rotated, and how to log and audit every login. Platforms like TinyFish then operate within that framework, using MFA inputs you’ve sanctioned while preserving 24/7, unattended operation. This keeps security teams comfortable and keeps your pricing, eligibility, or availability data fresh enough to drive decisions in real time.
Why It Matters:
- Impact on reliability: A coherent MFA strategy avoids “automation blocked by code prompts” and eliminates human bottlenecks for every login.
- Impact on compliance and trust: Centralizing MFA patterns and secrets, plus having full audit trails, keeps security and risk teams aligned while you scale automation.
Quick Recap
Automating data collection from login- and MFA-protected portals is no longer about heroic Selenium scripts or armies of manual operators. The durable pattern is Web Agents that authenticate, navigate, and extract live data behind logins—at scale, with observability, and under enterprise-grade security controls. You define workflows, pick MFA strategies that work with automation, and let a platform like TinyFish run 1 to 1,000+ concurrent operations, returning structured, decision-ready outputs instead of brittle HTML.