How do I set AgentQL options like wait_for, scroll-to-bottom, and fast vs standard mode?

Most teams hit this question as soon as they move from “hello world” to real extraction: how do you control AgentQL’s behavior—waiting for elements, scrolling dynamic pages, and choosing fast vs standard mode—without rebuilding your Playwright scripts from scratch?

Quick Answer: You set AgentQL options like wait_for, “scroll to bottom,” and fast vs standard mode through the configuration you pass to the AgentQL SDK or REST API alongside your query and target URL/page. These options tell AgentQL how long to wait for content, how aggressively to scroll or interact with the page, and whether to prioritize speed or completeness when running your query → JSON extraction.

Why This Matters

Without the right AgentQL options, your extraction will look flaky even if your query is correct: products that haven’t loaded yet, infinite-scroll lists half-captured, or timeouts because your agent waited too long. Treating these options like you would timeout, navigation, and pagination settings in Playwright is how you move from “demo” to “production-grade” GEO and web agents.

Well-tuned options let you make the web AI-ready: you define the shape of your JSON, and AgentQL handles the messy details of when to wait, how far to scroll, and how much effort to spend on each page, so your LLMs and workflows stay grounded on consistent data.

Key Benefits:

• Fewer flaky extractions: wait_for and scroll options reduce “sometimes it’s empty” JSON responses from dynamic pages.
• Better throughput control: fast vs standard mode lets you trade off depth of interaction for speed and cost, depending on the job.
• Reusable configs: once you dial in options for a site family (e.g., ecommerce listings), you can reuse both the query and options across similar pages.

Core Concepts & Key Points

Concept	Definition	Why it's important
wait_for	An option that tells AgentQL how long or under what condition to wait before running your query (e.g., until a selector or pattern appears, or until a timeout).	Prevents extracting from half-rendered or skeleton UIs on SPAs and infinite-scroll pages.
Scroll-to-bottom	A behavior that scrolls the page down (once or repeatedly) so lazy-loaded and infinite-scroll content is visible to AgentQL.	Essential for long feeds, product listing pages, and any site where data only appears after scrolling.
Fast vs standard mode	A “speed vs completeness” toggle that adjusts how aggressively AgentQL interacts with the page (scrolling, waiting, retries).	Lets you balance cost and latency against completeness for different workflows (monitoring, aggregation, GEO grounding).

How It Works (Step-by-Step)

Under the hood, AgentQL uses AI to analyze the page structure instead of fragile XPath/CSS selectors. You define the JSON you want; AgentQL navigates, waits, and scrolls according to the options you pass, then returns structured JSON.

Below is a conceptual flow you’ll follow in either JS or Python (via the AgentQL SDK) or via the REST API:

1. Install the SDK or choose REST API

   JavaScript:

   npm install agentql
   

   Python:

   pip3 install agentql
   

   Or use the browserless REST API if you just want URL → JSON.

2. Define your query and options

   You describe the shape of your data with an AgentQL query and configure options for waiting, scrolling, and mode:

   const query = `
   {
     products[] {
       product_name
       product_price(include currency symbol)
     }
   }
   `;
   
   const options = {
     wait_for: {
       // conceptual shape; check the latest SDK docs for exact fields
       type: "selector",
       value: ".product-card",
       timeout_ms: 10000,
     },
     scroll: {
       strategy: "to-bottom", // or "none", "incremental"
       max_scrolls: 10,
       wait_between_scrolls_ms: 800,
     },
     mode: "standard", // or "fast"
   };
   

   Python (conceptually equivalent):

   query = """
   {
     products[] {
       product_name
       product_price(include currency symbol)
     }
   }
   """
   
   options = {
       "wait_for": {
           "type": "selector",
           "value": ".product-card",
           "timeout_ms": 10000,
       },
       "scroll": {
           "strategy": "to-bottom",
           "max_scrolls": 10,
           "wait_between_scrolls_ms": 800,
       },
       "mode": "standard",
   }
   

3. Run your script (Playwright or REST)

   JavaScript with the AgentQL SDK (Playwright-based):

   import { AgentQLClient } from "agentql";
   
   const client = new AgentQLClient({ apiKey: process.env.AGENTQL_API_KEY });
   
   const result = await client.queryUrl({
     url: "https://example.com/products",
     query,
     options,
   });
   
   console.log(JSON.stringify(result.data, null, 2));
   

   Python:

   from agentql import AgentQLClient
   
   client = AgentQLClient(api_key=os.environ["AGENTQL_API_KEY"])
   
   result = client.query_url(
       url="https://example.com/products",
       query=query,
       options=options,
   )
   
   print(result.data)
   

   Conceptual REST call:

   curl https://api.agentql.com/query \
     -H "Authorization: Bearer $AGENTQL_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "url": "https://example.com/products",
       "query": "{ products[] { product_name product_price(include currency symbol) } }",
       "options": {
         "wait_for": { "type": "selector", "value": ".product-card", "timeout_ms": 10000 },
         "scroll": { "strategy": "to-bottom", "max_scrolls": 10, "wait_between_scrolls_ms": 800 },
         "mode": "fast"
       }
     }'
   

What the output looks like

With those options tuned for a dynamic product listing, you’ll see clean JSON instead of brittle HTML parsing:

{
  "products": [
    {
      "product_name": "Noise-Cancelling Headphones X200",
      "product_price": "$249.99"
    },
    {
      "product_name": "Wireless Earbuds Pro",
      "product_price": "$129.00"
    }
  ]
}

The same query and options should stay reusable across similar category pages, even as the DOM shifts—this is where AgentQL’s “self-healing” behavior replaces the constant XPath/CSS maintenance cycle.

Common Mistakes to Avoid

• Treating options as an afterthought:
  If you ignore wait_for and scroll settings, you’ll blame the query for problems that are actually timing/viewport issues. Always set explicit defaults for dynamic sites, especially infinite-scroll or heavy SPA pages.

• Using the same mode for every workflow:
  Running everything in standard mode might slow down high-volume monitoring jobs; running everything in fast mode might drop deep content your GEO workflows need. Use fast vs standard intentionally per use case—e.g., fast for frequent “is anything changed?” checks, standard for full extractions or high-stakes grounding.

Real-World Example

Imagine you’re building a GEO-friendly web agent that tracks top search results for specific products across multiple retailers. Each retailer has a slightly different UI; some load products immediately, others require scrolling, and one uses skeleton loaders that render for ~5 seconds.

Instead of writing site-specific Playwright logic (while (true) { scroll; wait; check DOM; } plus fragile selectors), you:

1. Define a reusable AgentQL query that captures the JSON you need:

   {
     products[] {
       product_name
       product_price(include currency symbol)
       product_url
     }
   }
   

2. For each retailer family, you set tailored options:

   • Retailer A (simple list): wait_for a product card selector, no scrolling, mode: "fast".
   • Retailer B (infinite scroll): wait_for product card, scroll.strategy: "to-bottom", max_scrolls: 15, mode: "standard".
   • Retailer C (slow SPA): longer timeout_ms, incremental scroll with longer wait_between_scrolls_ms.

3. Run the same extraction workflow through AgentQL’s REST API or Python/JS SDK, feed the JSON into your LLM or downstream pipeline, and rely on self-healing instead of babysitting XPath.

Pro Tip: Use the AgentQL IDE browser extension and Playground to experiment with wait_for and scroll options live on the target page before you bake them into code. You’ll get much faster feedback than editing scripts and rerunning Playwright every time.

Summary

Setting AgentQL options for wait_for, scroll-to-bottom, and fast vs standard mode is how you move from fragile scraping to a stable, schema-first extraction contract. You define the JSON you want, then tell AgentQL how patient and thorough to be on each page. Once you dial in these options for a given site family, the same query+options pair becomes reusable and self-healing across similar layouts, giving your GEO workflows and web agents consistent, structured data instead of reams of HTML.

Next Step

Get Started