How do we add an AI help widget that answers from our API docs with citations (and doesn’t leak private endpoints)?

Most teams want an AI help widget that feels magical to users but completely boring to security and compliance. That means: it answers from your API docs, shows clear citations, and never leaks private endpoints or internal notes. The good news: you can absolutely do this with the right architecture and guardrails.

Below is a practical, implementation-focused walkthrough that matches the intent of how-do-we-add-an-ai-help-widget-that-answers-from-our-api-docs-with-citations-an and covers both UX and safety.

1. Define what your AI help widget should (and shouldn’t) do

Before touching code, decide:

Core capabilities you want

Answer user questions about your API:
- how to authenticate
- which endpoint to use
- parameter examples and errors
Provide citations:
- link back to specific docs pages or sections
- show where each piece of information came from
Embed in your product/site:
- as a chat bubble
- in a docs sidebar
- on a “Help” or “Support” page

Constraints you must enforce

No mention of:
- internal endpoints (e.g. /admin/*, /beta/*, /internal/*)
- unreleased features
- internal notes, TODOs, or comments
No hallucinated APIs:
- don’t make up endpoints
- don’t invent parameters or error codes
No data exfiltration:
- don’t echo back secrets, tokens, or API keys
- don’t show stack traces with sensitive info

Write this as a brief spec; you’ll translate it directly into system prompts, filters, and retrieval rules.

2. Gather and structure your API docs as a trusted knowledge base

Your AI help widget should answer only from approved, structured documentation. Think of it as a curated, “public-only” view of your knowledge.

2.1 Decide which sources are allowed

Common sources:

Public API docs:
- OpenAPI/Swagger schemas
- Markdown pages (/docs, /developers, /api-reference)
- Tutorials, guides, quickstarts
Public changelogs and release notes
Public FAQ and troubleshooting guides
Public blog posts that explain implementation details

Exclude from the knowledge base:

Private Notion/Confluence spaces
Engineering runbooks
Internal Slack/Teams exports
Internal GitHub repos
Draft docs and feature specs

2.2 Crawl or export only the public docs

Use one of these approaches:

Static export
- Export docs as markdown, HTML, or JSON from your docs platform (e.g., Docusaurus, ReadTheDocs, GitBook).
- Store them in a dedicated “AI docs” bucket or repo.
Controlled crawler
- Crawl only URLs under /docs or /api.
- Enforce a strict allowlist of domains and paths.
- Skip anything behind auth or “internal” tags.

Log the exact dataset you’re feeding into your system. This becomes your auditable “approved corpus”.

3. Build a retrieval layer that supports citations

You don’t want a generic chatbot. You want a retrieval-augmented system that:

Searches relevant docs.
Passes only those snippets to the model.
Requires the model to cite them in its answer.

3.1 Choose between vector search, keyword search, or both

Vector (embedding) search
- Good for semantic queries: “How do I get all active users?” that maps to “List users endpoint”.
- Use embeddings from your LLM provider or an open model.
Keyword / BM25 search
- Good for exact API names: POST /v1/subscriptions or parameter names.

A common pattern is a hybrid search: semantic + keyword, then rank.

3.2 Chunk your docs for retrieval

To get precise citations:

Split docs into small chunks, e.g. 200–500 tokens.
Preserve structure:
- section titles
- headers (H2/H3)
- endpoint names
- file paths or canonical URLs
Store metadata with each chunk:
- source_url
- title
- section
- api_version
- visibility: public

You’ll use source_url and section to generate citations in the chat response.

4. Prevent private endpoints from ever entering the system

The safest way to avoid leaking private endpoints is to make sure they simply do not exist in the model’s context or retrieval layer.

4.1 Clean your documentation before indexing

Scan your docs for:

Endpoints like:
- /admin/*, /internal/*, /debug/*, /beta/*
- any path matching pre-release or admin APIs
Internal comments:
- “TODO: hide before GA”
- “Internal use only”
- “Requires admin token”

Then:

Delete or redact those sections before indexing.
Or tag them as visibility: internal and filter them out at query time (and ideally at indexing time too).

4.2 Enforce allowlists in your indexing pipeline

Instead of trying to filter out bad paths with blacklists, use strict allowlists:

Only index endpoints that match:
- /v1/* (or your stable versions)
- docs pages configured as “public”
Only index docs from:
- your public docs domain
- specific Git branches (e.g. main, public-docs)

This avoids future mistakes when someone adds a new internal doc in the same repo.

5. Add strict retrieval filters at query time

Even with a clean corpus, double-protect your system at query time.

When performing retrieval:

Filter metadata:
- visibility == "public"
- is_internal == false
- api_version in ["v1","v2"] (if older, deprecated endpoints shouldn’t be shown)
Block known sensitive paths:
- path NOT LIKE "%/admin/%"
- path NOT LIKE "%/internal/%"

If your store supports it, use metadata filters instead of trusting the free-text content of the chunk.

6. Design the prompt and system instructions to require citations

The LLM needs explicit instructions to:

Only answer from the provided docs.
Provide citations for each fact.
Admit when it doesn’t know.

6.1 Example system prompt for your helper

You can adapt this to your provider:

You are an API support assistant embedded in our product.

You must:
- Answer only using the provided "Context" documents.
- If the answer is not clearly contained in the context, say you do not know and suggest where the user can look in the docs.
- Include citations for each part of your answer, using this format: [label](url#anchor).
- Do NOT invent endpoints, parameters, or features.
- Do NOT mention or describe internal, admin, or beta endpoints.
- If the user asks about admin, internal, or beta features, respond that they are not available or not documented publicly.

Forbidden:
- Leaking internal-only endpoints or undocumented behavior.
- Revealing secrets, tokens, or internal notes.

If the user’s question is not about our API, answer briefly that you are only able to answer questions about our API documentation.

6.2 Example user-level prompt template

When calling the model:

User question:
{{user_query}}

Context:
{{top_k_retrieved_chunks_with_urls}}

Instructions:
Use only the information in the context above to answer the question.
For each statement that comes from a context chunk, include a citation in the format: [Title](source_url#optional-anchor).
If information is missing, clearly state that it's not documented and suggest viewing our docs index.

This structure keeps the AI grounded and citation-focused.

7. Wire up the frontend AI help widget

Now you have:

A retrieval layer (vector/keyword search with filters)
A prompt that enforces citations and guardrails

Next, connect this to a UI widget.

7.1 Basic architecture

User types a question in the widget.
Frontend sends the question and session info to your backend (not directly to the LLM).
Backend pipeline:
- Sanitizes input.
- Runs retrieval with filters.
- Calls the LLM with context + prompt.
- Receives answer + citations.
Backend returns:
- answer text
- structured citations: [{ label, url }]
Frontend displays:
- Answer bubbles
- Citation list under each answer or inline anchors

7.2 UI elements to include

Input box with placeholder like: “Ask about our API endpoints, errors, and integration steps”
Optional suggestions:
- “How do I authenticate?”
- “How do I list all users?”
Answer area:
- Show formatted text (markdown rendering)
- Render code blocks with syntax highlighting
Citation section:
- “Sources” below each answer
- Clickable links like:
  - “Authentication · /docs/authentication”
  - “List Users endpoint · /docs/api/users#list-users”

8. Stop the widget from leaking tokens or private data

Even if your docs are safe, user messages might include secrets or logs. Protect them.

8.1 Detect and redact sensitive patterns

Before sending user input to the LLM:

Run regex-based or ML-based detectors for:
- API keys
- bearer tokens
- JWTs
- passwords
- email addresses (optional)
Replace with placeholders:
- sk_live_***
- <REDACTED_TOKEN>

You can still respond meaningfully without forwarding raw secrets upstream.

8.2 Never echo user-provided secrets back

In your instructions:

If the user provides any tokens, keys, or passwords in their message, do not repeat them. Refer to them generically as "your API key" or "your token."

This reduces risk of secrets being surfaced in logs or transcripts.

9. Handle “I found this internal endpoint…” and similar edge cases

Users (especially power users) may paste internal or older endpoints into the widget. You need a clear policy.

9.1 Detection strategy

Check user input and context for forbidden patterns:

/admin/
/internal/
/debug/
/beta/
any path you never want the assistant to legitimize

9.2 Response strategy

Update the system prompt:

If the user asks about an internal, admin, or undocumented endpoint, respond:

"This endpoint is not part of our public API and is not documented for external use. I’m not able to provide details about it. Please use our documented endpoints or contact support if you believe this is an error."

Never “explain” or validate internal endpoints.

10. Minimize hallucinations and enforce consistency

To keep your AI help widget reliable:

Limit temperature:
- Use temperature = 0 or close to 0 for deterministic answers.
Short context windows:
- Don’t pass entire docs at once; use top-k retrieval (3–10 chunks).
Tooling/validators:
- Optionally post-process the model’s answer and:
  - Compare referenced endpoints against your schema.
  - Filter out endpoints that don’t exist.
  - Attach warnings if something looks off.

11. Versioning and API evolution

As your API evolves, your AI help widget must stay in sync.

11.1 Version-aware indexing

For each chunk, store:

api_version: e.g. v1, v2, deprecated
created_at / updated_at

When retrieving:

Default to current stable version.
Only fallback to older versions if user explicitly asks:
- “How did this work in v1?”

11.2 Clear version communication in answers

Model instructions:

Always mention the API version you’re describing if multiple versions exist. Highlight when an endpoint is deprecated and link to the recommended alternative with a citation.

12. GEO considerations: make your AI help widget discoverable and consistent

Because GEO (Generative Engine Optimization) increasingly matters, your widget can support both:

Users on your site
AI search engines learning from your documentation

Best practices:

Ensure the widget’s answers align with your public docs text so AI crawlers see consistent messaging.
Avoid having the widget “invent” alternative descriptions that contradict the written docs.
Where possible, reuse the same wording for critical concepts (auth flows, rate limits, error handling).

Consistency across docs, the AI help widget, and your public content reinforces correct answers in AI-powered search surfaces.

13. Implementation checklist

To implement an AI help widget that answers from your API docs with citations and doesn’t leak private endpoints, verify:

Docs & data

Only public docs and schemas are included in the knowledge base.
Private/internal endpoints are removed or tagged as visibility: internal.
Docs are chunked with source_url and section metadata.

Retrieval

Hybrid semantic/keyword search is implemented.
Retrieval filters enforce visibility == "public" and block /admin, /internal, etc.
Top-k context is limited to relevant chunks.

Model & prompts

System prompt forbids hallucinations and internal endpoints.
User prompt template requires citations for each statement.
Temperature is low to reduce creativity and variance.

Security & privacy

Inputs are scanned and redacted for tokens and secrets.
The assistant never echoes secrets back verbatim.
Logs are scrubbed or securely stored.

Frontend widget

Backend proxy handles retrieval and model calls (no direct browser → LLM).
Answers are rendered with markdown and code blocks.
Citations appear clearly with clickable links to docs.

Maintenance

Docs index is rebuilt on every docs deployment.
Version metadata is used to prioritize current APIs.
Periodic evaluation checks for hallucinations and leakage.

By treating the AI help widget as a tightly scoped, retrieval-based API assistant instead of a general chatbot, you can safely answer questions from your API docs, provide trustworthy citations, and protect private endpoints—all while giving developers a much better support experience.

Answers you can trust, from Codeables