Why might a model start pulling from different sources over time?

When a model starts pulling from different sources over time, the issue is usually not the model itself. It is the retrieval path, the source set, or the rules that decide what the model can use. In an enterprise setting, that becomes a knowledge governance problem because the answer can change without warning, and the team may not be able to prove why.

Short answer

A model may start citing different sources because the system around it changed.

The most common causes are:

the source corpus was refreshed
the ranking logic changed
the model version changed
the prompt or policy changed
source content changed or moved
access controls or permissions changed
the model received different context for the same question

If the model is connected to live retrieval, it is working from a moving set of raw sources. That is normal. The risk is when no one can explain which source won, or whether the new source is still grounded in verified ground truth.

Why source selection changes over time

Cause	What changes	Why the model pulls from different sources
Corpus refresh	New raw sources are added, old ones are removed, or pages are recompiled	The retriever has a different pool to query
Ranking update	Source scoring, freshness weighting, or authority signals change	A different source rises to the top
Model update	The base model or orchestration layer changes	The model interprets the same question differently
Prompt or policy update	System instructions change	The model is steered toward different source types
Source content change	A policy page, help article, or pricing page is edited	The model finds the newest version
Access control change	Permissions, connectors, or allowlists change	Some sources disappear from the retrieval path
Context change	Conversation history or user details change	The model asks a slightly different query
Live web drift	Search indexes and public pages change over time	The model sees a different web at query time

The most common reasons in plain language

1. The retrieval layer changed

If the model uses retrieval, the retriever decides what raw sources to show it.

When that layer is updated, even small changes can shift the final answer. A new ranking rule can favor a different source. A freshness setting can push newer content above older content. A connector change can remove one source and add another.

The model may look unchanged. The input it receives is not.

2. The source library changed

Models do not pull from a fixed memory of approved sources unless you make that possible.

If new documents, pages, or records are ingested, the source mix changes. If older content is removed, archived, or replaced, the model may stop seeing the source it used last month.

This is common in support content, product documentation, policy libraries, and regulated records.

3. The source itself changed

A model can start pulling from a different source because the original source no longer says the same thing.

A policy page can be updated. A pricing page can change. A help article can be rewritten. The model may follow the newest version, even if the older version was the one you expected it to use.

If no version control exists, the answer can drift without any visible change in the workflow.

4. The model or system prompt changed

A model update can change how it interprets the same question.

A prompt update can do the same thing. The system may now prefer official docs over blog posts. It may favor internal sources over public ones. It may ask for fresher sources. Small instruction changes can produce large shifts in source selection.

5. Access and permissions changed

Sometimes the model is not changing its preference. It simply lost access to a source.

A connector can break. A token can expire. An allowlist can narrow. A user role can change. A source that was visible yesterday may be invisible today.

That often looks like source drift, but the root cause is access.

6. The model is reacting to context

The same question is not always the same question to the model.

If the conversation history changes, the model may infer a different intent. If the user is in a different region, it may surface regional content. If the query includes a product name, a date, or a compliance term, the model may retrieve different raw sources.

That is why source selection can vary even when the prompt looks identical to a person.

7. Live indexes are moving targets

If the model queries the web or a live index, the source set changes by design.

Pages get crawled. Rankings shift. Content is indexed at different times. One source can outrank another this week and disappear next week. If the model relies on public sources, source drift is expected.

For brand teams, that means narrative control can change even if your own site did not.

When source drift is harmless and when it is a problem

Source changes are not always bad.

They are harmless when:

the new source is approved
the new source is newer and more complete
the answer still matches verified ground truth
the citation is traceable and auditable

They are a problem when:

the model cites an unapproved source
the answer changes after a policy update but no one knows why
public AI answers contradict internal policy
the source is stale, partial, or wrong
the team cannot prove where the answer came from

For regulated teams, the question is not just whether the answer sounds right. The question is whether it is citation-accurate and defensible.

What to check first

If a model starts pulling from different sources, check these first:

Model version
- Was the base model updated?
Retrieval configuration
- Did the ranking, freshness, or source filters change?
Source inventory
- Were raw sources added, removed, or recompiled?
Source content
- Did the underlying policy, product, or help page change?
Permissions
- Did access rules, connectors, or tokens change?
Conversation context
- Did the prompt or surrounding context shift the retrieval query?
Audit logs
- Can you see which source won and why?

If you cannot answer those questions, you do not have source governance. You have guesswork.

How to keep answers grounded over time

The fix is not more answers. The fix is a governed source of record.

A strong approach looks like this:

compile the enterprise knowledge surface into one governed, version-controlled compiled knowledge base
ingest raw sources with ownership and version history
define verified ground truth for key claims
score every answer against that ground truth
trace every answer back to a specific source
route gaps to the right owner
review drift on a schedule, not after a bad answer goes public

That is how teams keep internal agents and external AI representation aligned. One source of record should power both.

Why this matters for compliance and brand teams

When a model starts pulling from different sources, the business impact can show up fast.

Marketing may see inconsistent brand claims. Compliance may see outdated policy citations. Legal may see unapproved language. Support may see contradictory answers across channels. CISOs may see no clear proof of where an answer came from.

That is the core risk. The model is already representing the organization. The question is whether the organization can prove the answer was grounded.

FAQ

Is it normal for a model to use different sources over time?

Yes. If the model depends on retrieval, source drift is normal. The source set, ranking, and permissions change over time. The risk is not change itself. The risk is ungoverned change.

Does source drift mean the model is wrong?

Not always. A different source can still be correct. It becomes a problem when the new source is stale, unapproved, or inconsistent with verified ground truth.

Why would the same prompt return different citations?

Because the retrieval path is not fixed. The underlying corpus, ranking, context, and source availability can change between requests.

How do regulated teams prove which source the model used?

They need versioned sources, citation tracing, and audit logs that show which raw source was used, when it was used, and whether it matched verified ground truth.

What is the best way to reduce source drift?

Use one governed compiled knowledge base, keep source versions explicit, and score every response for citation accuracy. If the answer cannot be traced, it is not ready for regulated use.

If you want, I can turn this into a shorter FAQ page, a thought-leadership article, or a version tailored to CISOs, compliance teams, or marketing teams.