Plan an internal knowledge assistant with reliable sources

An internal knowledge assistant should help a colleague find an answer they are entitled to see, understand which source supports it, and recognize when the business has no reliable answer yet. That is a more useful goal than building a chatbot that appears knowledgeable about everything.

For a small business, the hard work usually concerns the knowledge itself: conflicting procedures, missing owners, old price sheets, and files shared too broadly. An assistant can make those problems easier to encounter. It cannot make an obsolete procedure current merely by explaining it clearly. This guide shows how to plan a limited, source-based assistant while keeping your business’s own documents, permissions, and operating decisions in control.

Choose a question collection rather than an entire company

Start with a bounded collection of questions. Examples include finding current onboarding instructions, locating approved product specifications, or explaining how staff submit an internal purchase request. These questions have an identifiable audience and a small group of source owners.

Avoid beginning with every company file. A broad collection introduces private personnel records, client material, experimental drafts, and commercial information that many users should never see. It also makes it difficult to establish whether an answer is complete or merely the most convenient passage the system found.

Write a one-sentence purpose that an ordinary colleague can evaluate: help delivery staff locate the current handoff checklist and supporting instructions. Then list excluded purposes, such as answering legal questions or reporting a client’s private account history. The self-service knowledge base guide provides a related method for building useful question-led content before adding conversational search.

Inventory the sources before connecting them

Create a source register with document title, location, business owner, intended audience, status, effective date, and review date. Add a clear distinction between approved operating guidance and material kept only for reference.

A folder name such as “Policies” does not prove that everything inside it is current. Open representative files and inspect whether the owner is still responsible, links work, and the instructions describe the current process. Record conflicts for resolution rather than asking the assistant to choose whichever version sounds more convincing.

Use a small status vocabulary: draft, approved, superseded, and archived. An archived policy might remain available to authorized staff for historical questions, but it should not be presented as the current instruction. The assistant needs enough metadata to distinguish “what did we do last year?” from “what should I do now?”

This inventory is useful even if the business later chooses ordinary search instead of AI. It turns scattered documents into a maintainable knowledge collection with named responsibility. Include a substitute owner for absence, so a broken instruction does not remain unresolved simply because the usual document maintainer is away.

Make the source structure answerable

Write guidance so that a passage can be understood in context. Include the procedure name, audience, purpose, decision limits, and references near the instructions they explain. A table that depends on an unexplained abbreviation or a missing preceding page is difficult to interpret accurately.

Prefer one clear operating instruction to several overlapping documents. Link from the main procedure to supporting detail, and identify which document controls when a summary differs. A short answer should lead the user back to the authoritative source rather than becoming a competing policy.

For example, a hypothetical studio maintains a current “Client file handoff” procedure. Each section identifies the relevant file type, receiving role, completion evidence, and exception route. A paragraph about access does not rely on the reader remembering a different project’s folder rules.

Our document management guide explains the surrounding version, ownership, and access practices. Those practices reduce ambiguity before any model reads the material.

Understand what retrieval adds and what it does not

Retrieval augmented generation combines finding relevant material with generating an answer from that material. Microsoft’s official RAG guidance describes this general pattern using organization-specific knowledge. The pattern can provide a useful source trail, but a retrieved passage can still be outdated, incomplete, or misinterpreted.

Treat retrieval quality and answer quality as separate questions. Did the system find the correct approved instruction? Did it explain that instruction without changing its meaning? A fluent answer to the second question does not prove success on the first.

Decide how much synthesis is appropriate. For a simple operational question, the best response may be a short answer, a source link, and a note about a relevant exception. There is little value in producing a long essay when the user needs to find the right form.

If the source is missing, the assistant should say what it could not establish and identify a next route. General model knowledge should not quietly replace the company’s approved operating rules.

Preserve access at the point of retrieval

The user should only receive material they are authorized to access. This applies to source text, generated summaries, titles, previews, and citations. A confidential document can leak through a descriptive title even when the full file remains protected.

Ask whoever builds the system how permissions are checked for each request. A one-time import by an administrator can copy documents into a new store whose permissions differ from the originals. Deleting access to the source may not remove access to every indexed copy, cache, or saved answer.

Test with accounts representing different roles. A sales user and an operations user may share some documents but have different access to pricing exceptions or client delivery records. Do not use an administrator’s successful search as evidence that ordinary permissions work correctly.

Keep sensitive groups outside the first pilot. Personnel, private client, authentication, and financial account information require careful treatment and usually contribute little to a basic procedure-finding assistant. Narrower AI tool permissions help reduce the consequences of a mistaken answer or an unexpected request.

Define an answer contract users can recognize

Specify a consistent answer structure. For an internal procedure assistant, useful elements include a direct response, the current source title and link, relevant conditions, a brief uncertainty statement where needed, and the route for an unresolved question.

Avoid confidence percentages that users cannot interpret. A model’s confident tone or self-assigned score is not an operational guarantee. Showing the underlying source and limitation is more actionable than presenting a precise number without a validated meaning.

Make the source link open the relevant document where practical. A citation that only names a broad folder creates extra work and makes verification difficult.

Design the unanswered-question route

An unanswered question is useful information about the knowledge collection. Capture the question category, the missing or conflicting source, and the role responsible for resolving it. Do not automatically store the full conversation if it includes unnecessary personal or confidential information.

Give the user a clear message: the approved collection does not establish the requested answer; here is the relevant procedure owner or question queue. Distinguish missing knowledge from missing access. A user should not be told that a private policy does not exist merely because they cannot see it.

Decide which unanswered questions deserve a new article or procedure and which are one-off case decisions. Repeated questions about a standard purchase route may justify better guidance. A request to approve an unusual expense needs a decision, not a new answer invented by the assistant.

Keep the queue manageable by grouping duplicate questions and naming an owner. A growing list of unresolved questions without review becomes another unattended inbox.

Test a hypothetical delivery question from end to end

Consider a hypothetical business that sells downloadable resources. A staff member asks how to replace a customer’s expired download link. The approved procedure requires a verified order and permits regenerating access to the purchased version. An old note describes manually emailing an unrestricted archive.

The expected answer should point to the current verified recovery process. It should not blend the old method with the new one or imply that any file can be sent to any address. The source register marks the old note superseded, so current-answer retrieval should exclude it.

Now repeat the question from an account without access to customer order records. The assistant can still explain the general recovery procedure if that document is available, but it should not reveal an individual purchaser’s order details. The distinction between procedure guidance and case-specific data remains visible.

Finally, remove the current procedure from the pilot collection. The assistant should acknowledge the missing source. If it continues confidently describing the process from memory, the pilot has identified a boundary that needs improvement.

Evaluate questions that should be difficult

Build a small test collection with expected sources and acceptable answers. Include common questions, synonyms, abbreviations, outdated terminology, conflicting files, missing information, and requests outside the assistant’s scope. Test whether users can find the answer when they do not know the official document name.

Include permission tests and misleading content. A file might contain an instruction to reveal another document or to ignore the question’s scope. That text should remain source material, not authority to change the assistant’s job.

Have a source owner evaluate factual accuracy and a typical user evaluate usability. The owner may recognize a subtle policy error that a general reviewer misses. The user may reveal that a technically correct answer still fails to explain the next step.

Track correct source selection, unsupported statements, useful abstentions, permission failures, and the time required to complete the task. Our AI workflow evaluation guide explains how to keep those outcomes separate instead of relying on one impressive demonstration.

Use a question catalog to decide what to improve

Maintain a small catalog of the questions the assistant is intended to support. For each question type, name the user role, authoritative source, expected next action, and a useful example of an answer that should be withheld. This makes the scope concrete enough for a source owner to review.

For example, “Where is the current client handoff checklist?” should lead to one approved source. “Can I skip the client’s approval?” is a different question that may need a decision beyond the assistant’s authority. “What did another client approve?” may involve material the user cannot access. Similar words do not mean these questions belong to the same allowed task.

Use the catalog when reviewing feedback. A question that belongs in scope but has no current source reveals a documentation gap. A question outside scope may need a clearer limitation message. A correctly sourced answer that users cannot act on suggests that the underlying guidance needs better instructions.

Do not automatically add every new question to the supported scope. Each expansion creates source maintenance, permission, and testing work. Ask whether the business has an owner and a reliable answer before promising that the assistant can help.

This catalog can also support ordinary search and a written index. It is an asset the business controls independently of the assistant. If the conversational interface changes or is retired, the mapping between common questions and approved guidance remains useful to staff and to whoever maintains the documentation.

Plan updates, deletions, and retirement

The assistant needs a documented update path. Identify how an approved source enters the collection, how revised versions replace older material, how removed documents leave indexes and caches, and who verifies that the change took effect.

Test a routine revision during the pilot. Change a harmless procedure detail, refresh the source through the normal process, and check when answers begin using the new version. Record any delay that users need to understand. Do the same for a removed document and a revoked permission.

Assign responsibility for broken citations and unanswered questions. A small weekly review may be sufficient for a limited collection, while material that changes frequently may need a different cadence. Choose the cadence from the content’s use and consequences rather than copying a generic schedule.

Include a retirement route. If the assistant is no longer maintained, users need a reliable way to reach the underlying documents. Export the source register and preserve useful approved guidance so the business is not dependent on a conversational interface to understand its own processes.

Decide whether the assistant earns its place

Compare the assistant with a well-organized index and ordinary search. Both alternatives should use the same improved documents. Otherwise the pilot may credit the AI for benefits produced by simply cleaning up the knowledge base.

Measure task completion rather than the number of questions asked. A user who asks five questions because the first four answers were unclear is not necessarily receiving more value. Observe whether people find current guidance, understand the limitations, and finish their work with fewer avoidable interruptions.

NIST’s AI RMF Core emphasizes context, oversight, and ongoing evaluation in a voluntary framework. In a small knowledge pilot, those ideas translate into a named source owner, a bounded audience, repeatable tests, and a decision about whether the system is useful enough to maintain.

The strongest result may be a modest assistant that answers a narrow set of questions reliably. The business keeps control of the sources and can still operate when conversational search is unavailable.

Sources and editorial method

Checked on October 15, 2026. The source register, answer contract, delivery example, and pilot design are original practical analysis. The business example is hypothetical. This article does not claim that any particular assistant was configured, tested, or certified.

• Microsoft Learn: Enhance AI responses by using Retrieval Augmented Generation, for the general retrieval and generation pattern. Implementation-specific permissions must be checked separately.
• NIST: AI RMF Core, for voluntary lifecycle risk management and oversight context.