How to Build an Internal AI Knowledge Base: Four Library Decisions

At many firms, a staff policy question gets answered in a familiar way. Someone searches the shared drive, finds the employee handbook and a later memo that say slightly different things, and then messages the operations lead, who answers from memory when there is time. The weak points are easy to see: the answer depends on who is available, the drive keeps every version of every document, and the few people who know the current rule spend part of each week as an informal help desk. Atlassian's State of Teams 2025 report, based on a survey of 12,000 knowledge workers and 200 executives, found that leaders and teams lose 25% of their time searching for answers, a self-reported figure from a company that sells collaboration software, but a fair indication of scale.

An AI assistant that answers questions from your firm's own documents can take part of that load off the operations lead. Our view, though, is that the quality of its answers depends far less on the model than on decisions your own people make about the documents it reads. We call the result a governed library: a defined set of document collections where the firm has decided which version of each document counts, who may see each collection, who maintains it and how documents leave it, and which questions it must answer or decline. Below we explain why, compare two ways your firm can start, follow one policy question before and after, and work through the four decisions involved.

Why the model is the smaller part of an internal knowledge base

Internal knowledge tools work by retrieval, which means the tool first finds the passages in your documents that look relevant to a question and then writes an answer from them. So the answer can only be as current and correct as the passages it finds, and the citation attached to it shows where the text came from, not that the conclusion is right. Retrieval can reduce errors compared with a general chatbot, but it does not remove them.

One example comes from legal research. In a preregistered study published in 2025 in the Journal of Empirical Legal Studies, researchers at Stanford and Yale ran 202 legal queries through retrieval-based research tools from LexisNexis and Thomson Reuters between March and May 2024. Depending on the tool, between 17% and 33% of responses were hallucinations, meaning they contained incorrect information or falsely claimed that a source supports a statement. The study covers case law and legal guidance rather than internal documents, its queries were built to find weaknesses, and its results describe the versions tested in 2024, so it is not an error rate for your system. It does however show that a cited answer from a carefully built product can still be wrong.

Microsoft's guidance on evaluating retrieval systems describes the same failure. An answer can be grounded, meaning it is based on the retrieved text, and still be incorrect, which Microsoft describes as "referencing the correct source material but drawing incorrect conclusions." When correctness is low, the guidance says to check first that the retrieved content is itself correct and, if needed, to "correct any problems in the source documents." So when answers are wrong, the library is one of the first places to look.

Connecting the whole drive versus building a governed library

You can start in one of two ways. The first is to switch on an assistant over the existing drive or the firm's Microsoft 365 environment and let search sort out what is relevant. It can be quick if the assistant is already available, but it can surface old versions and files shared too widely. Microsoft's Copilot privacy documentation states that Copilot "only surfaces organizational data to which individual users have at least view permissions," and asks firms to use the permission models in services such as SharePoint to keep access right. Our reading is that oversharing already on the drive becomes oversharing in answers, because the assistant can find and summarize files a person could open but would never have looked for. Microsoft's deployment blueprint for Copilot lists "remediate oversharing" as the first of its three pillars.

The second way is to start with one or two collections the firm has decided on, such as current policies, procedures and approved templates, each with a named owner and a clear audience. It takes longer to begin, because people make decisions before anything is connected. We think most firms will try the first option because it is already available, but that the governed library is the likelier path to answers staff trust. In our judgment, a firm that starts with the whole drive is likely to end up building a governed library anyway, after the first answer that quotes a retired policy or surfaces a document shared too widely.

One staff policy question, today and with a governed library

Here is a hypothetical question that comes up at many firms: "Can I work from another country for three weeks, and who has to approve it?"

• Today. The employee finds the 2023 handbook, which allows remote work abroad with manager approval, and a later memo that requires sign-off from the operations lead and a tax and immigration check. They message the operations lead and wait two days, and the answer depends on who replies.
• With an assistant over the whole drive. The assistant retrieves both documents but writes its answer from the handbook, cites the section number and says manager approval is enough. The citation looks authoritative, so the employee books the trip, and the operations lead finds out afterward.
• With a governed library. The handbook section has been retired, and only the current policy is indexed, meaning added to what the assistant can search, along with its effective date. The assistant cites that section and date and names the operations lead as the approver. Because the policy does not settle the tax or immigration position for a specific country, the assistant says so and refers the employee to the policy owner.

The model is the same in the second and third versions, so the difference comes from the library. The employee gets the current rule with its source, the operations lead handles only the part that needs judgment, and the policy owner learns that the question came up when the employee follows the referral. Each improvement comes from one of the four decisions below.

Decide which documents are in the library and which version is current

Start from the questions staff actually ask, not from the folders you happen to have. Collect recent questions from messages and email, group them by topic, and inventory the documents that answer each group. Where two documents disagree, put the contradiction in front of the person accountable for that topic and let them decide which one counts. Only the current version goes into the library, marked with its date. The same rule applies to any AI system that answers from your records, as we explain in build, buy, or fix the process first.

Most of the early work sits in this step, and it is easy to underestimate. We expect to find two live versions of the same policy as a matter of course, which is why the inventory is its own step in our knowledge systems work. Microsoft's evaluation guidance points the same way, noting that "source quality is critical" and that "curated and vetted documents reduce the risk" of inappropriate content in answers.

Set who may see what before anything is indexed

An indexed document can appear in any answer the system's access rules allow, so the system has to check whether the person asking may read a document before it uses that document in an answer. Those rules have to be set for each collection before indexing rather than repaired afterward. The OWASP Top 10 for LLM Applications 2025, a security community's list of the main risks in AI applications, covers weaknesses in the stores that retrieval systems search. It warns that "inadequate or misaligned access controls can lead to unauthorized access" to sensitive information, and that where several users share one store there is "a risk of context leakage between users or queries." It recommends storage that enforces access permissions and classifying content in the knowledge base by access level. We apply the same thinking to an agent that reads a shared inbox in which inbox actions to hand an AI agent first.

For your firm, the practical boundaries are usually private HR records, partner-only finance and each client's files. We recommend starting with collections everyone may read, such as firm policies, procedures and approved templates, and keeping anything client-specific in separate collections with their own access rules until those harder decisions are made.

Name an owner for each collection and a way to retire documents

A library loses accuracy quietly, because a superseded policy stays indexed and keeps getting cited until staff act on an outdated answer. Each collection therefore needs a named owner who decides what goes in and comes out, a refresh date, and a documented retirement step that someone on staff can carry out without calling the vendor. Microsoft's guidance is direct on this point: evaluating a retrieval system "isn't a one-time operation," because "your collection of documents changes over time" and so do the questions people ask.

Our case study of a knowledge system for a growth consultancy shows source-checking and review routines in practice. Each client workspace explains where to find particular information and which source to use when tools define or report it differently. Information is shown with its date, the dashboard highlights old data automatically, and staff can mark an item resolved, dismiss it or explain why it is inaccurate.

Grade the answers against a question set before launch, then sample them after

Before staff use the system, write a set of questions with known answers and add questions outside the library that the system should decline. Grade each answer on whether it is correct, whether the cited passage supports it, and whether the system declined when it should have. Declining questions the library can answer is also a failure. In the Stanford study, Ask Practical Law AI gave incomplete answers on more than 60% of the 202 queries, a category that included refusals, answers that missed the question and factual claims without citations.

Because model answers vary from one run to the next, Microsoft recommends using "a target range instead of a single target" when you evaluate. Agree that range with the collection owner before launch, so the decision to go live rests on a standard set in advance. After launch, a person reads a sample of recent answers on a schedule and reruns the question set after any change to the library or the tool. In the consultancy case study, reviewers can repeat saved test examples after editing a skill, a set of reusable instructions for an AI task, and compare the answers, so they can check for a change in behavior before the team relies on it.

What we recommend before you start building

1. Start from the questions, not the drive. Collect the questions staff asked in the last two months from messages and email; we suggest gathering thirty to fifty as a first pass at finding the main topics. Build the first collection only from the documents that answer the largest group.
2. Name the owner and set access before anything is indexed. For that first collection, write down who decides what goes in and comes out, who may read it, its refresh date and the steps for retiring a document. Leave private HR records, restricted finance documents and client files out of the first version.
3. Grade it before staff use it, and keep grading. Turn the collected questions into a graded set with known answers, add questions the library should decline, and agree the passing range with the collection owner. Then book a regular slot, such as a monthly half-hour, for that owner to read a sample of answers, and rerun the set after any change to the library or the tool.

Questions buyers ask

Should we train or fine-tune an AI model on our documents instead of using retrieval?

For most firms, no: we recommend retrieval first. Retrieval keeps your documents separate from the model, so when a policy changes you update the document and its search index rather than retraining anything, and answers can point to the passages they use. Fine-tuning means further training the model itself, so on its own it does not give you a separate library where you can inspect the source of an answer or remove a retired policy. Start with retrieval over a governed library, and consider fine-tuning later only for a narrow need such as your firm's writing style.

Can we use the AI search already built into Microsoft 365 or Google Workspace instead of building something?

Often yes, as long as you still do the library work. Microsoft's assistant, for example, answers from whatever each user can already open, so it reflects the state of your drive, including old versions and broad sharing. Before you switch on any built-in assistant, read the vendor's documentation on how it applies permissions and fix oversharing in the folders it will search. A practical first step is one clean site or shared folder that holds only current policies.

Should client files go into the same knowledge base as our internal policies?

We recommend keeping them separate, at least at first. Internal policies are usually readable by everyone, while client files carry their own confidentiality terms and a smaller group of people who may see them, so a shared store needs access rules that stop an answer from drawing on a document the person asking should not see. Build the policy collection first, then add client material as separate collections, one per client or matter, each with its own access rules and owner.

How much staff time does it take to keep the knowledge base current after launch?

It depends mainly on how often the documents change and how many answers the owner reviews. The owner's time goes to reviewing new and retired documents, reading a sample of answers and rerunning the question set after changes. We suggest starting with a regular block for each collection owner, such as half an hour a month to read a sample of answers, plus time to update documents and rerun the question set when something changes. Adjust that block after the first quarter, based on how many corrections the samples turn up and how long the reviews take. The larger cost comes when nobody owns a collection, because errors build up until staff stop trusting the answers.

An internal AI knowledge base works when staff trust it for routine answers and know when to ask a person, and that trust comes from the library your firm governs more than the model it buys. If you want help making those decisions and building on them, see our knowledge systems work.