1. Decide which questions belong
Start with a defined collection and audience. An internal assistant for operating manuals differs from a public assistant answering product questions. Identify who can ask, which sources are authoritative and what the assistant must not answer. Legal, medical or financial decisions need appropriate professional review rather than a conversational substitute.
Our knowledge-assistant service can cover source preparation, retrieval design, answer instructions, evaluation and handover documentation. Define the intended users and deployment environment during scoping. A successful first boundary might be one approved policy collection; combining every department’s files makes permissions and conflicting guidance much harder to resolve.
2. Understand retrieval before generation
Retrieval-augmented generation, often shortened to RAG, retrieves relevant material and supplies it to a language model when producing an answer. This differs from fine-tuning, which adjusts a model using training examples. RAG is useful when answers need to reflect a document collection that changes and needs visible references.
Keyword search is effective for exact identifiers and specialised terms. Vector search compares numerical representations of text, called embeddings, to find related meanings. Hybrid search combines these approaches. None guarantees the right passage: evaluate retrieval separately from the final answer, otherwise polished wording can conceal that the system searched the wrong material.
3. Preserve structure and access rights
Documents are usually divided into smaller passages for indexing. Keep headings, table context and source locations attached to those passages. A paragraph about an exception can be misleading when separated from the rule it qualifies. Scanned documents may need optical character recognition before their text can be indexed accurately.
For SharePoint content, mirror the access rules that govern the underlying files rather than treating ingestion as permission to expose everything. Microsoft Graph is an API used to access Microsoft 365 resources; its permissions need deliberate configuration. Recheck access when answering, and ensure cached results cannot reveal a document after a user’s access is removed.
4. Require useful citations and abstention
A citation should take the reader to the supporting document and, where possible, the relevant section. A link to a large folder is not enough. Check that cited passages actually support the answer, especially where the model combines multiple sources or encounters different policy editions.
Abstention means declining to provide an unsupported answer. Give the assistant a clear route for missing, conflicting or inaccessible information: explain the gap and direct the user to a named team or source owner. Do not ask the model to guess a likely policy. For consequential tasks, an explicit gap is more useful than fluent certainty.
5. Evaluate ordinary and hostile questions
Build questions that require exact facts, comparisons, context and an admission that the collection has no answer. Include spelling variations, obsolete terminology and contradictory documents. Record retrieval quality, answer faithfulness, citation usefulness and access enforcement separately. The acceptance conditions should match the actual work rather than a generic chatbot demonstration.
Also test prompt injection: instructions embedded in user input or retrieved documents that try to redirect the system. Treat retrieved text as evidence, not as authority to change permissions or disclose secrets. The OWASP guidance for generative AI applications provides relevant security categories to consider during design and review.
6. Plan updates and responsible ownership
An assistant needs a process for adding, replacing and deleting sources. Record who approves a document, how its status reaches the index and how failures are detected. Deleting a source should also address its indexed passages, cached responses and retained copies where those exist.
Compare managed search services and self-managed indexes against access controls, operational capacity and supplier terms. Choose a model only after checking these requirements. For an initial discussion, describe the document collection, its owners and typical questions. Share a small redacted extract only through an agreed channel, not a complete internal knowledge base by ordinary email.
Plan a knowledge assistant