For AI engineers and architects designing systems that use business data. This reference explains what each technique does, when it helps and what to verify. Examples and suggested answers describe approaches, not claims about work the reader or author has performed.
Technical review: 30 September 2026
No matching terms. Try a broader phrase or clear the search.
Retrieval and search
RAG
Retrieval-augmented generation supplies retrieved evidence to a model before it answers.
Production judgement: Use it for questions that depend on an external, changing corpus. Evaluate retrieval and answer quality separately; irrelevant evidence can still produce a confident answer.
Link to this termEmbeddings
A model maps content to vectors that represent learned similarities.
Production judgement: Choose using representative queries, language coverage, cost and index size. A model or dimensionality change may require re-embedding the corpus.
Link to this termVector search and ANN
Approximate nearest-neighbour indexes, such as HNSW, find nearby vectors efficiently.
Production judgement: Useful for paraphrases. Benchmark approximate recall and latency; similarity is not proof of relevance, and exact identifiers may need lexical matching.
Link to this termBM25
A lexical ranking function uses term frequency, term rarity and document length.
Production judgement: A useful baseline for exact terms, names and codes. Tokenisation and vocabulary affect results; lexical search can miss paraphrases.
Link to this termHybrid search
Combine lexical and vector retrieval into one candidate set.
Production judgement: Test against both single-retriever baselines. It can balance exact matches and semantic matches, but adds tuning and operational work; it is not mandatory for every corpus.
Link to this termRRF: reciprocal rank fusion
Combine ranked lists using a score such as the sum of 1 / (k + rank).
Production judgement: Rank fusion avoids directly comparing incompatible score scales. The constant k is a fusion parameter, not the number of returned results; 60 is an implementation default in Elasticsearch, not a universal optimum.
Link to this termReranking
A second-stage model scores query–candidate pairs after initial retrieval.
Production judgement: Use when candidate ordering limits quality. Cross-encoders are one option; benchmark relevance gains against latency and cost. A reranker cannot recover evidence absent from the candidate set.
Link to this termQuery rewriting and HyDE
Rewriting reformulates a query; HyDE embeds a generated hypothetical document to retrieve real evidence.
Production judgement: May help underspecified questions, but can drift from the user’s intent. Treat generated text as a retrieval aid, never as a factual source.
Link to this termTop-k and recall
Top-k is the number of selected results; recall measures how much relevant evidence was recovered.
Production judgement: More context can add noise. Separate candidate count, reranking count and final context size, then measure each against labelled questions.
Link to this termChunking and indexing
Fixed-size chunking
Split content into bounded token windows, optionally overlapping.
Production judgement: A reproducible baseline. Overlap increases index size and can duplicate evidence; tables and sentence boundaries need care.
Link to this termStructural and semantic chunking
Structural splits follow headings or paragraphs; semantic splits use topic changes.
Production judgement: Preserve coherent passages, but keep bounds on size. These are related strategies, not interchangeable names for one algorithm.
Link to this termHierarchical chunking
Represent a document as nested sections and passages.
Production judgement: Useful when a precise match needs wider context. Retain stable parent identifiers and keep hierarchy current when content changes.
Link to this termParent-document retrieval
Retrieve a small child passage, then expand to its parent section.
Production judgement: Improves context completeness when small matches omit qualifications. Deduplicate parents and respect a context budget.
Link to this termContextual chunk headers
Attach document titles, section paths or generated contextual descriptions to chunks.
Production judgement: Titles and paths are simple metadata; model-generated context is a separate technique with cost and error risks. Validate whether either improves retrieval on your corpus.
Link to this termMetadata filtering and permissions
Restrict candidates using fields such as tenant, date, product or access policy.
Production judgement: Enforce authorisation server-side during retrieval and subsequent fetches. A prompt instruction or model-chosen filter is not an access-control boundary.
Link to this termGraphRAG and knowledge graphs
Use entities, relationships or graph-derived summaries to support retrieval and synthesis.
Production judgement: Consider it for relationship-heavy or corpus-wide questions. Graph construction, updates and extraction errors add cost; compare with simpler retrieval before adopting it.
Link to this termAgents, tools and context
Agent harness
The runtime around the model manages tools, state, execution limits and recovery.
Production judgement: Design for cancellation, timeouts, permissions and inspectable failures. More autonomy is useful only when the task and operating controls justify it.
Link to this termTool calling
A model requests a structured tool invocation that application code validates and executes.
Production judgement: Validate schemas, identities and permissions at execution. Make mutations idempotent where possible; retries must not duplicate payments or messages.
Link to this termReAct loop
An agent alternates reasoning, actions and observations.
Production judgement: Bound iterations, cost and elapsed time. Keep an execution trace of actions and outcomes; private model reasoning is not required for operational auditing.
Link to this termMCP: Model Context Protocol
A protocol for connecting AI applications to tools, resources and prompts.
Production judgement: Interoperability still depends on supported protocol features and client behaviour. MCP does not grant permission: authentication, authorisation and safe execution remain necessary.
Link to this termMemory
Persist selected information beyond the current interaction and retrieve it when useful.
Production judgement: Define provenance, retention, deletion, tenant isolation and conflict handling. Stored text can be stale or hostile; memory should not silently become trusted instructions.
Link to this termContext engineering
Select and organise instructions, evidence, history and tool results for each model call.
Production judgement: Prompt design remains part of the work. Compaction can lose constraints, so preserve decisions and source references and evaluate long interactions.
Link to this termSkills
Packages of instructions and supporting resources loaded for relevant tasks.
Production judgement: They can reduce repeated context, depending on the runtime. Review their provenance and any executable scripts; packaging does not make instructions trustworthy.
Link to this termMulti-agent systems
Delegate scoped work to agents with separate contexts or responsibilities.
Production judgement: Useful for separable tasks. Compare with a single-agent baseline: coordination, duplicate work, shared-state conflicts and token cost may outweigh parallelism.
Link to this termGuardrails and prompt injection
Controls reduce unsafe inputs, outputs and actions; prompt injection places hostile instructions in untrusted content.
Production judgement: Use least privilege, sandboxing, boundary validation and human approval for consequential actions. Classifiers and prompts alone do not eliminate injection risk.
Link to this termEvaluation and production
Evaluation harness
A repeatable runner tests a system against versioned tasks and scoring rules.
Production judgement: Record model, prompt, retrieval settings and dataset version. Keep development examples separate from held-out tests to avoid tuning to the test.
Link to this termGolden dataset
A curated collection of representative questions, evidence and expected outcomes.
Production judgement: Start with real workflows and documented labels; no single dataset size suits every risk level. Include negatives, ambiguous requests, permission boundaries and failure cases.
Link to this termRetrieval metrics
Recall@k tracks relevant evidence recovered; precision@k tracks relevant results returned. MRR emphasises the first relevant result; nDCG rewards useful ranked order.
Production judgement: Use explicit relevance labels and consistent denominators. A retrieval metric alone cannot tell whether a final answer is correct or useful.
Link to this termFaithfulness and answer quality
Faithfulness asks whether claims are supported by the supplied context; correctness asks whether they are true.
Production judgement: A grounded answer can repeat an incorrect source. Measure relevance, evidence quality and completeness too; automated RAG metrics are proxies needing human checks.
Link to this termLLM-as-judge
A model grades outputs against a rubric.
Production judgement: Calibrate against human labels, test order and verbosity bias, and inspect disagreements. It is a scalable signal rather than independent ground truth.
Link to this termAgent evaluation
Assess task outcomes and the trajectory of tool use that produced them.
Production judgement: Track success, policy violations, tool errors, recovery, cost and time. Check final system state, not just a persuasive final message.
Link to this termObservability and tracing
Connect requests to retrieval, model calls, tool actions and outcomes.
Production judgement: Capture enough to diagnose failures while redacting secrets and personal data. Limit access and retention; logging every raw prompt by default can create a new data exposure.
Link to this termPrompt caching
Reuse eligible computation for repeated prompt prefixes where a provider supports it.
Production judgement: Cache rules, retention and pricing vary by provider. Stable prefixes may help, but caching does not correct answers and should be verified with actual usage metrics.
Link to this termLatency and cost budgets
Define acceptable response time and spend per successful task.
Production judgement: Measure tail latency, retries, retrieval and tool time as well as tokens. Route models only after evaluating task quality; streaming improves perceived responsiveness, not necessarily total runtime.
Link to this termArchitecture and interview questions
Explain the technique, the trade-off and a real example you can defend. Use measured results only when you have them. If you have not used a method, say so and explain how you would evaluate it.
Why not use vector search alone?
Sometimes it is sufficient. Compare it with lexical and hybrid retrieval on your query set, especially identifiers and rare names. Add reranking if ordering, rather than candidate recall, is the bottleneck.
How do you choose chunking?
Begin with a reproducible baseline, then test structural boundaries and parent expansion where context is missing. Compare retrieval and answer quality, index cost and update behaviour.
How do you debug a wrong RAG answer?
Check source correctness and access first, then whether the required evidence was retrieved, retained after reranking and actually included in context. If it was, inspect generation, conflicting evidence and the answer rubric. Preserve the failing case as a regression.
What makes an agent harness reliable?
Explicit tool contracts, permission boundaries, durable state, bounded execution and recovery from partial failure. Explain what happens when a tool times out after applying a mutation.
How do you evaluate an agent?
Use representative tasks with verifiable final states, including failures and adversarial inputs. Measure task success, policy adherence, tool errors, cost and latency, then inspect traces for the reasons behind failures.
How do you reduce hallucinations?
Improve evidence quality, test source support and allow abstention when evidence is insufficient. Citations help users check claims but do not guarantee correctness. Consequential outputs may require qualified human review.
How do you manage cost and latency?
Measure the whole workflow before optimising. Test smaller models for bounded steps, eligible prompt caching, narrower context and fewer unnecessary tool calls. Re-run quality evaluations after each change.
When would you not use RAG?
For a small stable context, deterministic processing or calculations best expressed as code or governed SQL, simpler approaches may work better. Mixed document-and-database questions can still benefit from combining retrieval with structured queries.
A small evaluation example
Illustrative, not a benchmark result. For an internal policy assistant, label a small set of real questions with expected source passages. Add an outdated policy, a missing answer and a document the requester cannot access. Compare lexical, vector and hybrid retrieval; then score source support, correct abstention and permission enforcement. Expand the dataset from failures before drawing broad conclusions.
Primary sources and further reading
Specifications and product behaviour evolve. Check the linked documentation for the version you deploy.
Review history
30 September 2026: first edition. Separates evaluation guidance from universal defaults, distinguishes source support from correctness, and adds permission, privacy and failure-recovery considerations.
For related implementation context, explore AgentData’s governed data access and AgentDesk’s delivery approach. To assess your own use case, review one workflow with Tal.