When an AI Answers — and When It Doesn't: Observations from a Small RAG Experiment

Technical case study | Preliminary observations | October 2026

Abstract. This case study examines a small document-based question-answering prototype using a demonstration photovoltaic manual. The available records document PDF extraction, character-based segmentation, installation of an embedding library and application activity. Two responses recorded before this editorial review illustrate different behaviors: a source-consistent answer with a page reference and an abstention-style response. Neither response was independently reproduced during this review. The available evidence does not establish the full answer-generation mechanism, the correctness of the abstention, or the meaning of a similarity value of 0.228. The study is a preliminary technical observation, not a model-reliability benchmark.

1. Introduction: Why Answering and Abstention Both Matter

Retrieval-Augmented Generation (RAG) is commonly used to support question answering over external documents. Instead of relying exclusively on information learned during model training, a document-based system can select relevant source passages and use them to inform an answer. This is particularly useful for technical manuals, where a maintenance interval or operating limit must be traced to the actual documentation.

Two behaviors deserve separate evaluation: returning an answer that is consistent with available source evidence, and declining to answer when evidence appears insufficient. Answering every question risks unsupported statements; refusing too often can make an otherwise useful system ineffective. The question is not simply whether the prototype answered, but whether the available evidence was sufficient for that response. This small case provides two previously recorded examples, not an estimate of either behavior's reliability.

2. The Prototype: Document-Based Question Answering

The experimental environment hosted a Python-based prototype for questions about a short demonstration photovoltaic manual. Its extracted text included three technical facts: nominal system capacity of 50 kW, an inverter inspection interval of six months, and an operating temperature range of −10°C to 45°C. These are statements in the demonstration source, not general recommendations for photovoltaic equipment.

Available code extracted text from the PDF, normalized whitespace, segmented the text and retained source page numbers. The extraction produced a single segment from page one. With only one segment, this is a substantially simpler retrieval setting than a large collection containing competing passages, document revisions or contradictory information. The recorded behavior should not be extrapolated to those more complex settings.

3. Technical Setup: Extraction, Chunking and Retrieval

PDF processing and page metadata

The saved processing code used PyMuPDF, imported as fitz, to open the demonstration PDF and extract text page by page. Whitespace was normalized, and each resulting segment retained its page number. This structure can support source attribution, although storing page metadata does not independently establish that every answer's page reference was produced through a verified retrieval-and-citation mechanism.

Chunking

The observed extraction function used a maximum segment length of 500 characters and an overlap of 100 characters. Overlap can preserve context at segment boundaries, but the source produced only one segment, so no overlapping boundary occurred. The experiment cannot establish whether the configured overlap improved retrieval.

Embedding and retrieval configuration

The environment record confirms installation of FastEmbed 0.7.4, a library used for vector embeddings. In a typical semantic retrieval pipeline, text passages and questions are represented numerically and compared using a similarity or distance function. However, installation alone does not prove that FastEmbed, or any particular embedding model, was invoked during the previously recorded responses. The embedding model actually used, retrieval implementation, vector storage, scoring method and thresholds are not independently verified.

Answer-generation configuration

The exact generative model, model version, generation parameters and system instructions are also not independently verified. The observations therefore concern outputs attributed to the prototype as a whole, not the independently established behavior of any named model. No production tests were rerun as part of this editorial review.

4. Previously Recorded Responses: An Answer and an Abstention

Observation A: A source-consistent answer

One response recorded before this review was “Ogni 6 mesi. (Pagina 1)” (“Every six months. (Page 1)”). The extracted source text explicitly stated that the inverter should be checked every six months. The response was therefore consistent with an identifiable source fact, and its page reference matched the recorded extraction page.

This is an observation of source consistency, not independent proof of the full grounding mechanism. The original end-to-end execution trace has not been recovered. It is not possible to isolate the roles of retrieval, prompt instructions, deterministic application processing and generation in producing this output. The response was not independently reproduced during this editorial review.

Observation B: An abstention-style response

A separate previously recorded response stated “Non trovo questa informazione nel documento.” (“I cannot find this information in the document.”). Rather than providing a specific technical claim, the prototype indicated that it could not locate the requested information. This kind of response can be useful when a document genuinely lacks the evidence needed to answer.

However, the exact original question, retrieved context and complete execution trace have not been independently linked to this response. Its correctness is therefore not independently verified. The requested information may have been absent; alternatively, retrieval may have failed, the question may have been misunderstood, or an instruction may have triggered an unnecessary refusal. This response, too, was not reproduced during the review.

Why the contrast is interesting

The contrast highlights an important evaluation distinction. A source-consistent answer should be assessed for factual support and accurate attribution; an abstention should be assessed for whether the evidence was truly insufficient. Two recorded outputs cannot establish the rates of correct answers, justified refusals, missed answers or unsupported claims. They illustrate behaviors that a larger evaluation would need to measure systematically.

5. What a Similarity Value of 0.228 Does—and Does Not—Tell Us

A value of 0.228 was previously recorded as a similarity measure associated with the prototype. Its computation and request-level association have not been independently verified. Retrieval implementations may use cosine similarity, dot products, distances, normalized values or transformed ranking scores. Their scales and interpretations depend on the embedding model, vector normalization and scoring pipeline.

Accordingly, 0.228 cannot be described as 22.8% confidence, as a probability of answer correctness, or as definitive evidence of a poor retrieval match. Nor can it be compared meaningfully with another system's score without verifying the respective methods. A reproducible evaluation would preserve the candidate passages, raw scores, scoring function, embedding model identifier and any decision threshold.

6. Methodology, Evidence and Limitations

The available records support a partial reconstruction of the workflow: PDF extraction, whitespace normalization, character-based segmentation, page metadata and installation of an embedding library, followed by evidence of traffic to a question-answering interface. They do not establish a complete, reproducible execution trace for each previously recorded answer.

An aggregated web-log analysis for October 25, 2026 identified 29 HTTP requests to the question-answering interface from one observed client. These are application requests, not 29 independently validated experiments or successful question-answer pairs. The count cannot be used to calculate accuracy or an abstention rate.

Technical evidence table

The principal limitations are the single small source document, one extracted segment, incomplete original question-response records, unknown runtime configuration and lack of a controlled comparison. The available material cannot support general claims of reliability, hallucination resistance, optimal retrieval or performance attributable to a particular language model. These limitations constrain the conclusions without negating the narrow observations.

7. What a Larger Reproducible Evaluation Would Require

Future work could use a varied document collection, including longer manuals, tables, competing passages and multiple document versions. A labeled question set should contain direct fact questions, paraphrases, ambiguous requests, multi-step questions and questions whose answers are genuinely absent. Each item should specify the expected answer or abstention and, where applicable, supporting passages.

Complete configuration records should preserve model identifiers, embedding settings, retrieval implementation, scoring method, prompts, thresholds and generation parameters. Original questions, retrieved passages, outputs and timestamps should be logged in a sanitized, reproducible form. Controlled repetitions could help identify variability. Evaluation should report retrieval relevance, answer correctness, citation correctness, justified abstention, unnecessary refusal and unsupported claims separately. A system can retrieve a correct passage but generate a wrong answer, or refuse an answerable question; a single overall success rate would hide these differences.

8. Conclusion: Distinguishing Sufficient from Insufficient Evidence

This preliminary case study documents two previously recorded response behaviors from a small document-based question-answering prototype. One response was consistent with an explicit fact in the demonstration document and included a page reference. Another indicated that the requested information could not be found.

Neither response was independently reproduced during this editorial review. The available evidence supports the consistency of the first answer with the extracted source text, but does not establish the complete mechanism by which it was generated. Similarly, the second response demonstrates abstention-style behavior without independently establishing whether abstention was warranted.

The central research distinction is between sufficient and insufficient evidence. A document-based AI system should provide an answer when the available sources adequately support it, and express uncertainty or abstain when they do not. Evaluating that distinction requires examining both the correctness of generated answers and the appropriateness of refusals.

The experiment does not establish general reliability, hallucination resistance or the performance of any particular language model. Its contribution is a limited but documented observation that can inform the design of more rigorous evaluations of source consistency, evidence sufficiency, citation accuracy and abstention behavior.