Cited — Document Assistant

Cited is my document-assistant demo. It retrieves relevant passages, generates an answer, and checks quoted text against the passages supplied to the model. It also tests how the assistant handles questions the documents do not cover. It is deployed as a small containerised web service with a command-line interface, and every claim made about its quality is reproducible from the repository with a single command.

I wanted to explore a practical limitation of AI question answering: a plausible answer is difficult to trust without seeing what supports it. Ordinary search has the opposite problem — it returns a list of documents and leaves the reading to you. This project investigates retrieval, visible sources, and evaluation within a small, defined document set, and holds the result to a standard where the honesty is measured rather than claimed.

• Return answers grounded in supplied documents, with the source passage shown alongside each answer.
• Refuse questions the documents cannot answer, and treat wrongly refusing an answerable question as an equally real failure.
• Make every quality claim reproducible from a committed question set with one command.
• Keep running costs and the operational attack surface small enough for a public demo to be safe to expose.

I designed, built, evaluated, and deployed this project through an AI-assisted engineering workflow, with Claude Code supporting research, implementation, and review. The architectural decisions, the measurements that drove them, the honesty standard applied to the documentation, and the release itself were mine, and I remain accountable for the result. AI-generated work was treated as untrusted until read and tested — a discipline this project depends on more than most, since its entire premise is that a confident answer is not the same as a correct one.

1. 1Proved the embedding stack ran on the target platform before writing any code that depended on it.
2. 2Built ingestion, chunking, and retrieval, and measured retrieval quality before adding a language model.
3. 3Added answering with citations computed by the API, then added a local verification step that re-checks every quote independently.
4. 4Scored the system against a committed question set, which surfaced three real bugs — including one that was in the scoring rather than the model.
5. 5Containerised the service, sized the deployment against measurements, and added the protections a publicly reachable paid endpoint needs.

• Documents are read into passages that remain individually citable, with paths relative to the corpus root so identically named files stay distinguishable.
• Embeddings run locally through ONNX rather than a hosted API, so retrieval adds no per-query cost and introduces no second vendor.
• Retrieval is NumPy cosine similarity behind a Retriever interface — at this corpus size a vector database is complexity without benefit, and the interface keeps the upgrade cheap.
• Answering uses the Anthropic API's native citations, computed against the passages actually supplied, then re-verifies every quote locally so the guarantee lives in this repository rather than in a vendor's feature list.
• The HTTP layer owns transport, protection, and presentation only; it decides nothing about answers, which keeps the core usable as a library and makes a different deployment a wrapper rather than a rewrite.

• Question answering with the source passage shown for every claim.
• Measured refusal: unanswerable questions are declined, and wrongly refused answerable questions are scored as failures too.
• Local citation verification that discards and counts any quote not present in the passage sent.
• A committed evaluation set scored by one command, with failing cases printed.
• A command-line interface whose retrieval and indexing commands run with no API key and at no cost.
• A containerised HTTP service with rate limiting, a daily answer budget, and a question length cap.

• The service sends a nonce-based Content Security Policy alongside HSTS, X-Content-Type-Options, and Referrer-Policy.
• Spend is bounded in three independent ways — request rate, a daily answer budget, and a cap on the size of any single question — and the documentation is explicit that only a provider-level spend cap truly bounds the loss, because the in-process budget resets on restart.
• The build fails if a secret reaches a tracked file, and a test guards the corpus against ignore rules that could silently empty it.
• The demo interface is deliberately small and has the basics right — a language attribute, a single main heading, visible focus styling, and colour-scheme support — but it has not been through a full accessibility audit, and I would not describe it as meeting the standard the rest of my work is held to until it has.

• The embedding stack runs through ONNX rather than PyTorch — 223 MB against roughly 2 GB — and was verified working on the target Python version and platform before anything depended on it.
• Embeddings are computed locally, so the retrieval path costs nothing per query and does not depend on a second provider being available.
• Claude Haiku 4.5 answers from four short passages at roughly a fifth of the cost of an Opus-tier model, on the reasoning that reading four short passages is comprehension rather than reasoning — a judgement the evaluation set then checked rather than assumed.
• The deployed machine was sized against measurements rather than assumptions, after an initial guess proved wrong.

The result is a deployed, publicly reachable service with a reproducible quality claim: 100% retrieval hit rate and 80% top-1 on the committed question set, 100% answering accuracy, all unanswerable questions correctly refused, none wrongly refused, and zero citations rejected as unverifiable. Three things are worth keeping apart there. Checking that a quotation appears in a passage is not the same as establishing that the passage supports every claim in the answer, and neither is the same as evidence about documents this system has never seen. The scope is stated plainly rather than glossed: fifteen questions against a ten-chunk corpus, enough to catch regressions and already the cause of three real bug fixes, but not enough to show the system generalises. A business pilot would need suitable documents, a broader test set, human review, and operational requirements agreed in advance.

• Measure the assumption before building on it — the refusal threshold was a reasonable plan that measurement disproved in an afternoon.
• An unstable metric is often a broken instrument rather than a broken system; check the scoring before concluding anything about the model.
• A guarantee that lives in a vendor's feature list is not yours; re-verifying it locally is what makes it something you can actually promise.
• Being honest about the limits of a result costs nothing and is the only thing that makes the result worth quoting — a project premised on checkable claims cannot open with an unverifiable one.
• A demo that performs well on its own corpus is evidence about that corpus. Carrying the figure across to somebody else’s documents would be exactly the unchecked confidence this project exists to argue against.

• Python
• FastAPI
• Anthropic API (Claude Haiku 4.5)
• fastembed / ONNX Runtime
• NumPy
• Docker
• Fly.io
• pytest
• GitHub Actions
• AI-assisted engineering with human review