Executive Summary

We researched how to reduce end-to-end latency in retrieval-augmented generation (RAG) for domain-specific technical documentation without sacrificing answer quality.

The winning design combines boundary-aware adaptive chunking, cheap first-stage vector retrieval, and a lightweight reranker. It reduces p95 latency by 67% while increasing retrieval quality.

Abstract

RAG systems frequently fail in production due to slow retrieval and noisy context. Classic fixed-size chunking (e.g., 512 tokens) is simple but misaligns with document structure and produces poor semantic boundaries. We propose a hybrid approach: adapt chunk sizes to semantic boundaries, reduce the candidate set with metadata filters, and rerank only a narrow shortlist.

We evaluate latency, precision, recall, and cost per query on an 850-query benchmark.

Problem Statement

The system must satisfy three constraints simultaneously:

• Latency: sub-200–250ms retrieval p95 to keep interactive UX responsive.
• Quality: maintain or improve precision/recall under real queries.
• Cost: avoid expensive reranking across large candidate sets.

Baseline

• Corpus: 12,500 technical documents (avg. 2,800 tokens each)
• Embeddings: 1536-dimensional
• Chunking: fixed 512 tokens with 50-token overlap
• Retrieval: top-k vector search, no reranking
• Baseline p95 latency: 640ms

Approach

1) Adaptive Semantic Chunking

We segment documents into 200–800 token chunks aligned to structure (headings, lists, code blocks, table boundaries). The goal is to keep each chunk internally coherent while preventing oversized chunks from harming recall.

2) Candidate Set Reduction via Metadata

Before vector search, we apply metadata filters (doc type, product area, recency) to reduce the candidate pool. This reduces wasted retrieval on irrelevant content.

3) Two-Stage Retrieval

1. Stage A: fast vector search over filtered candidates
2. Stage B: rerank the shortlist with a lightweight model tuned for semantic match vs. cost

Experimental Methodology

We tested:

• 5 chunking strategies (semantic boundary detection, sliding windows, hierarchical variants)
• 3 reranking models under compute and latency budgets
• Ablations: no metadata filter, fixed chunks, rerank-all, rerank-top-n

Primary metrics:

• Precision@5, Recall@10 (850 test queries)
• End-to-end retrieval latency (p50/p95)
• Cost per query (inference + infra)

Results

Winning Configuration

• Adaptive semantic chunking (200–800 tokens, boundary-aware)
• Two-stage retrieval: vector shortlist + lightweight reranker
• Metadata filtering pre-applied to reduce candidate set by 73%

Quantitative Outcomes

• Latency (p95): 640ms → 210ms (-67%)
• Precision@5: 0.68 → 0.84 (+24%)
• Cost per query: $0.0043 → $0.0011 (-74%)

Operational Observations

• Boundary-aware chunks significantly reduced “context drift” (retrieved paragraphs that were syntactically similar but semantically unrelated).
• Metadata filters helped most on mixed corpora (policy + technical + operational docs).

Deployment Notes

• Keep chunking deterministic and versioned so retrieval behavior is reproducible.
• Store chunk provenance (doc id, section path) so citations and debugging remain straightforward.
• Use guardrails to prevent reranking costs from scaling with corpus size (strict shortlist limits).

Commercial Application

This research maps directly to production outcomes:

• Faster customer-support and operator-assist experiences
• Lower per-query cost at scale
• More stable behavior under growing corpora (less degradation with size)

Licensable Outcomes

1. Adaptive chunking library (Python, Apache 2.0): boundary-aware segmentation and chunk provenance
2. Reranker integration framework (TypeScript): pluggable two-stage retrieval architecture
3. Evaluation harness: 850-query test suite with automated regression checks

Limitations and Next Work

• Domain-specific structure detection can be improved for PDFs and scanned documents.
• Future iterations can incorporate user feedback signals (clickthrough, resolution rate) as weak labels.

Evaluation Date: August 2025
Status: Production-ready, licensed to 2 commercial partners