As organizations push AI code assistants into large monorepos and sprawling polyrepo landscapes, retrieval infrastructure — the system that chooses what context gets passed to an LLM — becomes the bottleneck for accuracy, latency, and cost. Teams building repo-aware code assistants for codebases in the 100M+ lines-of-code (LOC) range face hard engineering trade-offs: how to chunk source, how to shard and index those chunks, how often to refresh embeddings, and when to fall back to hybrid retrieval techniques.

Why retrieval design matters at this scale

LLM-based code assistants rely on retrieval to ground generations. With a small repo, you can afford fine-grained chunks, frequent reindexing, and brute-force vector searches. At 100M+ LOC, naive approaches break:

  • Index size explodes: millions of vectors raise storage, memory and search costs.
  • Freshness degrades quickly: frequent commits mean embeddings become stale and introduce hallucinations.
  • Latency budgets tighten: developers expect sub-second to low-second responses during edit-time completions.
  • Relevance metrics (hit@k, recall) become sensitive to chunking granularity and metadata quality.

Chunking: unit of retrieval — tradeoffs and approaches

Chunking determines the atomic piece you store and retrieve. Common units include file, function/method, class, code block, or sliding-token windows. Each has tradeoffs:

  • File-level: Fewer vectors, lower index cost, but lower precision: irrelevant parts of large files pollute context.
  • Function/method-level: Good semantic granularity for most languages; aligns with developer mental model and tests.
  • AST- or semantic-aware chunks: Use language parsing to create meaningful units (e.g., top-level functions, classes, interfaces). Best relevance but more complex to implement.
  • Sliding-token windows: Language-agnostic and easy to implement; high redundancy and index size.

Recommendation: for 100M+ LOC monorepos, favor function/method-level or AST-aware chunking where language tooling is available. Use file-level chunks only for small, cohesive files. Implement a fallback sliding-window chunker for languages lacking robust parsers.

Sharding and index layout

Sharding transforms a single monolithic index into multiple smaller indexes by repository, team, language, or time. Effective sharding reduces query latency and enables targeted freshness policies.

  • Repo-level shards: Natural and simple. Works well if teams mostly query their own repo.
  • Language-level shards: Helpful when embeddings or tokenizers differ by language (e.g., Python vs. Rust).
  • Team/ownership shards: Enables permissions, targeted reindexing, and cost isolation.
  • Temporal shards: Partition by commit timeframe to support efficient delta reindexing for recent changes.

Practical pattern: mix shards — keep repo-level shards (or top-level project shards) plus a “recent-changes” temporal shard for the last N weeks. At query time, search recent shards first for freshness, then fall back to broader shards if needed.

Freshness policies: balancing accuracy and cost

Embeddings age. Commits that change function signatures, docstrings, or semantics make old vectors misleading. Teams use three common strategies:

  • Full reindexing on schedule: Recompute embeddings for the entire repo nightly/weekly. Simple but expensive at scale.
  • Delta reindexing: Track changed files or AST nodes and reembed only deltas. Requires fine-grained change detection, but reduces compute by orders of magnitude.
  • Hybrid: immediate embedding for HEAD commits + lazy background reembedding: New commits get provisional embeddings (e.g., using lighter-weight embedding models) and are reembedded with the production model during off-peak hours.

Recommendation: implement delta reindexing driven by VCS change events and maintain a small “hot” index covering the last 7–14 days. For teams with frequent commits, use asynchronous reembedding pipelines and optimistic background updates to keep latency low while controlling cost.

Hybrid retrieval: metadata, filters and reranking

Pure vector search is rarely sufficient. Combine filters and heuristics:

  • Static metadata filtering: Language, filename patterns, ownership tags, and dependency boundaries drastically reduce candidate sets.
  • Symbol tables and call graphs: Narrow retrieval to transitively related modules (imports/calls) for code-understanding queries.
  • Two-stage retrieval: Use fast BM25/keyword search + lightweight reranker (e.g., a small embedding model) to produce candidates before final reranking with the production embedding model.

Two-stage retrieval reduces vector search cost while improving precision. It also makes it easier to incorporate security or license filters before hitting the LLM.

Evaluation metrics and continuous monitoring

Define clear metrics and test harnesses. Useful metrics include:

  • Hit@k: Fraction of ground-truth relevant chunks appearing in top-k results.
  • Precision@k / Recall@k: Classical IR metrics for relevance.
  • LLM downstream accuracy: Measure whether the assistant’s answers/tests pass after adding retrieved context (unit tests, compilation checks).
  • Staleness impact: Track drift in hit@k as a function of time-since-commit.
  • Latency and cost per query: End-to-end time and compute/embedding costs per retrieval.

Build synthetic queries from historical PRs and capture canonical "gold" contexts (files, functions, tests) to automate evaluation. Continuously monitor and alert on drops in both retrieval and downstream LLM metrics.

Cost and latency considerations

Costs scale with:

  • Number of vectors stored and dimension size
  • Query rate and search replicas
  • Embedding compute for reindexing

Cost controls:

  • Reduce embedding dimensionality or use compact models for the bulk index and reserve high-dim embeddings for hot shards.
  • Employ capped top-k retrieval and early-exit reranking.
  • Use pruning: expire rarely-accessed vectors or heat-map by team/user activity.

Latency strategies:

  • Keep hot shards resident in RAM on dedicated nodes for sub-100ms vector queries.
  • Perform multi-stage retrieval locally (keyword filter) to avoid cross-shard fanout.
  • Cache recent query results and developer-session context aggressively.

Real-world deployment checklist

  1. Pick chunk granularity: start with function-level or AST-aware chunks for supported languages.
  2. Design shard keys: repo + recent temporal shards, plus language shards where needed.
  3. Implement delta-change pipelines: reembed changed chunks and update indices asynchronously.
  4. Adopt two-stage retrieval: keyword filter → lightweight semantic rerank → final vector rerank.
  5. Build evaluation harness: synthetic PR-based queries, hit@k, downstream correctness tests.
  6. Monitor freshness and automate alerts when staleness degrades hit@k beyond a threshold.
  7. Control costs with dimensionality, pruning and hot/cold storage separation.

Case examples and when to deviate

Smaller teams with 50M LOC can get away with coarser shards and nightly full reindexing. At 100M+ LOC, the mixed shard + delta reindexing pattern becomes essential. If your org has strict security or compliance needs, prefer on-prem index nodes with team-level shards to enable access controls and audit trails.

Conclusion

Retrieval at monorepo scale is a systems engineering problem as much as an ML problem. There is no one-size-fits-all solution: the best design depends on commit velocity, team boundaries, language mix, latency targets, and budget. But practical patterns emerge: semantic-aware chunking (function/AST), hybrid sharding that includes a hot temporal shard, delta reindexing, and two-stage retrieval provide the best balance of relevance, freshness, latency and cost for 100M+ LOC codebases.

Teams moving from prototyping to production should prioritize building a measurement pipeline and treating retrieval as the first-class, versioned service in the code-assistant stack. Optimizing retrieval yields outsized gains in assistant correctness and developer trust — the core determinants of adoption.