How a leading oncology center improved medical retrieval accuracy by 40% — benchmarked across 7 models with FloTorch

The Challenge

A leading oncology center set out to build a clinical AI assistant that could help patients and clinicians access treatment plans, drug options, and evidence-backed medical insights — in real time. The challenge was significant: no ground truth dataset existed, the content was highly domain-specific, and the system had to perform reliably within strict cost and latency constraints. Without a structured evaluation framework, choosing the right retrieval stack was guesswork. 

The Solution

1. Multi-model benchmarking — FloTorch evaluated 7 embedding models across 2,000 queries, measuring NDCG@k, recall@k, cost, and latency across AWS and Azure infrastructure
2. LLM-as-a-Judge evaluation — With no ground truth available, FloTorch deployed LLM-as-a-Judge to generate unbiased pseudo-ground-truth and ensure cross-model fairness
3. Text normalization layer — Discovered that Azure and open-source embeddings were highly sensitive to Unicode inconsistencies; implemented canonical normalization (Unicode standardization, symbol cleanup, spacing) that produced dramatic accuracy gains
4. Cost & latency profiling — Generated a full cost-per-million-query breakdown and latency comparison across both cloud stacks to support production decision-making