Mark prompts with data sensitivity tags (e.g., PII, PHI, GDPR) and enforce routing rules that comply with organizational or regional policies. Use Routing Guards to restrict sensitive workloads to specific models, infrastructure, or geographic regions—ensuring compliance without sacrificing flexibility.

Connect to OpenAI, Anthropic, Mistral, Cohere, or your own hosted models using vLLM, TGI, or LMDeploy—through FloTorch’s extensible connector interface. You can define fallback chains, normalize responses across providers, and monitor performance within a unified orchestration layer.

Design dynamic prompt-routing flows using a declarative graph-based syntax. Each node in the graph can classify intent, match embeddings, enforce policies, or call a specific LLM. Easily route tasks like code generation, legal summarization, or creative writing to the right model—based on user profile, input content, or even historical behavior. Routing logic is version-controlled and can be dynamically updated without redeploying your application.

FloTorch natively captures telemetry—such as model latency, token usage, output confidence, and user feedback—and injects it into routing decisions at runtime. These metrics enable adaptive behaviors like switching models during high-latency periods or dynamically adjusting fallback strategies as model performance changes over time.

Define fallback sequences to ensure your users always get a response—even if the primary LLM fails, times out, or returns an error. Every step in the execution path is logged—including retries, fallback triggers, and final outcomes—giving you deep observability into routing behavior and system reliability.

Run A/B tests across models, prompts, or routing strategies using built-in tools. Split traffic deterministically or randomly, monitor key performance indicators (e.g., latency, feedback score, token efficiency), and make data-backed decisions—all without impacting the end-user experience.