Decision Studio: Claims Model Calibration
Benchmark and calibrate AI claims models against expert-labeled historical data — before they reach production. Built for Sprout.ai.
Overview
Decision Studio is a model calibration platform for InsurTech teams — built as a consulting engagement for Sprout.ai. Claims AI models are benchmarked against historical claims with expert labels: run eval suites, inspect confusion matrices, track accuracy and referral rates, and re-run after each tuning cycle. Non-technical claims teams get the visibility underwriters need without waiting on engineering for every policy change.
Challenge
Insurance AI fails quietly — a model can look fine in demo but drift on real claim types. The challenge was giving claims and ML teams a shared calibration workflow: compare predicted vs actual outcomes at scale, spot where the model confuses coverage classes, and validate improvements before pushing to production.
Outcomes
- Benchmark suites against historical claims with expert ground-truth labels
- Confusion matrix views for predicted vs actual claim classifications
- Overall accuracy, referral rate, and confidence tracking per benchmark run
- Re-run workflow for iterating model changes before production deploy
- Pipeline manager supporting 6 live claim types across multiple carriers
- 93.1% average accuracy across live production pipelines