Case Study
ClaimGuard-AI
Pre-submission healthcare claim denial-risk engine trained on real CMS CERT audit data (ROC-AUC 0.745, PR-AUC 0.295 vs 0.141 base rate). CARC/RARC crosswalk, expected-recovery-$ queue, model-card page, 76 tests.
Executive Summary
ClaimGuard-AI is a pre-submission risk engine for healthcare revenue cycle management, started at the AIxBio Hackathon and since rebuilt around real data. The model is trained and evaluated on public CMS CERT improper-payment audit files with a temporal split (train 2021-2023, validate 2024, test 2025) — the honest deployment scenario of predicting next year from past years.
Problem & Constraints
Claims get denied over documentation gaps, procedure mismatches, and payer policy violations. Manual scrubbing is slow and ignores expected financial recovery when ordering the review queue.
Architecture
Claim fields + physician note → LLM JSON extraction (validated, with fallback) → XGBoost denial probability → CARC/RARC denial-reason crosswalk → expected-recovery-$ knapsack queue (DuckDB) → Next.js auditor worklist + model-card page.
Methodology
- XGBoost classifier on real CERT claims with temporal train/val/test split — no group leakage
- CARC/RARC denial-reason crosswalk derived from the audit data
- Review queue ranked by expected recovery dollars; the overturn-rate assumption (0.5) is labeled as an assumption in the UI
- Per-claim risk drivers from XGBoost
pred_contribs(no external explainability dependency) - Appeals letters cite the specific driver evidence and carry a human-review banner
- Public model-card page documents data provenance, split, metrics, and limitations
Results & Metrics
| Metric | Result |
|---|---|
| ROC-AUC (2025 test) | 0.745 |
| PR-AUC | 0.295 vs 0.141 base rate |
| Brier score | 0.1096 |
| Test set | 163,940 real CERT claims |
| Tests | 76 (pytest), CI green |
These numbers are modest and honest — the public CERT feature set is thin by construction, and the model card says so.
Tech Stack
Next.js 16, FastAPI, XGBoost, DuckDB, Nebius/Groq LLM APIs, Pydantic
Future Work
Supabase persistence, EHR FHIR R4 integration, richer payer-policy features.