Monologue

Monologue is an AI dictation product from every.to — think Whisper Flow. I came in to fix what wasn’t working and now own the model layer that ships to users.

The eval pipeline

Before any improvements, I needed honest measurement. I built the evaluation pipeline first — what’s the model actually getting right and wrong, where, on what kind of input. That eval harness is the load-bearing piece that everything else hangs off.

Iterative prompt optimization

With evals in place, I ran iterative prompt improvement using DSPy and GEPA. Each pass produces measurable lift on the eval set; bad changes get caught immediately rather than after they’re shipped. Significant accuracy jump over the baseline before we ever touched fine-tuning.

Fine-tuning + production

After prompt optimization saturated, I moved to fine-tuning the model itself — synthetic data generation derived from eval failures, hyperparameter sweeps, multiple iterations. The fine-tuned model is deployed in production and serving real users today. I’m still iterating on it daily.

Latency

Big win on serving cost and UX. Comparing the previous deployment to the current fine-tuned model:

Roughly 4–5× P50 improvement and a similarly large drop at the tail.

What I own end to end

• Eval harness and test sets
• Prompt optimization pipeline (DSPy + GEPA)
• Synthetic data generation
• Fine-tuning runs and hyperparameter selection
• Production deployment infra
• Daily iteration loop