Evals jobs

Eval engineering became its own job the day teams noticed they were shipping on vibes: someone has to build the harnesses, curate the datasets, and calibrate the LLM-as-judge pipelines that say whether Tuesday's model beats Monday's. It's measurement science wearing an engineering badge.

The hard part is writing rubrics humans and models can apply consistently, catching benchmark contamination, and keeping eval sets fresh as products drift. Braintrust, promptfoo, and plenty of homegrown harnesses cover the tooling, with statistics doing the quiet heavy lifting. Companies have learned that bad evals cost more than eval engineers, and the listings have caught up.