Senior Software Engineer, Agent Eval Platform

Moveworks' AI agents don't just generate text — they act. They plan, call tools, and change real state in enterprise systems on behalf of 5.5 million employees. That makes the central problem of our team an unusually hard measurement problem: how do you score what an agent did — across a multi-step trajectory through a world it changed — precisely enough that the score can teach it to do better?

That signal is what this role owns. You'll build the judgement layer of our agent evaluation platform: the rubrics, the judges, the calibration against human labels, the methodology that makes a score mean something. And the payoff is larger than a report card — a judge good enough to grade a trajectory is a judge good enough to train against. The same calibrated signal that explains why an agent failed becomes the reward signal that stops it failing.

This isn't a pretraining role, and it isn't a testing role. It's applied ML at a point where the methodology genuinely isn't settled: LLMs judging LLMs is an open research problem, and we're working it against agents that take real, irreversible actions in stateful, multi-tenant enterprise environments.

• Build eval orchestration at scale including runtime for multi-turn agent scenarios, scheduling, retries, high-concurrency execution, and run isolation at production dataset sizes

• Lead the move to OpenTelemetry-native observability for the agent platform, defining span data models for agent trajectories and enabling trace context propagation across async boundaries

• Develop stateful simulation environments with fakes of enterprise systems backed by real datastores, including LLM-driven user simulators and scripted state-machine simulators

• Establish reliability floors and SLOs for the evaluation harness and consolidate one-off workflows onto a single orchestration service

• Lay the foundation for using eval signal to optimize the agent, not just measure it