Principal Software Engineer

As a Principal Software Engineer on the Copilot evaluation team, you will own the technical vision for a platform built AI-first and agent-first: one where autonomous agents are not only measured, but also help operate, extend, and heal the system itself. You will take on our hardest problem: delivering trustworthy results across a deep chain of dependencies—including Copilot’s model, retrieval, and scoring services—that we push far beyond their normal operating envelope, with hundreds of calls per job under relentless and growing load. You will architect how we evaluate agentic Copilot experiences, from multi-turn trajectories and tool selection to end-to-end task outcomes, and set the standard for how the broader product defines and measures quality.

Sets the technical direction for the Copilot offline evaluation platform and partners with teams across the Copilot organization to turn ambiguous quality questions into rigorous, reproducible evaluations—the scenarios, metrics, and scorecards that gate what ships.

Leads the architecture of an AI-first, agentic-first evaluation platform in which autonomous agents are first-class operators, designing the services, pipelines, and tooling that agents can operate, extend, and reason about, with the observability and guardrails required to make agent-driven operation trustworthy.

Owns the platform’s hardest challenge: delivering trustworthy results across a deep chain of dependencies pushed far beyond their normal operating envelope, with designs for graceful degradation, intelligent retries, dependency-aware gating, and reliability under sustained and growing load.

Defines how Copilot’s agentic experiences are evaluated end to end, from multi-turn trajectories and tool selection to task outcomes and modalities such as UX and voice, building the simulation, scraping, and scoring capabilities needed to make agent behavior measurable.

Leads by example and mentors engineers across teams to build extensible, maintainable systems, driving modernization of the evaluation runtime and platform so it can scale with rapidly growing demand while improving cost efficiency and latency.

Stays at the forefront of agentic systems, large-scale AI evaluation, and autonomous operations, bringing emerging patterns and practices into the organization and sharing that knowledge to raise the bar for how Copilot measures quality.