About the job
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Science Manager with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to lead a team ensuring the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.
Responsibilities
- Lead and mentor a team of Applied Scientists who develop comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
- Guide the team in designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that align with core scientist team developing Amazon Nova models.
- Manage the development and maintenance of LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
- Oversee expert-level manual audits, meta-audits to evaluate auditor performance, and provide coaching to uplift overall quality capabilities across the team.
- Recruit, hire, and develop team members, conduct performance reviews, set clear expectations and growth plans, and foster a culture of scientific excellence and innovation.
- Communicate with senior leadership, cross-functional technical teams, and customers to collect requirements, describe product features and technical designs, and articulate product strategy.
Qualifications
Minimum
- 3+ years of scientists or machine learning engineers management experience
- Knowledge of ML, NLP, Information Retrieval and Analytics
Preferred
- Experience building machine learning models or developing algorithms for business application
- Experience building complex software systems, especially involving deep learning, machine learning and computer vision, that have been successfully delivered to customers