Applied Science Manager, Artificial General Intelligence , Quality Automation

Amazon Amazon · Big Tech · Bellevue, WA · Data Science

Applied Science Manager for AGI team focusing on quality automation, auditing, and evaluation of LLMs and multimodal systems. Leads a team of scientists to develop quality strategies, auditing frameworks, and research new methodologies to ensure data integrity and model performance. Manages team development, cross-functional communication, and drives research into data impact and utility measurement for AI models.

What you'd actually do

  1. lead and mentor a team of Applied Scientists who develop comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows.
  2. guide the team in designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that align with core scientist team developing Amazon Nova models.
  3. oversee expert-level manual audits, meta-audits to evaluate auditor performance, and provide coaching to uplift overall quality capabilities across the team.
  4. lead research in areas related to HIL data impact to LLM models, and define utility measurement strategies for data generated by AGI-DS for Nova models.
  5. responsible for recruiting, hiring, and developing team members, conducting performance reviews, setting clear expectations and growth plans, and fostering a culture of scientific excellence and innovation.

Skills

Required

  • 3+ years of scientists or machine learning engineers management experience
  • Knowledge of ML, NLP, Information Retrieval and Analytics

Nice to have

  • Experience building machine learning models or developing algorithms for business application
  • Experience building complex software systems, especially involving deep learning, machine learning and computer vision, that have been successfully delivered to customers

What the JD emphasized

  • ensuring the highest standards of data quality
  • build industry-leading technology with Large Language Models (LLMs) and multimodal systems
  • develop comprehensive quality strategies and auditing frameworks
  • safeguard the integrity of data collection workflows
  • designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies
  • align with core scientist team developing Amazon Nova models
  • expert-level manual audits
  • meta-audits to evaluate auditor performance
  • uplift overall quality capabilities
  • lead research in areas related to HIL data impact to LLM models
  • define utility measurement strategies for data generated by AGI-DS for Nova models
  • quality solution design
  • root cause analysis on data quality issues
  • drive research into new auditing methodologies
  • optimizing data quality
  • quality assurance best practices and standards
  • automated judging systems
  • Knowledge of ML, NLP, Information Retrieval and Analytics
  • Experience building machine learning models or developing algorithms for business application
  • Experience building complex software systems, especially involving deep learning, machine learning and computer vision, that have been successfully delivered to customers

Other signals

  • leading a team of applied scientists
  • developing comprehensive quality strategies and auditing frameworks
  • safeguarding the integrity of data collection workflows
  • designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies
  • align with core scientist team developing Amazon Nova models
  • oversee expert-level manual audits, meta-audits to evaluate auditor performance
  • provide coaching to uplift overall quality capabilities
  • lead research in areas related to HIL data impact to LLM models
  • define utility measurement strategies for data generated by AGI-DS for Nova models
  • recruiting, hiring, and developing team members
  • conducting performance reviews, setting clear expectations and growth plans
  • fostering a culture of scientific excellence and innovation
  • communicate with senior leadership, cross-functional technical teams, and customers
  • collect requirements, describe product features and technical designs, and articulate product strategy
  • lead quality solution design
  • guide root cause analysis on data quality issues
  • drive research into new auditing methodologies
  • find innovative ways of optimizing data quality
  • setting examples for the team on quality assurance best practices and standards
  • work closely with talented engineers, domain experts, and vendor teams
  • put quality strategies and automated judging systems into practice
  • conduct regular 1:1s with team members, provide mentorship and coaching
  • ensure the team delivers high-impact results aligned with organizational goals
  • Knowledge of ML, NLP, Information Retrieval and Analytics
  • Experience building machine learning models or developing algorithms for business application
  • Experience building complex software systems, especially involving deep learning, machine learning and computer vision, that have been successfully delivered to customers