Applied Scientist, Support Agent Intelligence and Evaluation

Amazon Amazon · Big Tech · Seattle, WA · Applied Science

Applied Scientist role focused on building and improving AI models for advertiser support agents, including conversational AI, retrieval, and generation models. The role also involves designing and owning the evaluation framework for agent quality and driving multilingual capabilities. This position is part of the Agentic Intelligence organization, aiming to scale AI-powered experiences for advertisers.

What you'd actually do

  1. Enhance support agent capabilities across the broad suite of Amazon Advertising products — expanding coverage, depth of resolution, and advertiser task completion across Sponsored Products, Sponsored Brands, DSP, AMC, and more
  2. Design and own the evaluation framework for agent quality — including automated LLM-based scoring of answer correctness, confidence calibration, and conversation-level resolution signals
  3. Develop novel metrics that capture whether advertisers actually got the help they needed (beyond surface-level deflection rates)
  4. Build and improve retrieval and generation models that power real-time advertiser interactions under strict latency SLAs
  5. Drive multilingual science — improve non-English resolution rates through cross-lingual retrieval, translation quality modeling, and locale-aware evaluation

Skills

Required

  • building models for business application experience
  • PhD, or Master's degree and 4+ years of CS, CE, ML or related field experience
  • Experience in patents or publications at top-tier peer-reviewed conferences or journals
  • Experience programming in Java, C++, Python or related language
  • Experience in any of the following areas: algorithms and data structures, parsing, numerical optimization, data mining, parallel and distributed computing, high-performance computing

Nice to have

  • Experience using Unix/Linux
  • Experience in professional software development
  • Experience in advertising

What the JD emphasized

  • build models for business application experience
  • patents or publications at top-tier peer-reviewed conferences or journals
  • strict latency SLAs

Other signals

  • build and improve models that power advertiser-facing interactions
  • design and own the evaluation framework for agent quality
  • build and improve retrieval and generation models
  • drive multilingual science