Sr. Applied Scientist, Alexa AI

Amazon Amazon · Big Tech · Boston, MA · Applied Science

This role focuses on building and maintaining evaluation metrics and models for conversational AI assistants, leveraging LLMs for evaluation (LLM-as-a-Judge) and distillation. It involves research into novel evaluation methods, data preparation, model training, and partnering with engineers for infrastructure. The primary output is the evaluation system itself, which acts as a quality gate for the conversational AI product.

What you'd actually do

  1. Own the design, development, and long-term maintenance of flagship quality-evaluation metrics for a state-of-the-art conversational assistant — spanning ground-truth definition, data preparation, model training, and production maintenance.
  2. Research and build LLM-based evaluators, including LLM-as-a-Judge systems, and distill large judge models into efficient, cost-effective models suitable for scaled online use.
  3. Set the technical direction for evaluation science and raise the bar for scientific rigor across the team; mentor scientists and engineers and review their work.
  4. Ensure data quality throughout all stages of acquisition and processing, including data sourcing/collection, ground-truth generation, normalization, and transformation.
  5. Partner with engineers to develop efficient data-querying and inference infrastructure for both offline and online use cases.

Skills

Required

  • PhD, or Master's degree and 5+ years of applied research experience
  • 3+ years of building machine learning models for business application experience
  • Experience programming in Java, C++, Python or related language
  • Experience with neural deep learning methods and machine learning
  • Experience in developing and deploying LLMs in production on GPUs, Neuron, TPU or other AI acceleration hardware, or experience building complex software systems that have been successfully delivered to customers

Nice to have

  • Experience with Machine Learning and Large Language Model fundamentals, including architecture, training/inference lifecycles, and optimization of model execution, or experience with vLLM, SGLang, TensorRT or similar platforms in production environments
  • Research or applied experience in conversational-assistant or LLM evaluation, including ownership of a production quality metric.

What the JD emphasized

  • end-to-end evaluation
  • state-of-the-art conversational AI
  • shipping the models and metrics
  • hands-on experience building Generative AI solutions with LLMs
  • LLM-as-a-Judge
  • model distillation
  • Supervised Fine-Tuning (SFT)
  • Learning from Human Feedback (LHF)
  • production maintenance
  • scaled online use
  • production environments

Other signals

  • end-to-end evaluation of state-of-the-art Conversational AI
  • shipping models and metrics that millions of customers depend on
  • building Generative AI solutions with LLMs
  • design, train, and maintain the evaluation models