Amazon

The Data Scientist will focus on developing evaluation and benchmarking datasets for generative AI capabilities within the Amazon Quick Suite enterprise AI platform. This includes leveraging LLMs for synthetic data generation, creating ground truth datasets, leading human annotation initiatives, and contributing to Responsible AI efforts to ensure enterprise-readiness, safety, and effectiveness of AI at scale.

The Data Scientist II will focus on developing evaluation and benchmarking datasets for enterprise AI features, specifically for Amazon Quick Suite. This involves leveraging Generative AI techniques, LLMs for synthetic data generation, and LLM-as-a-judge settings to assess model performance, ensure data quality, and contribute to Responsible AI initiatives. The role also includes building scalable data pipelines and tools for continuous evaluation.

The Data Scientist will focus on developing evaluation and benchmarking datasets for generative AI capabilities within the Amazon Quick Suite enterprise AI platform. This includes leveraging LLMs for synthetic data generation, creating ground truth datasets, leading human annotation initiatives, and contributing to Responsible AI efforts to ensure enterprise-readiness, safety, and effectiveness of AI at scale.

The AI Principal PMT for Alexa Responsible AI will define the standard for how Alexa earns and keeps customer trust. This role owns the product discipline of Responsible AI, defining customer experiences for safety guardrails, trust signals, and evaluation frameworks. The PMT will set product vision and strategy, lead cross-functional alignment across Applied Science, Engineering, Legal, Policy, and UX, and ensure the full responsible product experience including safety, privacy, and security. The role requires technical depth in LLMs and AI safety, understanding how models fail and writing requirements for safety model development and evaluation system design. The PMT will also mentor other PMs and influence Responsible AI scaling across Alexa.

Manager, Program Management for Alexa Sensitive Content Intelligence (ASCI) team, focusing on shaping how Alexa protects customers from harmful content using generative AI and responsible AI guardrails. The role involves strategic leadership, cross-functional program delivery, and team building, with a strong emphasis on data and LLM fluency, defining and executing roadmaps for responsible AI, and ensuring program execution through metrics and mechanisms.

Applied Scientist role focused on automated reasoning, privacy, and sovereignty within AWS. The role involves solving complex problems, designing and implementing solutions, and providing cross-organizational technical influence. Requires a PhD or Master's with significant applied research experience in areas like SAT, SMT, theorem proving, symbolic simulation, program analysis, or type systems. Experience with specific programming languages like O'Caml, Dafny, Haskell, Lean, or Rust is preferred.

Senior Software Development Engineer role focused on applying Automated Reasoning to verify Generative AI outputs, specifically addressing hallucinations within AWS services. The role involves designing and building new services and capabilities at scale, contributing to the evolution of the Automated Reasoning Checks (ARc) service, and making automated reasoning more accessible within AWS.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

Senior Applied Scientist role focused on developing evaluation frameworks and data collection protocols for robotic capabilities. The role involves designing how to measure, stress-test, and improve robot behavior, building infrastructure for teleoperation, evaluation, and learning, and analyzing results to identify performance gaps. It requires expertise in robotics, ML, and human-in-the-loop systems, with a focus on turning capability goals into measurable evaluation systems.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. The primary responsibilities involve benchmarking AI models, evaluating audits performed by a core auditing team, improving audit consistency, and enforcing quality standards. The goal is to scale AI model evaluation coverage and ensure high-quality outcomes for sellers.

This role focuses on evaluating AI systems, specifically LLMs, by designing and executing benchmarking and audit activities. The core responsibilities include assessing model quality, compliance, robustness, and fairness, as well as handling annotations for training and measuring AI models. The role also involves preparing audit reports and ensuring data quality.

The AI Benchmarking Lead will focus on ensuring the quality and reliability of AI model evaluations for Amazon's Seller Assistant copilot. This role involves benchmarking AI models, evaluating audit processes, improving audit consistency, and enforcing quality standards to support the scaling of the product to a wider seller base.

This role is for a Software Development Manager on the Agentic AI organization's Evaluations & Optimization team at AWS. The manager will lead a team of engineers to build systems for assessing the quality, performance, and reliability of GenAI and agentic systems, as well as optimization solutions. The work involves deep learning, distributed systems, and evaluation science, focusing on building infrastructure and tooling for evaluation workflows.

The Applied Scientist will develop and maintain LLM-as-a-Judge systems and auditing frameworks to ensure the quality of data used for training and evaluating Amazon Nova models, impacting LLM products and services.

Applied Science Manager for AGI team focusing on quality automation, auditing, and evaluation of LLMs and multimodal systems. Leads a team of scientists to develop quality strategies, auditing frameworks, and research new methodologies to ensure data integrity and model performance. Manages team development, cross-functional communication, and drives research into data impact and utility measurement for AI models.

This role focuses on evaluating AI systems, specifically LLMs, by designing and executing benchmarking and audit activities. It involves assessing model quality, compliance, robustness, and fairness, with a strong emphasis on handling annotations for training, measuring, and improving AI models. The role also includes preparing audit reports and ensuring data quality based on annotation guidelines.

This role focuses on evaluating AI systems, specifically LLMs, by designing and executing benchmarking and audit activities. It involves assessing model quality, compliance, robustness, and fairness, as well as handling annotations for training and measuring AI models. The role also includes preparing audit reports and ensuring data quality.