The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through industry leading generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising.
As the Applied Science Manager for the Continuous Model Evaluation and Learning workstream, you will own the quality backbone for this agentic brand-intelligence system. You will lead a mix of applied scientists and engineers who define what "good" looks like for each brand skill, instrument the system to measure it, diagnose why skills underperform, and close the loop by generating, validating, and deploying improvements. You will deliver the evaluation and remediation framework that attains accuracy targets, enables forward evaluation for skills as they develop, and establishes the autonomous detect-diagnose-remediate loop that lets us scale quality across all brand skills and multiple advertiser-facing surfaces.
This is a business-critical, greenfield initiative within SPB. You will set the scientific charter, grow the talent on your team, and ship the framework that every other brand-intelligence workstream depends on.
Key job responsibilities
- Lead, mentor, and grow the talent on a team composed of applied scientists and machine learning engineers, fostering a culture of scientific excellence, customer obsession, and ownership.
- Own the scientific vision and multi-quarter roadmap for continuous model evaluation and learning across the brand-intelligence system.
- Design and deliver evaluation frameworks for agentic brand-intelligence skills, including LLM-as-Judge rubrics, multi-model ensemble judging, gold-set construction, and calibration against human evaluators.
- Lead development of the optimization engine that programmatically refines prompts, generates synthetic training pairs, and composes agent decomposition strategies (orchestrator-worker patterns) when single-agent skills hit complexity limits.
- Establish rigorous offline-to-online consistency, A/B testing discipline, and drift monitoring so that quality improvements generalize to production traffic.
- Communicate scientific vision, research breakthroughs, and business outcomes to senior leadership, and drive alignment with broader Amazon Advertising objectives.
Basic Qualifications
- 4+ years of applied research experience
- 3+ years of scientists or machine learning engineers management experience
- 3+ years of building machine learning models for business application experience
- PhD, or Master's degree and 6+ years of applied research experience
- Knowledge of ML, NLP, Information Retrieval and Analytics
- Experience programming in Java, C++, Python or related language
Preferred Qualifications
- Experience working on recommender systems or personalization within search, e-commerce, shopping, advertising or other related fields
- Ph.D. in computer science, machine learning, engineering, or related fields, or Master's degree and 4+ years of a quantitative field such as statistics, mathematics, data science, business analytics, economics, finance, engineering, or computer science experience
- Have publications at top-tier peer-reviewed conferences or journals
- 5+ years of scientists or machine learning engineers management experience
- Hands-on experience designing, deploying, and evaluating LLM-based agentic systems, including tool use, multi-step planning and reasoning, retrieval-augmented generation (RAG), and multi-agent orchestration (orchestrator-worker, swarm, critic-actor patterns).
- Experience with large-scale LLM fine-tuning (SFT, RLHF, DPO), prompt engineering, and programmatic prompt optimization frameworks.
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, NY, New York - 202,200.00 - 273,600.00 USD annually USA, WA, SEATTLE - 183,800.00 - 248,700.00 USD annually