AI Product Management, Copilot

Microsoft Microsoft · Big Tech · Mountain View, CA +2 · Product Management

The Senior AI Product Manager will drive Copilot model capabilities such as tool-use to ensure that the language models that power Microsoft Copilot deliver high quality responses to our users whilst being grounded, reliable, and cost-efficient. This role works at the nexus of product and research, driving execution in partnership with engineers, language engineers, data scientists and researchers. Responsibilities include developing LLM platform strategy, prototyping approaches, identifying and prioritizing issues impacting quality/factuality/safety, defining and building evaluations, defining and deploying experiments for tool use, and partnering with teams to scale tool building and resolve dependencies.

What you'd actually do

  1. Develop and execute on LLM platform strategy for Copilot that extend language model's capabilities.
  2. Prototype approaches by steering language models to drive response quality across a wide range of scenarios.
  3. Identify and prioritize platform, orchestration and language model issues that impact quality, factuality and safety and working with engineers and researchers to find a path to resolution.
  4. Define and build measurable evaluations with relevant datasets to demonstrate quality improvements.
  5. Define, deploy and manage experiments in production that impact language model's tool use, driving measurable improvements in relevance for and engagement with Copilot users.

Skills

Required

  • Bachelor's Degree AND 5+ years experience in product management OR equivalent experience
  • 3+ years of experience leading ambiguous product areas, defining requirements, developing roadmaps, and working with multi-disciplinary teams to execute them
  • 2+ years of experience building ML-powered or LLM-powered products
  • Hands-on experience with LLM APIs (e.g. OpenAI, Anthropic, Azure OpenAI), embeddings, vector databases, and tool use
  • Hands-on experience with prompt design, context window management, and model evaluation

What the JD emphasized

  • tool-use
  • grounded, reliable, and cost-efficient
  • tool building
  • tool use

Other signals

  • driving execution in partnership with engineers, language engineers, data scientists and researchers
  • define and build measurable evaluations
  • define, deploy and manage experiments in production that impact language model's tool use
  • partner with product teams to scale tool building and work with inference, agents and orchestration teams