Product Manager Ii, AI Evaluations

Box Box · Enterprise · Redwood City, CA · Product Management

Product Manager II, AI Evaluations at Box. This role owns the evaluation, selection, and launch of AI models for Box's products, including Box Agents. Requires strong technical skills in Python/SQL, statistical literacy, and experience in product management, particularly in AI/ML evaluations and cross-functional launches. The role involves defining evaluation criteria, driving data generation, managing model launches, and building relationships with model providers.

What you'd actually do

  1. Run rigorous evaluations across frontier models (OpenAI, Anthropic, Google/Gemini) and continuously mature the stack, methodology, and process behind them.
  2. Define what makes a high-quality evaluation dataset and robust, cost-appropriate graders, and translate what end users care about into measurable criteria.
  3. Drive synthetic data generation and data-generation pipelines to ensure they are high-quality, reliable, and scalable.
  4. Own online evaluations, turning real production usage into rich insight on how our product is actually used.
  5. Lead each model launch end to end, managing the process, the stakeholders, and the timeline.

Skills

Required

  • Product management experience (minimum 2 years)
  • Deep knowledge of the AI landscape and frontier models
  • Statistical literacy
  • Comfortable in code (Python, SQL, notebooks) for building evals, running analyses, and prototyping data pipelines
  • Experience driving cross-functional launches
  • Excellent written and verbal communication skills
  • High agency and ability to drive ambiguous problems to resolution
  • Experience with AI/ML products, model evaluation, or benchmarking

Nice to have

  • Experience managing external partnerships with vendors or providers

What the JD emphasized

  • high-agency
  • deeply AI-fluent
  • own AI Evaluations
  • highly cross-functional
  • highly visible
  • front lines
  • assess, launches, and manages
  • rigorous evaluations
  • frontier models
  • own the decision
  • power Box Agents
  • shape how the world's leading model providers understand enterprise performance
  • own the full lifecycle
  • hands-on role
  • deeply knows the AI landscape
  • do their own work
  • statistically literate
  • comfortable in code
  • build the evals
  • run the analyses
  • prototype the pipelines themselves
  • rather than spec them out for someone else
  • technical depth
  • go-to-market execution
  • external partnerships
  • connective tissue
  • thrives in ambiguity
  • moves fast on tight timelines
  • brings order to messy, fast-evolving problem spaces
  • statistically literate
  • comfortable enough in code (Python, SQL, notebooks)
  • build your own evals
  • run your own analyses
  • prototype your own data pipelines
  • driven cross-functional launches
  • excellent written and verbal communicator
  • represent Box externally
  • presenting complex, technical topics simply to senior stakeholders
  • extremely high agency
  • ambiguous, undefined problems
  • driving them to resolution on tight timelines
  • natural disambiguator
  • makes messy, fast-moving AI problems simple and clear
  • Direct experience with AI/ML products, model evaluation, or benchmarking

Other signals

  • Product Manager for AI Evaluations
  • Assess, launch, and manage AI models
  • Rigorous evaluations of frontier models
  • Own decision of which models power Box Agents
  • Statistically literate, comfortable in code (Python, SQL, notebooks)
  • Drive synthetic data generation and data-generation pipelines
  • Own online evaluations
  • Lead each model launch end to end
  • Build and manage deep relationships with model providers
  • Assess compliance posture of production models