Director, Research - AI Evals

Figma Figma · Enterprise · Canada +1 · Design

Director of Research focused on owning AI evaluation methods and operations for Figma's AI-powered experiences. This role involves defining quality dimensions, building evaluation frameworks, establishing quality bars, and producing signal for product teams to make shipping decisions. It requires deep experience in evaluating AI/LLM products, blending human and automated approaches, and partnering with cross-functional teams. The role also includes managing a small team.

What you'd actually do

  1. Own AI evaluation methods and operations for Figma's AI-powered experiences — define quality dimensions, design how we measure them, and turn results into decision-ready signal
  2. Build and maintain evaluation frameworks, rubrics, golden datasets, and quality bars, combining human evaluation with automated/model-based approaches (e.g., LLM-as-judge) where appropriate
  3. Partner with engineering to stand up repeatable, reproducible evaluation pipelines and regression testing, so evaluation is a routine part of how AI features are built and shipped
  4. Produce clear readouts and dashboards that let stakeholders confidently make go/no-go and prioritization decisions
  5. Socialize a shared definition of quality so evaluation standards are adopted across teams rather than re-invented — and advocate for evaluation as a strategic partner in the product process

Skills

Required

  • 10+ years of experience in product, research, applied research, or a closely related field
  • 2+ years of management experience
  • Direct, hands-on experience owning the evaluation of AI/LLM-powered products
  • Expertise designing and running AI evaluation — human evaluation programs, rubric and benchmark/golden-dataset construction, inter-rater reliability
  • Sound judgment about when and how to apply automated/model-based approaches (e.g., LLM-as-judge), including their limitations
  • Strength across both qualitative and quantitative methods
  • Comfort with data and metrics
  • Ability to reason about model behavior
  • Demonstrated success in identifying the riskiest assumptions behind an ambiguous quality question, prioritizing them, and designing right-sized evaluation to build confidence
  • A proven track record of gaining buy-in from executive and cross-disciplinary stakeholders

Nice to have

  • Experience building or co-building automated evaluation pipelines and regression testing in partnership with engineering
  • Familiarity with eval tooling (e.g., Braintrust, LangSmith, DeepEval, or equivalents)
  • Experience standing up a new function, practice, or discipline from scratch
  • 2+ years in product design, user-centric product management, data science, product development, and/or front-end engineering
  • A familiarity and depth of experience using Figma's products

What the JD emphasized

  • Direct, hands-on experience owning the evaluation of AI/LLM-powered products
  • Expertise designing and running AI evaluation
  • A proven track record of gaining buy-in from executive and cross-disciplinary stakeholders

Other signals

  • AI evaluation methods and operations
  • Build and maintain evaluation frameworks, rubrics, golden datasets, and quality bars
  • Partner with engineering to stand up repeatable, reproducible evaluation pipelines and regression testing
  • Produce clear readouts and dashboards that let stakeholders confidently make go/no-go and prioritization decisions
  • Manage a small team to execute our AI evals