Evaluation Lead

Waabi Waabi · Robotics · Toronto, ON +3 · Remote · Autonomy & Algorithms

Lead and build a new centralized Evaluation team responsible for comprehensive analysis of the autonomy system's performance. This role involves designing scalable production frameworks for evaluation sets, metrics, and pipelines, and characterizing autonomy performance for technical and executive teams. The role also involves managing a team of software engineers, data analysts, and data scientists, and contributing as an individual contributor.

What you'd actually do

  1. Lead and build a cross functional team of software engineers, data analysts, and data scientists supporting automated workflows that provide high signal on autonomy performance.
  2. Design scalable production frameworks for sampling evaluation sets, developing and improving metrics, and systematically measuring the performance of both autonomy and the eval ecosystem itself.
  3. Design pipelines, tools, and dashboards to characterize autonomy performance for technical teams and executive leadership, collaborating closely with platform teams on implementation, and autonomy, systems and safety and product teams on requirements.
  4. Work closely with simulation and software teams to build solutions that leverage our data, metrics and simulation platforms effectively.
  5. Lead technical projects; contributing as an IC while also managing the team.

Skills

Required

  • autonomous vehicle industry experience
  • managing high performing teams
  • evaluating AI or machine learning models
  • statistical background
  • working with internal cross-functional partners/stakeholders
  • system design/architecture and algorithms

Nice to have

  • leading Autonomy Evaluation teams
  • large scale databases and analytics

What the JD emphasized

  • Minimum of 6+ years of autonomous vehicle industry experience including at least 2+ years managing high performing teams
  • Experience evaluating AI or machine learning models, ideally in self-driving or related fields

Other signals

  • building a new centralized Evaluation team
  • comprehensive and holistic analysis on all aspects of performance of the autonomy system
  • design scalable production frameworks for sampling evaluation sets, developing and improving metrics, and systematically measuring the performance of both autonomy and the eval ecosystem itself
  • design pipelines, tools, and dashboards to characterize autonomy performance