Scale AI

Scale AI is looking for an AI Research Engineer to join their Enterprise Evaluations team, focusing on building and improving GenAI Evaluation Suites for enterprise LLM-powered workflows and agents. The role involves creating human-rated datasets, designing LLM-as-a-Judge autorater frameworks, and researching new methodologies for evaluating AI systems.

This role focuses on building and scaling automated evaluation pipelines for AI systems, including LLMs and agentic models, to ensure their reliability, safety, and effectiveness in mission-critical government environments. It involves designing test datasets, benchmarks, and frameworks for various metrics, including LLM-judge evaluations, agent testing, and stress tests.