Scale AI

Scale AI is seeking a Staff Machine Learning Research Scientist to lead the development of novel evaluation methodologies, metrics, and benchmarks for large language models (LLMs). This role focuses on defining and measuring the capabilities and limitations of frontier LLMs, driving research that informs internal roadmaps and the broader community. Responsibilities include researching existing evaluation techniques, designing new benchmarks, implementing scalable evaluation pipelines, publishing findings, and mentoring junior researchers. The ideal candidate has 5+ years of experience in LLMs/NLP, a strong publication record, and experience leading research teams.

Scale AI is seeking a Tech Lead/Manager for their LLM Evals Research team. This role involves leading a team to develop and implement novel evaluation methodologies, metrics, and benchmarks for large language models, focusing on areas like instruction following, factuality, robustness, and fairness. The position requires research into LLM evaluation techniques, communication with clients and internal teams, implementation of scalable evaluation pipelines, and publishing research findings. The ideal candidate has extensive experience in LLMs, NLP, and Transformer modeling, with a proven track record of research impact and team leadership.

This role focuses on evaluating and interpreting advanced generative AI systems, designing datasets for rigorous evaluation, and contributing to research publications. It involves collaborating with AI labs and platforms to enhance model accuracy and reasoning, positioning it as a key player in the AI evaluation gate.

This role is for a Software Engineer Fellow focused on evaluating and interpreting advanced generative AI systems within the Human Frontier Collective program. The fellow will design datasets, evaluate AI models, and contribute to research publications, aiming to enhance AI accuracy and reasoning.

This role focuses on evaluating advanced generative AI systems by designing domain-specific problems and datasets, providing expert insights to enhance model performance, and contributing to research publications. It's a research-oriented role within a fellowship program aimed at shaping the future of AI.

This role focuses on building and scaling automated evaluation pipelines for AI systems, including LLMs and agentic models, to ensure their reliability, safety, and effectiveness in mission-critical government environments. It involves designing test datasets, benchmarks, and frameworks for various metrics, including LLM-judge evaluations, agent testing, and stress tests.

This role focuses on evaluating and interpreting advanced generative AI systems by designing domain-specific problems and datasets, providing expert insights to enhance model performance, and contributing to research publications. It's a research-oriented fellowship with a focus on AI evaluation.

This role focuses on evaluating and interpreting advanced generative AI systems by designing domain-specific problems and datasets, providing expert insights to enhance model performance, and contributing to research publications. It involves collaboration with AI labs and Scale's research team.

Product Manager for GenAI Test & Evaluation (T&E) in the Public Sector team at Scale AI. This role focuses on defining the vision and roadmap for evaluation capabilities, owning the T&E tech stack to measure and improve agentic applications. Requires strong engineering depth, experience with evaluation systems, problem distillation, ambiguity management, cross-functional leadership, and operational execution. Experience with GenAI implementation, public sector work, and security clearance are preferred.

This role involves a medical professional (MD, DO) collaborating with AI labs to evaluate generative AI systems, focusing on safety, accuracy, and reasoning frameworks within healthcare. The goal is to apply clinical expertise to shape AI decision-making and contribute to research publications. The role is a 6-month independent contractor position with potential for extension.

This role is for a Medical Fellow to collaborate on high-impact projects evaluating and interpreting advanced generative AI systems in healthcare. The fellow will design clinical scenarios, evaluate model safety and accuracy, shape reasoning frameworks, and contribute to research publications. Prior AI experience is not required but is a plus.