Cresta

Senior Machine Learning Engineer focused on Automatic Speech Recognition (ASR) and downstream NLP systems. The role involves designing and implementing evaluation frameworks, leading ASR quality improvement efforts, analyzing data, developing ML models, and optimizing pipelines for production systems. Requires strong background in speech recognition, model evaluation, and ML frameworks.

Lead the AI evaluation strategy and systems for Cresta's AI Agent product line, focusing on ensuring reliability, accuracy, and empathy at scale. This role involves designing test plans for LLMs, building automated testing environments, conducting manual UAT, and leading a team of QA analysts. The ideal candidate has strong systems thinking, operational leadership, and a "QA nose" for edge cases, with experience in AI or high-growth SaaS environments.

The AI Quality Assurance Intern will evaluate the behavior, accuracy, and reliability of Cresta’s AI Agents and Agent Assist systems, focusing on AI behavior analysis, intent recognition evaluation, conversation quality assessment, model output auditing, and prompt testing. Responsibilities include executing test plans, validating AI outputs, identifying inconsistencies, and providing feedback for improvement.