Senior Machine Learning Engineer

Cresta Cresta · Vertical AI · AB, Canada, Canada · Remote · Engineering

Senior Machine Learning Engineer role focused on building and scaling next-generation agentic AI systems and evaluation frameworks for contact center applications. The role involves designing multi-step agent workflows, RAG pipelines, and optimizing LLM-powered systems for production, with a strong emphasis on improving reliability, robustness, and performance.

What you'd actually do

  1. Lead the design and development of Cresta’s next-generation AI Agents and Agentic Assist systems, defining system architecture and core modeling approaches.
  2. Architect intelligent, multi-step agent workflows that combine real-time guidance, knowledge retrieval, reasoning, summarization, and automated actions into cohesive production systems.
  3. Design, deploy, and optimize LLM-powered systems, including Retrieval-Augmented Generation (RAG) pipelines, multi-agent orchestration, and domain-adapted models.
  4. Develop evaluation strategies for complex, non-deterministic systems, including offline benchmarking, online experimentation, and LLM-as-a-judge methodologies.
  5. Diagnose and mitigate real-world failure modes such as hallucinations, retrieval errors, tool misuse, prompt brittleness, and multi-step reasoning breakdowns.

Skills

Required

  • LLMs
  • Prompting techniques
  • RAG
  • Multi-agent orchestration
  • Evaluation frameworks
  • NLP
  • Generative AI
  • Transformer architectures
  • Embeddings
  • Retrieval systems
  • PyTorch
  • TensorFlow
  • Hugging Face
  • Distributed/cloud-based infrastructure
  • ML system optimization
  • Technical leadership

Nice to have

  • Master’s or Ph.D.

What the JD emphasized

  • strong pre-LLM ML foundations
  • deep expertise in LLMs
  • proven ability to translate cutting-edge research into scalable, production-grade systems
  • design evaluation frameworks
  • diagnosing and mitigating failure modes
  • defining measurable quality metrics
  • architect and scale LLM and retrieval-augmented generation pipelines
  • ground models in enterprise data
  • building high-performance ML systems
  • extract structured insights
  • deliver real-time, actionable intelligence at scale
  • multi-step agent workflows
  • knowledge retrieval
  • reasoning
  • summarization
  • automated actions
  • cohesive production systems
  • Retrieval-Augmented Generation (RAG) pipelines
  • multi-agent orchestration
  • domain-adapted models
  • reasoning, planning, and tool-use capabilities
  • real-world AI applications
  • evaluation strategies for complex, non-deterministic systems
  • offline benchmarking
  • online experimentation
  • LLM-as-a-judge methodologies
  • hallucinations
  • retrieval errors
  • tool misuse
  • prompt brittleness
  • multi-step reasoning breakdowns
  • accuracy
  • faithfulness
  • task completion
  • latency
  • cost
  • robustness
  • scalability
  • latency
  • security
  • cost efficiency
  • production environments
  • Bachelor’s degree in Computer Science, Mathematics, or a related field; Master’s or Ph.D. preferred.
  • 5–8+ years of industry experience building and deploying machine learning systems in production, including significant experience working with LLMs.
  • Strong expertise in NLP, Generative AI, transformer architectures, embeddings, and retrieval systems.
  • Proven experience designing and deploying Retrieval-Augmented Generation (RAG) systems in enterprise environments.
  • Experience building and evaluating complex agentic or multi-step LLM workflows.
  • Strong knowledge of modern ML frameworks and tools (e.g., PyTorch, TensorFlow, Hugging Face) and distributed/cloud-based infrastructure.
  • Demonstrated ability to optimize real-time ML systems for performance, scalability, and reliability.
  • Strong technical leadership skills, with the ability to influence cross-functional decisions and raise the engineering bar.

Other signals

  • LLM
  • Agentic AI
  • RAG
  • Evaluation Frameworks
  • Production Systems