Principal, Machine Learning Engineer

Lila Sciences Lila Sciences · AI Frontier · San Francisco, CA · AI

Principal ML Engineer at Lila Sciences, focusing on building and scaling ML infrastructure for generative models in medicine. The role involves owning end-to-end systems from training pipelines and distributed compute to model deployment and integration into a closed-loop discovery engine. Key responsibilities include designing and optimizing large-scale training pipelines, owning production ML systems, architecting ML infrastructure, driving the "Lab-in-the-Loop" lifecycle, and defining ML engineering standards. The role requires deep expertise in distributed training, production ML systems, and strong software engineering fundamentals, with a focus on generative models for biological data.

What you'd actually do

  1. Design, build, and optimize large-scale training pipelines for generative models on biological and chemical data, including distributed training across GPU clusters
  2. Own production ML systems end to end: model deployment, serving infrastructure, monitoring, and reliability for models used in Lila's scientific workflows
  3. Architect ML infrastructure that supports rapid iteration across sequence design, structure prediction, and multimodal scientific reasoning workloads
  4. Drive the engineering side of Lila's "Lab-in-the-Loop" lifecycle: build pipeline models, integrate experimental feedback loops, and ensure model outputs are actionable for downstream scientific workflows
  5. Define and advance ML engineering standards, tooling, and best practices across the AI organization

Skills

Required

  • Master's degree or higher in Computer Science, Machine Learning, or a related quantitative field (or Bachelor's with equivalent professional experience)
  • 10+ years of hands-on experience building and operating production ML systems at scale
  • Deep expertise in distributed training infrastructure, including experience with large-scale GPU clusters (AWS, GCP, or on-prem)
  • Strong software engineering fundamentals: system design, production-grade code, CI/CD, observability, and reliability practices
  • Proficiency in ML frameworks (PyTorch, JAX, or TensorFlow) with experience optimizing training and inference performance
  • Demonstrated ability to drive technical direction for ML infrastructure independently, from architecture through implementation
  • Track record of cross-functional collaboration with research scientists, translating between ML methodology and engineering execution

Nice to have

  • Experience building training or inference infrastructure for generative models applied to biological sequences, molecular structures, or scientific data
  • Experience with agentic frameworks, active learning loops, or closed-loop experimental workflows
  • Contributions to open-source ML tools, frameworks, or infrastructure projects
  • Familiarity with at least one life science domain (molecular biology, genomics, protein engineering, or nucleic acid design)
  • Experience with model evaluation frameworks for scientific applications where ground truth is sparse or delayed

What the JD emphasized

  • 10+ years of hands-on experience building and operating production ML systems at scale
  • Deep expertise in distributed training infrastructure, including experience with large-scale GPU clusters (AWS, GCP, or on-prem)
  • Strong software engineering fundamentals: system design, production-grade code, CI/CD, observability, and reliability practices
  • Track record of cross-functional collaboration with research scientists, translating between ML methodology and engineering execution

Other signals

  • ML engineers build and operate the systems that turn generative models and reasoning frameworks into production capabilities
  • own critical systems end to end, from training pipelines and distributed compute to model deployment and integration into Lila's closed-loop discovery engine
  • shape the technical direction for how ML models are trained, evaluated, and deployed at scale
  • collaborate closely with AI scientists and experimental researchers to close the computational-experimental loop
  • drive Lila's ML infrastructure toward the next generation of capabilities