Staff Inference ML Runtime Engineer

Cerebras Cerebras · Semiconductors · US and Canada Offices · Software

Staff Inference ML Runtime Engineer at Cerebras Systems, focusing on optimizing and scaling high-throughput, low-latency multimodal inference for generative AI models on custom hardware. The role involves designing and implementing APIs, ML features, and serving infrastructure, with a strong emphasis on performance, observability, and cross-functional collaboration.

What you'd actually do

  1. Drive and provide technical guidance to a team of software engineers working on complex machine learning integration projects.
  2. Design and implement ML features (e.g., structured outputs, biased sampling, predicted outputs) that improve performance of generative AI models at inference time.
  3. Design and implement high-throughput, low-latency multimodal inference models that support delivery of image, audio, and video inputs and outputs.
  4. Maintain our scalable serving backend for handling many concurrent requests per minute.
  5. Scale our inference service by implementing detailed observability throughout the entire stack.

Skills

Required

  • Python
  • C++
  • multi-threaded programming
  • performance optimization
  • system-level development
  • large-scale software engineering
  • deep learning
  • scalable systems
  • high-performance applications
  • ML frameworks (PyTorch)
  • problem-solving
  • communication
  • collaboration

Nice to have

  • LLM serving frameworks (vLLM, SGLang, TensorRT-LLM)
  • multimodal models
  • image
  • audio
  • video
  • observability

What the JD emphasized

  • 8+ years of experience in large-scale software engineering, with a focus on deep learning or related domains.
  • Advanced proficiency in C++, with an emphasis on multi-threaded programming, performance optimization, and system-level development.
  • Experience building and scaling large-scale inference systems for LLMs or multimodal models.
  • Familiarity with LLM serving frameworks, such as vLLM, SGLang, and TensorRT-LLM.

Other signals

  • Enabling fast generative inference solution through simple APIs powered by a distributed runtime that runs on large clusters of our own hardware.
  • Design and implement high-throughput, low-latency multimodal inference models that support delivery of image, audio, and video inputs and outputs.
  • Scale our inference service by implementing detailed observability throughout the entire stack.
  • Optimize software to accelerate generative LLM inference by achieving high throughput and low latency.