Senior / Staff ML Training Optimization Engineer

Waabi Waabi · Robotics · US & Canada, Dallas, TX +4 · Remote · Autonomy & Algorithms

Senior/Staff ML Training Optimization Engineer to build standardized distributed training frameworks, profile model runtime and memory, identify and evaluate emerging technologies for training and inference, and work with researchers on resource usage best-practices. Focus on optimizing training efficiency and stability.

What you'd actually do

  1. Build standardized distributed training frameworks for research and production, drive our training towards new levels of stability and efficiency.
  2. Comprehensively profile model runtime and memory to pinpoint performance bottlenecks.
  3. Identify and evaluate emerging technologies that can be adopted into Waabi’s training and inference frameworks. Examples include designing new CUDA kernels, quantization-aware training and inference, and compilation/deployment techniques.
  4. Work with researchers and ML engineers on best-practices for optimal resource usage.
  5. Create and improve tooling and dashboards to ensure broad adoption of your work.

Skills

Required

  • Python
  • C++
  • Rust
  • PyTorch
  • Jax
  • PyTorch Profiler
  • NVIDIA Nsight

Nice to have

  • CUDA kernels
  • Bazel
  • Kubernetes

What the JD emphasized

  • custom CUDA kernels

Other signals

  • distributed training frameworks
  • model runtime and memory profiling
  • quantization-aware training and inference
  • custom CUDA kernels