Senior Software Engineer, Foundation Model API

Databricks Databricks · Data AI · Mountain View, CA · Engineering

Senior Software Engineer role focused on building and scaling the Foundation Model APIs (FMAPI) serving layer for large language models at Databricks. The role involves developing infrastructure for real-time and batch inference, improving latency and efficiency, and shaping the product roadmap for enterprise AI use cases.

What you'd actually do

  1. Build LLM infrastructure powering large-scale inference workloads for customers through partner models (OpenAI, Anthropic, Gemini) and self-hosted models (Qwen, GPT-OSS, Llama)
  2. Shape the direction of the FMAPI product — from roadmap to execution — by leveraging deep customer empathy and direct engagement with enterprise users and model providers
  3. Improve reliability, latency, and efficiency of distributed AI workloads
  4. Collaborate with platform, infra, and ML teams to deliver seamless end-to-end experiences
  5. Shape how developers and data scientists build and interact with AI on Databricks

Skills

Required

  • backend or infrastructure engineering
  • distributed systems
  • scalable APIs
  • cloud-native infrastructure
  • real-time serving
  • ML infrastructure
  • GPU orchestration
  • service-oriented architecture
  • deployment pipelines
  • system observability
  • Scala
  • Go
  • Python

Nice to have

  • SageMaker
  • Vertex AI
  • Azure ML
  • platforms that support AI workflows

What the JD emphasized

  • 8+ years of experience in backend or infrastructure engineering
  • Experience with distributed systems, scalable APIs, or cloud-native infrastructure
  • Experience with real-time serving, ML infrastructure, or GPU orchestration

Other signals

  • LLM infrastructure
  • large-scale inference
  • enterprise scale
  • real-time serving
  • distributed AI workloads