Staff Software Engineer - Backend

Databricks Databricks · Data AI · Seattle, WA +1 · Engineering - Pipeline

Databricks is seeking a Staff Software Engineer - Backend to work on their data and AI infrastructure platform. The role involves designing, implementing, testing, and operating micro-services for the Databricks platform, focusing on backend systems. Responsibilities include building data pipelines, integrating with third-party applications, and interacting with cloud APIs. The role can join various teams such as Data Science and Machine Learning Infrastructure, Compute Fabric, Data Plane Storage, Enterprise Platform, Observability, Service Platform, or Core Infra. The ideal candidate will have extensive experience in backend development, distributed systems, SaaS platforms, and cloud technologies.

What you'd actually do

  1. work closely with your team and product management to prioritize, design, implement, test, and operate micro-services for the Databricks platform and product.
  2. writing software in Scala/Java, building data pipelines (Apache Spark™, Apache Kafka), integrating with third-party applications, and interacting with cloud APIs (AWS, Azure, CloudFormation, Terraform).
  3. Build services and infrastructure at the intersection of machine learning and distributed systems.
  4. Build the resource management infrastructure powering all the big data and machine learning workloads on the Databricks platform in a robust, flexible, secure, and cloud-agnostic way.
  5. Deliver reliable and high performance services and client libraries for storing and accessing humongous amount of data on cloud storage backends, e.g., AWS S3, Azure Blob Store.

Skills

Required

  • BS/MS/PhD in Computer Science, or a related field
  • 10+ years of production level experience in one of: Java, Scala, C++, or similar language.
  • Experience in architecting, developing, deploying, and operating large scale distributed systems.
  • Experience working on a SaaS platform or with Service-Oriented Architectures.
  • Good knowledge of SQL.
  • Experience with software security and systems that handle sensitive data.
  • Experience with cloud technologies, e.g. AWS, Azure, GCP, Docker, Kubernetes.

Nice to have

  • Scala/Java
  • Apache Spark™
  • Apache Kafka
  • AWS, Azure, CloudFormation, Terraform

What the JD emphasized

  • critical to making customers successful on our platform
  • large scale software platforms
  • millions of virtual machines
  • terabytes of logs
  • exabytes of data per day
  • cloud hardware, network, and operating system faults
  • gracefully shield our customers
  • 10+ years of production level experience
  • architecting, developing, deploying, and operating large scale distributed systems
  • operating with the resilience, security and scale