Staff Software Engineer, Metrics and Logging

Databricks Databricks · Data AI · Mountain View, CA · Engineering

Staff Software Engineer to drive the next generation of Databricks' logging infrastructure, focusing on scalable and efficient logging solutions for observability across all Databricks services. The role involves designing and scaling a platform processing petabytes of logs daily, optimizing delivery pipelines for low-latency ingestion and querying, enhancing log accessibility, and improving reliability and cost-efficiency.

What you'd actually do

  1. Build the future of logging at Databricks by designing and scaling our next-generation logging platform that processes petabytes of logs daily.
  2. Develop and optimize log delivery pipelines to support low-latency, high-throughput log ingestion and querying, ensuring seamless observability across all Databricks services.
  3. Enhance log accessibility and usability, developing tools that enable engineers to efficiently search, analyze, and derive insights from logs.
  4. Collaborate with teams across Databricks to define best practices for structured logging, standardizing formats and improving the developer experience.
  5. Improve reliability and cost-efficiency by optimizing log retention, indexing, and query performance to reduce operational overhead.

Skills

Required

  • Scala
  • Rust
  • Go
  • Python
  • Java
  • C++
  • large-scale distributed systems
  • log collection
  • health monitoring
  • observability tools

Nice to have

  • BS (or higher) in Computer Science, or a related field
  • 7+ years of production-level experience