Software Engineer, Evals

Glean Glean · Enterprise · Bangalore, India · Engineering

Software Engineer, Evals role focused on building platforms for measuring, explaining, and improving AI quality at scale for Glean's Work AI platform. This includes designing and building large-scale evaluation pipelines, agent observability infrastructure, and backend systems to support AI feature shipping and quality loops.

What you'd actually do

  1. Design and build large-scale evaluation pipelines that measure assistant and agent quality across thousands of real user and synthetic workflows.
  2. Evaluate frontier model releases and the latest OSS model drops, building the infrastructure and quality signals that help Glean understand regressions, tradeoffs, and launch readiness.
  3. Build agent observability infrastructure, including trace enrichment, durable telemetry pipelines, dashboards, and debugging workflows that make AI behavior inspectable.
  4. Own backend systems from architecture and design docs through production rollout, reliability, monitoring, and iteration.
  5. Partner with product, ML, and infrastructure engineers to make evals a first-class part of how Glean ships AI features.

Skills

Required

  • 4+ years of software engineering experience building backend systems, infrastructure, distributed systems, or data platforms
  • Strong coding skills in Go, Python, Java, C++, or similar languages
  • Comfortable working with distributed data pipelines, production services, observability systems, or cloud-native infrastructure
  • Analytically rigorous
  • Deep care about quality

Nice to have

  • Experience with LLM applications
  • Experience with evals
  • Experience with tracing
  • Experience with data warehouses
  • Experience with workflow orchestration
  • Experience with ML infrastructure

What the JD emphasized

  • core systems for running large-scale evaluations
  • quality
  • AI quality at scale
  • large-scale evaluation pipelines
  • agent observability infrastructure
  • backend systems
  • evals a first-class part
  • quality loop
  • balance speed, reliability, enterprise security, and cost
  • 4+ years of software engineering experience building backend systems, infrastructure, distributed systems, or data platforms
  • strong coding skills in Go, Python, Java, C++
  • comfortable working with distributed data pipelines, production services, observability systems, or cloud-native infrastructure
  • analytically rigorous
  • care deeply about quality

Other signals

  • evaluation pipelines
  • quality evalsets
  • LLM-powered judges
  • agent observability
  • measurement and quality layer