Senior Software Engineer, Data Infrastructure

Decagon Decagon · Vertical AI · New York, NY · Engineering

Senior Software Engineer, Data Infrastructure at Decagon, an AI platform company. This role focuses on designing, building, and operating the data systems that power Decagon's AI products, including data pipelines, streaming systems, and analytical data layers. The engineer will ensure reliability, performance, and scalability of these data infrastructures, partnering with research and product teams. Experience with Kafka, Flink, Airflow, ClickHouse, BigQuery, and observability tools is required.

What you'd actually do

  1. Design and implement high-throughput data pipelines and streaming systems with strong SLOs, clear runbooks, and actionable telemetry.
  2. Build and operate real-time and batch ingestion infrastructure using tools like Kafka, Flink, and Airflow.
  3. Own our analytical data layer — schema design, query performance, and cost optimization across ClickHouse, BigQuery, or similar.
  4. Partner with research and product teams to architect data solutions, evaluate performance, and scale new features.
  5. Tune pipeline and query latencies: optimize data paths, apply smart caching/partitioning, and hit tight p95/p99 targets.

Skills

Required

  • 5+ years building and operating production data infrastructure at scale
  • Hands-on experience with Tier 1 data technologies: ClickHouse, Kafka (or MSK/Pub-Sub/RabbitMQ), and Flink or dbt
  • Proven track record meeting high availability and low latency targets across streaming and batch workloads
  • Excellent observability chops (OpenTelemetry, Prometheus/Grafana, Datadog) and strong incident response discipline
  • Clear written communication and the ability to turn ambiguous data requirements into simple, reliable designs

Nice to have

  • Experience with CDC tooling (Debezium) and orchestration frameworks (Airflow, Dagster, or Prefect)
  • Familiarity with Spark or Dask for large-scale data processing
  • Experience with cloud data warehouses (Snowflake, BigQuery, Redshift, Databricks)
  • Experience being an early data/platform/infrastructure engineer at another company
  • Strong Kubernetes experience (GKE/EKS/AKS) and multi-cloud exposure (GCP, AWS, Azure)
  • Experience with customer-managed deployments

What the JD emphasized

  • production data infrastructure at scale
  • high availability and low latency targets
  • low-latency systems