Staff Data Platform Engineer

Wayve Wayve · Robotics · London, United Kingdom · Product & Delivery

Staff Data Platform Engineer to own and lead Wayve's Data Transfer Hub project, a mission-critical infrastructure for transferring petabytes of sensor data daily from partner vehicles globally to enable AI model training at scale. The role involves designing and building a distributed ingestion system using technologies like Flyte, Kafka, and Azure, with a focus on technical leadership and hands-on delivery.

What you'd actually do

  1. Set technical direction for the Data Transfer Hub, making trade-offs across reliability, throughput, cost, partner constraints, observability and operational support.
  2. Lead the technical design and delivery of the Data Transfer Hub project, ingesting large volumes of video, LiDAR and sensor data from global partners at up to PB/day scale
  3. Design and build parallelised, distributed data transfer pipelines using Flyte for workflow orchestration, Kafka/event-driven patterns for lifecycle tracing, and Azure for storage and transfer infrastructure
  4. Build and operate a globally distributed hub-and-spoke data transfer model to retrieve and share sensor data at scale with partners
  5. Write infrastructure-as-code in Terraform; build pipeline logic primarily in Python

Skills

Required

  • Proven experience building and operating large-scale data transfer pipelines at multi-TB or PB scale
  • Experience orchestrating parallelised transfer jobs: Flyte preferred; other workflow orchestration tools (e.g. Airflow, Prefect) will be considered
  • Experience building reliable, observable data systems, including retry strategies, backfills, data integrity checks, lifecycle/state tracking and operational alerting.
  • Strong cloud platform skills (Azure preferred); experience working with multiple cloud providers (AWS/GCP) is a strong advantage
  • Proficiency in Python and Terraform
  • Cloud-to-cloud and networking experience (e.g. blob/object transfer and storage at scale, cross-cloud data movement)
  • A product mindset: you think about the customer and build solutions that fix real problems, not just implement requirements
  • Comfort working in a fast-changing environment with evolving requirements: this is a startup within a scale-up
  • High degree of autonomy and ownership: you take responsibility for outcomes, not just tasks

Nice to have

  • Experience with Kafka for event-driven transfer observability
  • Multi-cloud experience across Azure, AWS and/or GCP
  • Experience working directly with external data partners
  • Background in autonomous vehicles, robotics or similar data-intensive domains

What the JD emphasized

  • petabytes of sensor data per day
  • AI model training at scale
  • large-scale data transfer pipelines at multi-TB or PB scale
  • orchestrating parallelised transfer jobs
  • building reliable, observable data systems
  • cloud platform skills
  • Python
  • Terraform
  • Cloud-to-cloud and networking
  • product mindset
  • fast-changing environment with evolving requirements
  • High degree of autonomy and ownership

Other signals

  • petabytes of sensor data per day
  • AI model training at scale
  • globally distributed hub-and-spoke ingestion system