Lead Data Engineer

Honeywell Honeywell · Industrial · Atlanta, GA +1

Lead Data Engineer role focused on architecting and owning data foundations for industrial AI and Generative AI pipelines, including IoT telemetry, medallion lakehouse, vector stores, embedding pipelines, and RAG workflows. The role involves technical leadership, mentoring, and collaboration with data scientists and AI engineers.

What you'd actually do

  1. Architect end-to-end data pipelines processing terabytes of IoT telemetry on Azure Databricks (PySpark DLT, Lakeflow) using medallion Lakehouse architecture.
  2. Design and optimize real-time ingestion pipelines from Azure Event Hub and Apache Kafka for high-volume industrial IoT telemetry.
  3. Build fault-tolerant, idempotent streaming architectures handling schema evolution, backpressure, and latency SLAs.
  4. Define technical direction for AI-ready data products including vector stores, embedding pipelines, and RAG-ready structured/unstructured data.
  5. Build production GenAI pipelines- RAG workflows, document ingestion, PII anonymization and vector database infrastructure.

Skills

Required

  • Apache Spark / PySpark
  • Azure Databricks
  • Apache Kafka
  • Azure Event Hub
  • Cloud data architecture (Azure preferred)
  • Data modeling
  • Schema design
  • RAG systems
  • Embedding pipelines
  • Document ingestion
  • MLOps familiarity
  • CI/CD (GitHub Actions)

Nice to have

  • LangChain
  • LangGraph
  • Apache Spark Streaming
  • Structured Streaming
  • AI model deployment
  • Time-series databases
  • IoT data modeling
  • Docker
  • Kubernetes
  • Data quality implementation for AI training data
  • Agile and Scrum Methodologies

What the JD emphasized

  • 8+ years of data engineering experience with at least 2 years in a lead or senior role, demonstrating progression in technical complexity and team leadership.
  • Hands-on experience building and operating medallion lakehouse architectures (Bronze / Silver / Gold).
  • Deep expertise in Apache Spark / PySpark with production experience on Azure Databricks at scale.
  • Strong proficiency with streaming platforms - Apache Kafka and/or Azure Event Hub for real-time IoT data.
  • Proven experience building data pipelines for GenAI or ML applications: RAG systems, embedding pipelines, and document ingestion.

Other signals

  • architect and own the data foundations that enable physical AI at scale
  • production-grade Generative AI pipelines
  • building systems that have real-world industrial impact
  • Define technical direction for AI-ready data products including vector stores, embedding pipelines, and RAG-ready structured/unstructured data
  • Build production GenAI pipelines- RAG workflows, document ingestion, PII anonymization and vector database infrastructure