Lead Data Engineer

Target Target · Retail · Bangalore, India

Lead Data Engineer responsible for designing and developing scalable, high-performance data solutions, owning data architecture, and optimizing data pipelines using big data technologies, distributed systems, and cloud platforms. Experience with API development for low-latency data serving and Customer Data Platforms is a plus.

What you'd actually do

  1. Architect and build scalable, high-performance data pipelines and distributed data processing solutions using Hadoop, Spark, Scala/Java, and cloud platforms (AWS/GCP/Azure).
  2. Design and implement real-time and batch data processing solutions, ensuring data is efficiently processed and made available for analytical and operational use.
  3. Develop APIs and data services to expose low-latency, high-throughput data for downstream applications, enabling real-time decision-making.
  4. Optimize and enhance data models, workflows, and processing frameworks to improve performance, scalability, and cost-efficiency.
  5. Drive data governance, security, and compliance best practices.

Skills

Required

  • Hadoop
  • Spark
  • Scala
  • Java
  • AWS
  • GCP
  • Azure
  • REST
  • GraphQL
  • gRPC
  • Kafka
  • Flink
  • Kinesis
  • data modeling
  • ETL pipeline design
  • performance optimization
  • data governance
  • security
  • compliance

Nice to have

  • Python
  • Customer Data Platforms (CDP)
  • customer-centric data processing

What the JD emphasized

  • technical anchor
  • scalable, high-performance data solutions
  • own and drive data architecture
  • big data technologies, distributed systems, and cloud platforms
  • architecting and optimizing data pipelines
  • Hadoop, Spark, Scala/Java
  • API development for serving low-latency data
  • Customer Data Platforms (CDP)
  • low-latency, high-throughput
  • data governance, security, and compliance
  • technical leadership and mentorship
  • 10+ years of experience
  • 7+ years in data engineering, software development, or distributed systems
  • Expertise in big data technologies
  • Hadoop, Spark, and distributed processing frameworks
  • Strong programming skills in Scala and/or Java
  • Experience with cloud platforms (AWS, GCP, or Azure)
  • data ecosystem
  • Proficiency in API development
  • REST, GraphQL, or gRPC
  • real-time and streaming data architectures
  • Kafka, Flink, Kinesis
  • data modeling, ETL pipeline design, and performance optimization
  • data governance, security, and compliance
  • Customer Data Platforms (CDP)
  • customer-centric data processing
  • complex, unstructured environments
  • cross-functional teams
  • big data, API, and cloud technologies
  • data architecture and real-time data services
  • high-impact business problems
  • scalable, low-latency data solutions