Lead Software Engineer - Data Engineering

Caterpillar · Industrial · Chennai, Tamil Nadu +1

Lead Software Engineer for Data Engineering at Caterpillar, focusing on building a next-generation Digital Manufacturing Data Platform. The role involves leading a team, architecting scalable data solutions using Snowflake, Kafka, Spark, and cloud services (Azure/AWS), and enabling advanced analytics and AI capabilities. Requires expertise in Big Data, Data Warehousing, real-time data movement, and CI/CD.

What you'd actually do

  1. Lead, mentor, and manage a team of data engineers and platform developers.
  2. Architect and oversee the development of scalable, secure, and resilient data platforms.
  3. Lead the development of robust data pipelines for ingestion, transformation, and delivery using Snowflake, dbt, and cloud-native tools.
  4. Build and maintain large-scale data lakes and data warehouses for structured and unstructured data.
  5. Collaborate with Data Science and AI teams to operationalize ML models and analytics workflows.

Skills

Required

  • Big Data processing
  • Data Warehousing
  • near real-time data pipelines
  • Data Platform architecture
  • Snowflake
  • SQL
  • NoSQL
  • distributed data systems
  • dbt
  • Apache Spark
  • Azure Data Factory
  • Azure
  • AWS
  • AI concepts

Nice to have

  • NVIDIA ecosystem
  • production-ready AI solutions
  • MLOps workflows
  • Power BI
  • Tableau
  • Looker

What the JD emphasized

  • Big Data processing
  • Data Warehousing
  • end-to-end near real-time data pipelines
  • scalable Data Platforms
  • Snowflake
  • SQL
  • NoSQL
  • distributed data systems
  • dbt
  • Apache Spark
  • Azure Data Factory
  • Azure
  • AWS
  • AI concepts
  • Gen AI
  • MCP
  • ATA