Software Engineer - Data Engineering

Caterpillar Caterpillar · Industrial · Chennai, Tamil Nadu

Software Engineer focused on data engineering, responsible for designing, developing, and deploying data pipelines and state-of-the-art digital platforms. The role involves working with large data lakes and data warehouses, near real-time data movement solutions, and various AWS services. While AI concepts are mentioned as a 'good understanding' area, the core focus is on data infrastructure and engineering.

What you'd actually do

  1. Design and implement end-to-end near real-time data movement solutions.
  2. Works independently on complex systems or infrastructure components that may be used by one or more applications or systems.
  3. Drives application development focused around delivering business valuable features
  4. Maintains high standards of software quality within the team by establishing good practices and habits.
  5. Mentor junior developers fostering a culture of technical excellence, collaboration, and continuous learning.

Skills

Required

  • Data Pipelines in Python
  • Python as an Object Oriented programming language
  • Big Data
  • Large Data Lakes and Data Warehouse (Snowflake Preferred)
  • end-to-end near real-time data pipelines for OLTP & OLAP
  • SQL and NO-SQL databases
  • Message brokers and AWS services (Kafka/Kinesis, AWS SQS, AWS SNS, Lambda, API Gateway, DynamoDB, Aurora, AWS RDS PostgreSQL)
  • CI/CD tools (Azure Devops, Jenkins, GoCD, AWS CloudFormation)
  • public clouds (AWS or Azure)
  • computer science fundamentals like data structures and algorithms and object-oriented design

Nice to have

  • AI concepts and latest developments (Gen AI, MCP, ATA, etc.)
  • Test driven development and behaviour driven development
  • analytical skills
  • leadership on medium to large-scale projects

What the JD emphasized

  • 5+ years or more of software development experience
  • 5+ years or more of experience in designing and developing Data Pipelines in Python
  • Proven experience in many of the following
  • At least 2+ plus years of deploying and maintaining software using public clouds such as AWS or Azure.
  • Bachelor’s degree in Computer science or Electrical engineering or related field is required.