Senior Data Engineer

Caterpillar Caterpillar · Industrial · Bangalore, Karnataka

Seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines and infrastructure. Responsibilities include data ingestion, processing, integration, quality checks, and performance optimization using ETL tools and SQL. Requires 5-8 years of experience and a Bachelor's degree in a related field.

What you'd actually do

  1. Design, develop, and maintain scalable and reliable data pipelines to ingest, process, and store large volumes of data from various sources.
  2. Implement data workflows using ETL (Extract, Transform, Load) processes to ensure data quality and integrity.
  3. Collaborate with data architects, product owners, business analysts, and other stakeholders to understand data requirements and deliver data solutions that meet their needs.
  4. Integrate data from multiple sources, including databases, APIs, and external data providers, into a unified data platform.
  5. Manage and optimize data storage solutions, ensuring efficient data retrieval and processing.

Skills

Required

  • advanced SQL
  • ETL tools (Datastage, Informatica, Snaplogic)
  • data integration
  • ETL architecture and design
  • Databases
  • SQL
  • PL/SQL
  • CI/CD tools (Github Actions, AzDO pipelines)
  • Unix commands
  • AWS Glue
  • Qlik replication tool
  • analytical and problem-solving skills
  • communication and interpersonal skills
  • manage multiple tasks and prioritize workload

Nice to have

  • AWS Glue
  • Qlik replication tool
  • Unix commands