Senior Data Engineer

Metropolis Metropolis · Vertical AI · Bengaluru, Karnataka, India · Data Engineering & Analytics

Senior Data Engineer role focused on building and optimizing end-to-end data pipelines, data architecture, and data models for large-scale projects. Requires expertise in SQL, Python, ETL/ELT, Snowflake, dbt, and Terraform, with experience in cloud environments and Big Data solutions.

What you'd actually do

  1. Collaborate with engineers, product managers, and data scientists to understand data needs, and develop end-to-end data pipelines, solutions, and foundational data sets
  2. Design and own the data architecture for large-scale projects, making informed decisions about design and operational trade-offs
  3. Build, launch, and optimize sophisticated data pipelines, data models, and visualizations that support diverse use cases across multiple products or domains
  4. Define and manage Service Level Agreements (SLAs) for all data sets within areas of ownership
  5. Implement data security models based on privacy requirements, ensure compliance with safeguards, address data quality issues, and evolve data governance processes

Skills

Required

  • SQL
  • ETL/ELT
  • data modeling
  • Snowflake
  • dbt
  • Python
  • Big Data solutions
  • ETL ecosystems
  • RDBMS (MySQL, MS SQL Server, Postgres)
  • Airflow
  • Automic
  • CI/CD pipelines
  • code quality
  • system reliability
  • performance
  • Spark
  • PySpark
  • Terraform

Nice to have

  • master's degree
  • Scala
  • Java
  • Copilot on GitHub
  • Datadog

What the JD emphasized

  • Strong proficiency in Terraform is a must-have.