Senior Principal - Databricks Platform Engineering Lead (python)

JPMorgan Chase JPMorgan Chase · Banking · Jersey City, NJ +1 · Corporate Sector

This role focuses on building and leading the design of scalable Databricks/Spark data platforms and analytics workloads across cloud environments. The engineer will develop secure, production-grade Python code, own the delivery of data pipelines, and solve complex cloud data platform problems. The role emphasizes best practices in data engineering, data lake architecture, and distributed computing, with a strong focus on operational stability and mentoring technical teams.

What you'd actually do

  1. Leads the design, build, and troubleshooting of a scalable Databricks/Spark data platform and analytics workloads across cloud environments.
  2. Develops and maintains secure, production-grade Python code; perform code reviews, debugging, and engineering quality improvements across the team.
  3. Owns delivery of scalable data pipelines and analytics solutions on Databricks, applying strong distributed computing and performance troubleshooting practices.
  4. Executes creative software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems
  5. Develops secure high-quality production code, and reviews and debugs code written by others

Skills

Required

  • Databricks
  • Apache Spark
  • Python
  • distributed data processing
  • system design
  • application development
  • testing
  • operational stability
  • terraform
  • GitHub / Bitbucket
  • Jenkins
  • pypi / maven artifactory

Nice to have

  • Agile development processes
  • SCRUM/KANBAN
  • JIRA
  • Data pipelines using Spark
  • managing product release lifecycle

What the JD emphasized

  • Expert-level proficiency in Databricks, Apache Spark, and distributed data processing
  • Hands-on experience with Python application program development with use of automated unit testing
  • Hands-on practical experience delivering system design, application development, testing, and operational stability
  • Hands-on practical experience with terraform development and understanding of terraform enterprise