Senior Data Engineer

Adobe Adobe · Enterprise · San Jose, CA

Senior Data Engineer responsible for developing and maintaining analytics infrastructure, including ETL pipelines and automated data quality tools. Collaborates with product, analytics, and engineering teams to build, construct, and enhance scalable data pipelines and analytic solutions that generate insights into product usage. Focuses on the collection, integration, and transformation of large and varied data sets from multiple platforms to support business intelligence, product development, and strategic decision-making.

What you'd actually do

  1. collaborating with multi-functional teams to develop and refine adaptable data pipelines and analytical solutions
  2. build connectors to a variety of API sources, enabling the integration of diverse data sources
  3. apply cloud-based platforms to perform advanced analytics while upholding rigorous standards for data quality and security
  4. actively contribute to planning, solution design, and deployment of analytic tools that drive and support business goals
  5. ensure that the data infrastructure efficiently serves the evolving needs of the organization

Skills

Required

  • Python
  • SQL
  • Bash
  • Databricks
  • Spark SQL
  • Spark
  • DBFS
  • Airflow
  • Databricks Workflows
  • AWS
  • Azure
  • CI/CD
  • Jenkins

Nice to have

  • LLMs
  • vector databases
  • embeddings
  • graph databases
  • Elastic Stack
  • Kafka
  • Kinesis
  • Kubernetes

What the JD emphasized

  • 5+ Years Experience with Python
  • Experienced in developing data pipelines using Python, SQL, and Bash on Linux
  • Expertise working with Databricks, Spark SQL, Spark and DBFS systems
  • Proficient in orchestration tools like Airflow or Databricks Workflows
  • Experience deploying and managing services and applications on AWS and Azure Cloud platforms
  • Application design and architecture with a drive for delivering data with high throughput and low latency
  • Experience implementing and managing CI/CD pipelines using Jenkins or similar automation tools