Lead Data Engineer

Visa Visa · Fintech · Bengaluru, India, IN

Lead Data Engineer responsible for architecting, scaling, and optimizing enterprise-wide data platforms for an Agentic AI and Data Platform. This role involves technical leadership, team mentoring, and driving innovation in data engineering, with a focus on supporting AI/ML initiatives and integrating with agent frameworks.

What you'd actually do

  1. Define and drive the data engineering vision, strategy, and roadmap in alignment with business goals.
  2. Architect and oversee the design of complex, distributed data systems, ensuring scalability, reliability, and security.
  3. Lead, mentor, and grow a team of data engineers, fostering a culture of innovation, accountability, and continuous improvement.
  4. Own end-to-end delivery of large-scale data engineering projects, including prioritization, planning, and stakeholder management.
  5. Lead the evaluation and adoption of emerging technologies (e.g., cloud platforms, streaming, ML/AI integration).

Skills

Required

  • Proven experience architecting and leading enterprise-scale data engineering initiatives (Hadoop, Hive, Spark, Kafka, etc.)
  • Strong programming proficiency in Python, Java, or Scala
  • Production-level experience architecting Hadoop, Spark, and Databricks data pipelines
  • Hands-on experience with AWS or Azure (Glue, Synapse, Redshift, Delta Lake, S3)
  • Deep expertise in cloud platforms (AWS, Azure, GCP) and associated services (EC2, S3, SageMaker, etc.)
  • Advanced proficiency in programming languages (Scala, Python, Java) and scripting (Unix/Shell, Python)
  • Strong background in data modeling, ETL/ELT, and optimization of big data pipelines
  • Experience with CI/CD, automated testing, and DevOps tools (Jenkins, Artifactory, Git, Selenium, Chef)
  • Demonstrated ability to lead teams, influence stakeholders, and deliver results in a fast-paced, global environment.

Nice to have

  • Familiarity with Kafka, Airflow, Kubernetes, and containerized data services
  • Understanding of RAG pipelines, vector databases, and AI data flows
  • Experience designing data pipelines and APIs compatible with Model Context Protocol (MCP)-based agent frameworks, enabling seamless integration between AI agents, data services, and enterprise API
  • Familiarity with RDBMS (MS SQL, DB2, Oracle) and visualization tools (Tableau, Power BI)

What the JD emphasized

  • architecting, scaling, and optimizing enterprise-wide data platforms
  • architect and oversee the design of complex, distributed data systems
  • production-level experience architecting Hadoop, Spark, and Databricks data pipelines
  • Deep expertise in cloud platforms (AWS, Azure, GCP)
  • Understanding of RAG pipelines, vector databases, and AI data flows
  • Experience designing data pipelines and APIs compatible with Model Context Protocol (MCP)-based agent frameworks

Other signals

  • building a next-generation Agentic AI and Data Platform
  • architecting, scaling, and optimizing enterprise-wide data platforms
  • drive innovation, set technical direction
  • deliver robust, secure, and high-quality data solutions that power business-critical analytics and applications
  • lead the evaluation and adoption of emerging technologies (e.g., cloud platforms, streaming, ML/AI integration)
  • Understanding of RAG pipelines, vector databases, and AI data flows
  • Experience designing data pipelines and APIs compatible with Model Context Protocol (MCP)-based agent frameworks