About Us Visa is a world leader in payments technology, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories, dedicated to uplifting everyone, everywhere by being the best way to pay and be paid.
At Visa, you'll have the opportunity to create impact at scale — tackling meaningful challenges, growing your skills and seeing your contributions impact lives around the world.
Join Visa and do work that matters – to you, to your community, and to the world. Progress starts with you.
Job Description
Role Overview
We are seeking a Systems Engineer with strong experience in Hadoop and Big Data platform operations. The ideal candidate will have hands-on expertise with Hadoop ecosystem components, automation using Shell/Bash and Python, and experience supporting large-scale data pipelines and distributed systems.
This role will be responsible for provisioning, maintaining, monitoring, scaling, and troubleshooting Hadoop clusters, as well as supporting automation, CI/CD processes, QA validation triggers, and operational improvements across Big Data platforms.
Responsibilities
Develop and maintain Shell/Bash scripts and Python-based automation scripts to streamline platform operations and improve overall efficiency.
Automate recurring platform and data operations, including:
- Pipeline execution
- Data validation triggers
- Job scheduling and orchestration
- Log collection and monitoring
- Alerting and notification workflows
- Environment health checks
- Data archival processes
- Deployment activities
Design and implement automation solutions for new data job onboarding, configuration updates, infrastructure maintenance, and routine operational tasks.
Support the automation, deployment, and operational management of ETL pipelines, data processing jobs, and Big Data workloads.
Integrate automated QA validation checks into pipeline execution workflows to ensure data accuracy, completeness, and reliability.
Capture, publish, and share QA validation results with engineering, data, and QA teams to improve visibility and accelerate issue resolution.
Collaborate with data engineering teams to troubleshoot job failures, data quality issues, pipeline execution errors, and performance bottlenecks.
Ensure data pipelines adhere to operational standards for reliability, observability, scalability, repeatability, and maintainability.
Support migration initiatives for Big Data workloads from on-premises Hadoop environments to cloud platforms, preferably AWS.
Assist with containerization and deployment of Big Data applications and services using Docker and Kubernetes.
Work with CI/CD tools to automate build, test, and deployment processes, with hands-on experience in at least one of the following:
- Jenkins
- GitLab CI
- GitHub Actions
This is a hybrid position. Expectation of days in the office will be confirmed by your Hiring Manager.
Visa requires at least 3 days in office, expectations of these days will be confirmed by your Hiring Manager.
Qualifications
Basic Qualifications
- 2+ years of relevant work experience and a Bachelors degree, OR 5+ years of relevant work experience
Preferred Qualifications
- 3 or more years of work experience with a Bachelor’s Degree or more than 2 years of work experience with an Advanced Degree (e.g. Masters, MBA, JD, MD)
- 3 - 6 years of hands-on experience supporting Hadoop, Big Data platforms, ETL pipelines, and distributed data systems
- Experience with automation, scripting, CI/CD, and cloud migration initiatives, preferably involving AWS
Visa is an EEO Employer
Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.