(usa) Principal, Software Engineer

Walmart · Retail · Sunnyvale, CA

Principal Engineer on the Walmart Advertising technology team, defining and leading the test engineering strategy for AI-powered advertising systems. This includes ML model validation, intelligent audience pipelines, real-time bidding infrastructure, and programmatic data platforms. The role involves architecting test frameworks, embedding quality into AI/ML lifecycles, and operating as a senior technical leader.

What you'd actually do

  1. Architect and own enterprise-scale test automation frameworks for APIs, data pipelines, UI, and services layers — built for maintainability, scale, and CI/CD integration
  2. Drive shift-left quality — embedding automated quality checks at every stage of the development lifecycle, from schema design through production monitoring
  3. Define and enforce quality gates in CI/CD pipelines at an organizational level; own release criteria and go/no-go decisions for major platform changes
  4. Lead data pipeline testing at scale — ETL/ELT validation, schema governance, event-level tracking accuracy, audience segment integrity, and aggregation correctness (BigQuery, Spark, Kafka)
  5. Own test strategy for AI/ML systems — including model validation, data drift detection, feature pipeline integrity, inference correctness, and algorithmic fairness testing for advertising models (lookalike, propensity, audience prediction)

Skills

Required

  • 8+ years in quality/test engineering
  • 3+ years at Staff or Principal SDET level in distributed, large-scale systems
  • Proven expertise testing AI/ML systems — model validation, feature store integrity, inference pipeline testing, or data quality for ML training datasets
  • Expert-level experience designing test automation frameworks for APIs, data, and services layers (Pytest, Selenium, Appium, WebDriver, or equivalent)
  • Deep expertise in data pipeline quality — ETL/ELT validation, schema governance, data completeness, transformation correctness at scale (BigQuery, Hive, Spark)
  • Strong experience testing event-driven and async systems — Kafka pipelines, REST APIs, webhook integrations
  • SQL as a first-class testing tool — writing complex queries against large analytical datasets to validate data correctness
  • Experience enforcing quality gates in CI/CD at organizational scale

Nice to have

  • Experience with LLM/GenAI quality — prompt regression testing, output evaluation frameworks, hallucination detection, or RAG pipeline validation
  • Expertise in programmatic advertising quality — RTB auction integrity, DSP/SSP integrations, ad serving correctness, measurement/attribution validation
  • Hands-on with GCP data platform: BigQuery, Dataflow, Pub/Sub, Vertex AI
  • Experience in privacy and compliance testing — GDPR, CCPA, consent signal propagation, data deletion verification, CMP validation
  • Proficiency with performance and load testing for high-throughput, low-latency systems
  • Familiarity with MLOps tooling — experiment tracking (MLflow, Vertex AI), model registries, feature stores, and deployment pipelines
  • Experience with data clean room and privacy-preserving measurement validation
  • Background in building and growing SDET teams — hiring, mentoring, technical interview design

What the JD emphasized

  • Proven expertise testing AI/ML systems
  • designing test automation frameworks
  • data pipeline quality
  • quality gates in CI/CD

Other signals

  • AI/ML model validation
  • quality gates in CI/CD
  • test automation frameworks
  • data pipeline testing
  • GenAI and LLM evaluation