Sr. Data Engineer

Visa•Foster City, CA

1d•$123,000 - $190,900•Hybrid

About The Position

As a Senior Data Engineer, you will play a critical role in designing and delivering scalable, high-quality data solutions that enable intelligent decision-making across Global Data Engineering capabilities. You will leverage your expertise to design and implement large-scale data pipelines and integrated data solutions using the latest technologies, ensuring efficient data storage, processing, and presentation. You will also build and utilize advanced data modeling frameworks and adopt modern architectural patterns such as Medallion Architecture to deliver well-structured, efficient, and future-ready data solutions.

Requirements

2 or more years of work experience with a bachelor’s degree or an Advanced Degree (e.g. Masters, MBA, JD, MD, or PhD)
Hands-on data engineering experience building and operating production data pipelines.
Extensive experience in Hadoop ecosystem and associated technologies, (For e.g. Apache Spark, Python, Pandas etc.), Open table formats (e.g. Iceberg)
Proficiency in Python, SQL, and PySpark.
Experience with Unix/Linux systems with scripting experience in Bash.
Experience with data pipeline and workflow management tools like Airflow, etc.
Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
Practical experience with AWS data engineering services, especially S3, EMR, and Glue Catalog.
Strong analytic skills related to working with structured & unstructured datasets.
Proficiency in managing and communicating data warehouse plans to internal clients.
Experience with building processes supporting data transformation, data structures, metadata, dependency, and workload management.
Experience designing for data quality, resiliency, and observability, including validation checks, monitoring, alerting, and runbooks
Practical experience using generative AI tools such as Claude code or similar systems to accelerate coding, documentation, or incident analysis while applying appropriate review controls.
Strong communication skills and the ability to convert ambiguous business requirements into clear technical implementation plans.

Nice To Haves

3 or more years of work experience with a bachelor’s degree or more than 2 years of work experience with an Advanced Degree (e.g. Masters, MBA, JD, MD)
Experience with platforms such as databricks or snowflake

Responsibilities

Experience in Requirement Gathering, Estimating, Managing large scale Data Engineering Projects.
Requirement Analysis: Understand and translate business needs into data models supporting long-term solutions.
Data Modeling: Work with the Business team to implement data strategies, build data flows and develop conceptual data models.
Data Pipeline Design: Create robust and scalable data pipelines and data products in a variety of domains.
Data Integration: Develop and maintain data lakes by acquiring data from primary and secondary sources and build scripts that will make our data evaluation process more flexible or scalable across data sets.
Testing: Define and manage the data load procedures to reject or discard datasets or data points that do not meet the defined business rules and quality thresholds.
Operations: Able to independently manage the data load operations of the solutions or products in scope and be able to debug, manage stakeholder communications and provide clarification on the data produced.
Understanding of and ability to Implement Data Engineering principles and best practices.
Stakeholder Management: Collaborate with stakeholders across the organization to understand their data needs and deliver solutions.
Continuous Integration and Continuous Deployment (CI/CD): Implement and maintain CI/CD pipelines for data solutions, ensuring rapid, reliable, and streamlined updates to the data environment.
Leverage AI on day to day tasks to bring in efficiency in implementation and quicker time to market.
Use Claude Code or Github copilot to generate and refactor boilerplate PySpark jobs, Hive or Snowflake SQL, Airflow DAG scaffolding, and documentation, while retaining full ownership of review, hardening, validation, and production readiness.
Define and maintain Claude agents and skills for recurring workflows such as requirement intake, code review, PySpark job generation, backfill planning, incident triage, and artifact documentation.
Integrate Claude Code with Jira, GitHub, SharePoint, Airflow, and internal catalog or data-quality systems through MCP or comparable connector patterns.
Build and evolve data quality checks, observability signals, and operational runbooks to improve reliability of the datasets
Drive engineering best practices through GitHub workflows, code reviews, automated tests, schema controls, and safe deployment patterns for both on-prem and cloud components.