Senior Data Engineer

Walmart • Bentonville, AR

12d • $92,934 - $180,000 • Onsite

About The Position

The Senior Data Engineer will be responsible for understanding and applying data strategy principles to solve business problems. This role involves identifying suitable data sources, performing initial data quality checks, extracting and transforming data, and creating data pipelines. The engineer will also translate business problems into data solutions, develop business cases, and create conceptual, physical, and logical data models. Responsibilities include writing code, testing solutions, deploying software, and establishing data governance practices. The role requires collaboration with business stakeholders and peers to implement data governance policies and ensure compliance with company and regulatory standards.

Requirements

• Bachelor’s degree or the equivalent in Computer Science or a related field plus 3 years of experience in software engineering or related experience OR Master’s degree or the equivalent in Computer Science or a related field plus 1 year of experience in software engineering or related experience
• Designing and implementing large-scale data pipelines and ETL processes using Python, Scala, PySpark, Spark and SQL for batch and streaming workloads
• Designing and building production streaming and event-driven pipelines (Kafka, Google Pub/Sub) including real-time ingestion and processing
• Architecting and managing enterprise cloud data solutions on GCP (BigQuery, Dataproc, Cloud Storage, Pub/Sub, Cloud SQL, Secret Manager, Compute Engine)
• Experience with open‑source data technologies (Hadoop, Spark, Kafka, Hive)
• Designing data models that translate business requirements into scalable, query-efficient schemas for analytics using BigQuery and Hive
• Developing, optimizing and maintaining complex analytical queries across BigQuery, Hive and relational databases
• Managing scheduling, orchestration and monitoring of ETL and ELT workflows using Apache Airflow while implementing comprehensive data warehousing concepts, dimensional modeling and ETL best practices
• Performing performance tuning and cost optimization for Spark jobs, Dataproc clusters and cloud storage systems by analyzing execution metrics and resource utilization
• Implementing and managing comprehensive CI/CD pipelines for data applications using git-based version control systems and advanced cloud infrastructure provisioning tools such as Terraform
• Establishing and maintaining data quality frameworks, security protocols and governance standards while applying standard procedures for managing data, monitoring compliance requirements, and ensuring efficient operations across all distributed data processes.

Responsibilities

• Understand, articulate, and apply principles of the defined data strategy to routine business problems.
• Support the understanding of the priority order of requirements and service level agreements.
• Identify the most suitable source for data that is fit for purpose.
• Perform initial data quality checks on extracted data.
• Extract data from identified databases.
• Create data pipelines and transform data to a structure that is relevant to the problem by selecting appropriate techniques.
• Develop knowledge of current data science and analytics trends.
• Translate/ co-own business problems within one's discipline to data related or mathematical solutions.
• Identify appropriate methods/tools to be leveraged to provide a solution for the problem.
• Share use cases and give examples to demonstrate how the method would solve the business problem.
• Provide recommendations to business stakeholders to solve complex business issues.
• Develop business cases for projects with a projected return on investment or cost savings.
• Translate business requirements into projects, activities, and tasks and align to overall business strategy and develop domain specific artifact.
• Serve as an interpreter and conduit to connect business needs with tangible solutions and results.
• Identify and recommend relevant business insights pertaining to their area of work.
• Analyze complex data elements, systems, data flows, dependencies, and relationships to contribute to conceptual, physical, and logical data models.
• Develop the Logical Data Model and Physical Data Models including data warehouse and data mart designs.
• Define relational tables, primary and foreign keys, and stored procedures to create a data model structure.
• Evaluate existing data models and physical databases for variances and discrepancies.
• Develop efficient data flows.
• Analyze data-related system integration challenges and propose appropriate solutions.
• Create training documentation and train end-users on data modeling.
• Oversee the tasks of less experienced programmers and stipulate system troubleshooting supports.
• Write code to develop the required solution and application features by determining the appropriate programming language and leveraging business, technical, and data requirements.
• Create test cases to review and validate the proposed solution design.
• Create proofs of concept.
• Test the code using the appropriate testing approach.
• Deploy software to production servers.
• Contribute code documentation, maintain playbooks, and provide timely progress updates.
• Establish, modify, and document data governance projects and recommendations.
• Implement data governance practices in partnership with business stakeholders and peers.
• Interpret company and regulatory policies on data.
• Educate others on data governance processes, practices, policies, and guidelines.
• Provide recommendations on needed updates or inputs into data governance policies, practices, or guidelines.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior