Data Engineer II
About The Position
Our dev/ops team builds and supports the data engineering solutions that power analytics, reporting, and operational decision-making across the business and Field. We develop and maintain scalable ELT pipelines using tools like Python, Spark, Databricks, and orchestration frameworks to deliver reliable, high-quality data for downstream consumers, including Power BI and SSRS reporting solutions. We focus on production reliability, data quality, observability, and continuous improvement, and we partner closely with business and technology teams to turn complex data needs into usable data products.
Requirements
- 2–5 years professional experience in data engineering or software engineering with production systems.
- Strong Python programming skills and experience with distributed processing (PySpark or Scala + Spark).
- Solid SQL skills and experience with at least one cloud data platform (Snowflake, Databricks, or equivalent).
- Working knowledge of Power BI, including building datasets and reports, data modeling, and troubleshooting dataset refreshes and performance issues.
- Experience with orchestration tools (Airflow, Control-M, or equivalent) and CI/CD pipelines.
- Familiarity with event streaming systems (Kafka, SQS, or similar) and batch/stream integration patterns.
- Experience implementing data quality checks, testing data pipelines, and handling schema changes.
- Proficient with Git and modern DevOps practices; comfortable reading and troubleshooting logs.
- Has or develops understanding of 1-3 subject areas/domains of data.
- Good communicator; able to explain technical solutions to peers and stakeholders; self-directed.
Nice To Haves
- Familiarity with Databricks ELT architecture (Declarative Automation Bundles, Delta Lake, Jobs/Workflows, and Delta Live Tables).
- Experience with Terraform, Kubernetes, Docker, or other IaC/container tooling.
- Familiarity with observability/monitoring tools (Grafana, Dynatrace, etc.) and alerting best practices.
- Experience with data quality frameworks (DQX, Deequ) and data governance/catalog tools.
- Domain knowledge in one or more business areas outside of the Field domain (e.g., Client, Product, etc).
Responsibilities
- Design, implement, and maintain ELT pipelines and transformations using Python/PySpark/SQL.
- Own production reliability: monitor SLAs, respond to incidents, notebooks, and reduce MTTR.
- Implement and maintain CI/CD pipelines to automate addition and modification of custom stages in ELT workflows.
- Orchestrate workflows with Airflow, Control M (or equivalent); handle retries, backfills, and schema changes.
- Build and maintain streaming integrations (Kafka/Kinesis or equivalent) for near‑real‑time use cases.
- Implement and maintain data quality, observability and auditing (e.g., DQX, data expectations, pydat, custom checks).
- Optimize pipelines for cost and performance on cloud platforms (Databricks, Snowflake, or similar).
- Participate in code reviews, design discussions, and documentation; mentor junior engineers.
- Identify root causes of data issues and implement robust fixes and preventative measures.
- Collaborate with stakeholders to translate business requirements into reliable data solutions.
Benefits
- Geographic specific pay structures, compensation and benefits could be applicable
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
