Data Engineer (Mid Level)

Irth Solutions

6h•Remote

About The Position

Irth is building a new AI-driven threat and risk management platform for pipeline asset integrity. The platform brings together three capabilities that have historically been separate at Irth: A governed, cross-product data platform built on Databricks and Azure; An AI-powered ingestion layer that normalizes, repairs, and enriches customer data without services-heavy onboarding; A reusable analytical layer that runs industry-standard, Irth-developed, and customer-built risk models against the data. As a Data Engineer, you will build the pipelines that make this platform real. Pipeline operators hold their integrity data across inline inspection reports, GIS systems, maintenance records, spreadsheets, scanned documents, and enterprise systems of record. Getting that data ingested, cleaned, aligned, and made model-ready is one of the biggest obstacles to adoption in this market—and it is the problem this role exists to solve. You will implement ingestion and transformation pipelines based on patterns established by the Data Architect, build the AI-assisted ingestion layer in partnership with data scientists, and help operationalize both. This role is well suited to a mid-level engineer who wants to deepen their expertise in Databricks, Spark, and modern lakehouse engineering while helping build a platform from the ground up.

Requirements

3–5 years of experience in data engineering, ETL development, or cloud data platform engineering.
Hands-on experience with Databricks, Spark, PySpark, or comparable distributed data-processing technologies.
Strong SQL skills and experience with structured data transformation.
Experience with at least one major cloud platform; Azure experience preferred.
Familiarity with data modeling, data-quality practices, schema evolution, and pipeline troubleshooting.
Experience with workflow orchestration and scheduling frameworks.
Understanding of core data-security practices, including access control, encryption, and credential management.
Experience with Git-based development and comfort working within a code-reviewed engineering team.

Nice To Haves

Experience integrating oil and gas or utility asset data, including pipelines, facilities, and GIS assets, into a data platform.
Understanding of asset integrity concepts, including inspection data, risk scoring, corrosion, and defect tracking.
Familiarity with regulatory and compliance reporting requirements for pipeline or asset integrity data.
Experience migrating customers from legacy or spreadsheet-based systems to modern data platforms.

Responsibilities

Build and maintain ingestion pipelines for structured and semi-structured sources, including GIS, inline inspection data, SCADA, maintenance systems, and enterprise systems of record.
Implement batch and streaming ingestion using Databricks Workflows, Spark, PySpark, SQL, and declarative pipeline tooling.
Apply medallion architecture patterns (Bronze, Silver, Gold) for transformation, standardization, and enrichment.
Implement change data capture (CDC), slowly changing dimensions (SCD), schema evolution, and data-validation rules.
Normalize third-party and public data feeds, including weather history, soil characteristics, satellite-derived data, and one-call ticket data, into the shared data model.
Build pipelines that automate normalization of units, schemas, and semantics across inconsistent customer data.
Implement automated data-quality repair workflows, including gap filling, error correction, and reconciliation, with clear provenance for every synthesized value.
Work with data scientists to productionize document-extraction pipelines that parse reports, spreadsheets, and field records into the target schema.
Build human-in-the-loop review and exception workflows so low-confidence extractions are surfaced rather than propagated silently.
Configure and manage Delta Lake tables, partitioning strategies, and optimization routines.
Implement metadata, lineage, and cataloging standards using Unity Catalog.
Build and maintain connectors to customer systems of record with configurable refresh cadences.
Support geospatial data processing, including spatial joins and alignment of results to pipeline centerline geometry.
Implement data-quality tests, profiling, and drift monitoring based on standards established by the Data Architect.
Apply access-control policies, security rules, and classification tags defined by the governance model.
Implement lineage capture sufficient to support regulatory traceability from ingestion through model output.
Build, schedule, and monitor data workflows, and own alerting and failure handling for the pipelines you develop.
Contribute to CI/CD for pipeline code, including version control, automated testing, and environment promotion.
Troubleshoot production incidents, recover failed pipeline runs, and optimize performance and infrastructure costs.
Work closely with the Data Architect to translate architectural designs into production implementations and identify gaps or ambiguities in the design.
Participate in architecture, design, and code reviews.
Document pipelines, transformation logic, data dictionaries, job schedules, operational procedures, and runbooks.