Data Engineer Role

OpenDataJobs•Washington, DC

About The Position

Data Engineers build and operate the systems that move data from its sources to the people, applications, analyses, and models that depend on it. They ingest data, transform it, organize it for use, and keep it accurate, secure, traceable, and available at scale. Their work turns fragmented files, documents, databases, application programming interfaces (APIs), and event streams into reliable data products. Artificial intelligence (AI) is one important consumer of that work, alongside reporting, visualization, analytics, software, and operational systems. Some openings may involve preparing dependable data for machine learning, document retrieval, or AI evaluation. The center of the Role remains dependable data engineering from source to use.

Requirements

A working foundation in programming and query languages used for data engineering. Python and Structured Query Language (SQL) are common, but the specific stack varies by opening.
Experience or strong grounding in data ingestion, transformation, storage, schema and data-model design, and the performance characteristics of distributed data systems.
An understanding of batch, streaming, and event-driven processing, together with orchestration, testing, deployment, monitoring, recovery, and documentation.
Practical experience with data quality, metadata, lineage, provenance, versioning, access controls, and secure data handling.
The judgment to work in environments where accuracy, privacy, security, traceability, reproducibility, resilience, performance, and cost matter.

Nice To Haves

Each opening will identify the experience, platform, tooling, data, performance, security, location, work-authorization, citizenship, suitability, clearance, and domain knowledge the work requires. Those requirements will vary, and no candidate is expected to cover every specialization.
An opening may emphasize extract, transform, and load (ETL), extract, load, and transform (ELT), batch or stream processing, data modeling, a warehouse or lakehouse, document extraction, a feature store, machine-learning data, search or vector indexing, retrieval-augmented generation data preparation, telemetry and feedback pipelines, data-governance implementation, or platform operations.
Specific openings may name Amazon Web Services, Microsoft Azure, Google Cloud, Spark, Airflow, Kafka, Flink, dbt, relational or nonrelational databases, data warehouses, lakehouses, search platforms, vector databases, Linux, container platforms, or infrastructure-as-code tools. OPEN Data Jobs will state those requirements with the opening rather than treat every technology in this Role description as universal.

Responsibilities

Ingestion and transformation pipelines for batch, streaming, and event-driven data from APIs, databases, files, documents, object stores, messaging systems, and operational platforms.
Cloud and on-premises data platforms, including databases, data lakes, warehouses, lakehouses, and serving layers for reporting, visualization, software, analytics, and other operational uses.
Quality, validation, metadata, lineage, provenance, and access-control capabilities that make data trustworthy, explain how it changed, and keep its use within approved boundaries.
The operational layer around data products: orchestration, testing, monitoring, backfills, replay, recovery, retention and deletion implementation, performance and cost tuning, infrastructure as code, and technical documentation.
Versioned feature, training, testing, or evaluation datasets; document and retrieval-index pipelines; or governed telemetry and feedback data that support machine learning and generative AI systems.