Senior Data Engineer

Integres, LLC • Washington, DC

5d • Remote

About The Position

Integres, LLC is seeking a Senior Data Engineer for a remote position supporting the Office of the Inspector General (OIG). This role requires a highly skilled individual to provide expertise in data engineering methods and best practices, focusing on modern pipeline design and code-first development. The engineer will be responsible for designing, implementing, and maintaining a secure, stable, and flexible data architecture, including ELT/ETL pipelines in Azure Synapse and Azure Machine Learning. Key responsibilities include reviewing and improving existing architecture, establishing quality controls, optimizing data processing, developing self-service capabilities for analysts, and collaborating with data scientists. The role also involves developing standard operating protocols, documenting the data architecture, expanding the environment with new datasets, and staying current with emerging AI tools for data engineering automation.

Requirements

• Five (5) years of hands-on experience in maintaining SQL databases and conducting advanced operations in SQL and T-SQL.
• Five (5) years of hands-on experience in designing, implementing, and maintaining ELT/ETL processes in cloud-based data analytics environments.
• Three (3) years of hands-on experience working in Azure Synapse and Azure Machine Learning, with the modern data stack.
• Three (3) years of hands-on experience manipulating data in Python. Pandas required.
• US Citizen

Nice To Haves

• Certifications preferred (DP-203 or equivalent).
• PySpark/Polars preferred.
• Experience developing reusable, modular code preferred.
• Implementing pipelines and infrastructure using code-first approaches (Python SDK, CLI, REST APIs, or IaC tooling).
• Implementing source control and CI/CD workflows.
• Demonstrated familiarity with AI coding assistants and LLM integration patterns.

Responsibilities

• Assist TSD with data products by providing highly skilled and authoritative expertise on data engineering methods and best practices, including code-first development approaches and modern pipeline design patterns.
• Design, implement, and maintain an efficient, secure, stable, and flexible data architecture that supports products and end-users, with all assets managed via source control.
• Design, implement, and maintain ELT/ETL pipelines for efficient processing of source data in Azure Synapse and Azure Machine Learning (using SDK V1 and SDK V2).
• Review, maintain, and improve existing architecture and pipelines, including periodic audits to address bottlenecks, deprecated dependencies, and architecture drift.
• Establish quality controls for maintaining all pipelines, and introduce error handling, logging mechanisms, and validation checks.
• Incorporate source control for all pipelines and data analytics codebases to enable iterative code development while ensuring data architecture stability.
• Optimize the ingestion, processing, and storage of a wide variety of datasets and data types, including modern columnar formats such as Parquet.
• Develop self-service capabilities for SBA OIG analysts to query and export data for investigations and audits.
• Coordinate with data scientists to ensure the architecture efficiently supports machine learning algorithms and data pipelines in Azure Machine Learning.
• Develop robust standard operating protocols (SOPs) dictating the authoring, development, validation, publishing, execution, and monitoring of all data pipelines and assets in Azure environment.
• Provide detailed documentation of the data architecture, including data dictionaries, ER diagrams, and pipeline process maps.
• Maintain and expand the environment with additional datasets and services upon request, following a defined intake and testing process prior to production deployment.
• Stay current with emerging AI tools relevant to data engineering and contribute to exploratory efforts evaluating automation and LLM-assisted capabilities.

Benefits

• Healthcare and Insurance: medical, dental, vision, short- and long-term disability protection, basic life and AD&D insurance
• 401(k) Savings Plan
• Accrued Paid Time Off (PTO)
• Employee Recognition and Rewards
• Employee Referral Bonuses

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed