Senior Big Data Engineer

Omniscius Consulting•Washington, DC

12d

About The Position

Our client is seeking an experienced Big Data Engineer to design, develop, and support large-scale data pipelines and transformation processes. The ideal candidate will have strong SQL, Python, ETL/ELT, and data engineering experience, along with the ability to communicate technical concepts clearly to clients, stakeholders, and cross-functional teams. Java experience is helpful for supporting existing systems and integrations but is not the primary focus of this position.

Requirements

Bachelor’s degree from an accredited college or university or equivalent professional experience.
Five or more years of professional experience in data engineering, database development, ETL/ELT development, or a related data integration role.
Strong hands-on SQL experience, including complex queries, performance tuning, stored procedures, and data modeling.
Experience designing, building, optimizing, and supporting large-scale data pipelines.
Strong Python programming experience supporting data transformation, integration, automation, or pipeline operations.
Experience parsing, transforming, and validating XML-based data.
Experience working in Linux and shell environments, including scripting, scheduled jobs, cron, and log troubleshooting.
Experience implementing data validation, error handling, pipeline monitoring, and production troubleshooting.
Strong written and verbal communication skills.
Ability to explain technical concepts clearly to clients and stakeholders.
Ability to collaborate across security, infrastructure, engineering, and delivery teams.
Self-motivated and proactive, with a positive and collaborative approach.
U.S. citizenship is required.
Must be willing and able to undergo a background investigation for a Public Trust suitability determination.

Nice To Haves

Experience using Python as the primary language for data pipeline automation, transformation, monitoring, or production troubleshooting.
Experience supporting Java-based applications, services, or integrations.
Working knowledge of AWS data services, including S3, Glue, Lambda, RDS, or Redshift.
Exposure to XBRL or other structured financial, regulatory, or compliance-driven data formats.
Experience supporting federal, government, or compliance-driven data environments.
Experience with Oracle, PostgreSQL, Redshift, or other relational database platforms.
Familiarity with reporting, analytics, or dashboarding tools such as Power BI or MicroStrategy

Responsibilities

Design, build, maintain, and optimize ETL/ELT pipelines that move and transform large-scale data across source and target systems.
Write and optimize complex SQL queries, stored procedures, data models, and validation processes.
Develop data transformation, integration, and automation solutions primarily using Python.
Use Java as needed to support existing applications, services, and system integrations.
Parse, transform, and validate XML-based data formats.
Work within Linux and shell environments to support scripting, scheduled jobs, cron processing, log reviews, and troubleshooting.
Implement data validation, error handling, monitoring, and traceability controls.
Troubleshoot pipeline failures, data discrepancies, performance issues, and integration defects.
Collaborate with client, security, infrastructure, data engineering, and delivery teams.
Communicate technical issues, risks, and recommendations clearly to technical and nontechnical stakeholders.
Participate in Agile planning, status reporting, documentation, issue tracking, and cross-team coordination.