Data Engineer (Databricks)

Cogent People Inc•Columbia, MD

1d•Hybrid

About The Position

Cogent People Inc. is seeking a skilled and motivated Data Engineer with Databricks experience to build and maintain data pipelines for a mission-critical, data-centric system supporting a large-scale Federal Government IT program in the healthcare domain. The role is based in Columbia, Maryland, with remote or hybrid work available for candidates local to Maryland; hybrid employees work in the office on Mondays and Wednesdays. Candidates must have lived in the United States for at least three of the past five years and already be authorized to work in the United States. Sponsorship is not offered. A Public Trust clearance is required, or candidates must be able to obtain one. The Data Engineer will translate business requirements into pipelines that fit the system architecture and security boundaries; build and maintain Databricks pipelines using Python, PySpark, and Spark SQL; develop Delta Lake ingestion patterns for batch loads and recurring reports; and migrate and validate Oracle SQL and PL/SQL logic in Databricks. The role also includes automating reports, scheduling and monitoring Databricks jobs, administering workspaces, applying governance and access controls, and following DevSecOps practices. The engineer will collaborate with cross-functional Agile teams and document their work. Cogent People describes its environment as collaborative, growth-driven, supportive, and inclusive. The company offers competitive salaries, career development opportunities, and the opportunity to contribute to mission-critical projects. It identifies itself as a Certified Great Place to Work.

Requirements

Bachelor’s degree in computer science, software engineering, data engineering, or a related field, or equivalent experience.
At least 5 years of hands-on data engineering experience, including at least 2 years building production pipelines on Databricks.
Strong Python and PySpark skills.
Strong SQL skills.
Experience with Delta Lake, Databricks Workflows or Jobs, and Databricks notebooks or Repos.
Basic Databricks administration experience, including configuring clusters and SQL warehouses, cluster policies, users, groups and service principals, workspace and job permissions, Unity Catalog grants, secret scopes, and usage and cost monitoring.
Experience with AWS data services such as S3, IAM, Glue, or Lambda.
Good understanding of software development processes.
Experience with Agile methodologies such as Scrum; SAFe is a plus.
Working knowledge of DevSecOps practices, including CI/CD and release management.
Ability to apply data security best practices and tune Spark jobs at scale while supporting system availability.
Strong communication and teamwork skills for working with cross-functional teams.
Must have lived in the United States for at least 3 of the past 5 years.
Must already be authorized to work in the United States; sponsorship is not available.
Must have or be able to obtain Public Trust clearance.

Nice To Haves

Database background, preferably Oracle, including data modeling, query tuning, indexing, and partitioning.
Oracle PL/SQL development experience, including stored procedures, packages, and functions.
Hands-on experience migrating Oracle databases or PL/SQL logic to Databricks or Spark.
Databricks Certified Data Engineer certification at the Associate or Professional level.
Experience in regulated or Government environments, especially healthcare.
Understanding of AI/ML principles or exposure to ML workflows on Databricks, such as MLflow.
Experience with analytical and BI tools.
Experience with SAFe.

Responsibilities

Translate business requirements into working pipelines that fit the overall architecture and security boundaries.
Build, test, and maintain Databricks data pipelines using Python, PySpark, and Spark SQL.
Design Delta Lake tables and ingestion patterns for batch loads and recurring reports.
Convert Oracle SQL and PL/SQL logic to Databricks and validate results against the source system.
Automate recurring reports on Databricks.
Schedule and monitor jobs with Databricks Workflows and tune them for performance, cost, and availability.
Administer Databricks workspaces, including compute and cluster policies, users, groups, service principals, permissions, and secrets.
Apply data governance and access controls using Unity Catalog within federal security boundaries.
Follow DevSecOps practices, including using Git, CI/CD for pipelines, and promoting changes through environments.
Collaborate with developers, project managers, business analysts, and testers in an Agile team.
Document work and solutions.