Data Engineer (Databricks)

Openings & Internships • Columbia, MD

18h • Hybrid

About The Position

Cogent People Inc. is seeking a skilled and motivated Data Engineer with Databricks experience to build and maintain data pipelines for a mission-critical, data-centric system supporting a large-scale Federal Government IT program in the healthcare domain. The position is based in Columbia, Maryland. Remote or hybrid work is available for candidates local to Maryland, with in-office work on Mondays and Wednesdays.

Candidates must have lived in the United States for at least three of the past five years and already be authorized to work in the U.S. (Citizen, Permanent Resident, or EAD). Sponsorship is not available. A Public Trust clearance, or the ability to obtain one, is required.

The role involves developing production pipelines and reports on Databricks, migrating Oracle logic, administering Databricks workspaces, and applying data governance and security controls within federal security boundaries. The engineer will collaborate with cross-functional Agile teams and follow DevSecOps practices.

Requirements

• Bachelor’s degree in computer science, software engineering, data engineering, or a related field, or equivalent experience.
• At least 5 years of hands-on data engineering experience, including at least 2 years building production pipelines on Databricks.
• Strong Python and PySpark skills.
• Strong SQL skills.
• Experience with Delta Lake, Databricks Workflows/Jobs, and Databricks notebooks or Repos.
• Basic Databricks administration skills, including configuring clusters and SQL warehouses, cluster policies, users, groups and service principals, workspace and job permissions, Unity Catalog grants, secret scopes, and usage and cost monitoring.
• Experience with AWS data services such as S3, IAM, Glue, or Lambda.
• Good understanding of software development processes.
• Experience with Agile methodologies such as Scrum; SAFe experience is a plus.
• Working knowledge of DevSecOps practices, including CI/CD and release management.
• Ability to apply data security best practices, tune Spark job performance at scale, and support system availability.
• Strong communication and teamwork skills for working with cross-functional teams.
• Must have lived in the U.S. for at least 3 of the past 5 years and be authorized to work in the U.S.; sponsorship is not available.
• Must hold a Public Trust clearance or be able to obtain one.

Nice To Haves

• Database background, preferably Oracle, including data modeling, query tuning, indexing, and partitioning.
• Oracle PL/SQL development experience with stored procedures, packages, and functions.
• Hands-on experience migrating Oracle databases or PL/SQL logic to Databricks or Spark.
• Databricks Certified Data Engineer certification at the Associate or Professional level.
• Experience in regulated or Government environments, especially healthcare.
• Understanding of AI/ML principles or exposure to ML workflows on Databricks, such as MLflow.
• Experience with analytical and BI tools.
• SAFe experience.

Responsibilities

• Translate business requirements into working pipelines that fit the overall architecture and security boundaries.
• Build, test, and maintain Databricks data pipelines using Python, PySpark, and Spark SQL.
• Design Delta Lake tables and ingestion patterns for batch loads and recurring reports.
• Convert existing Oracle SQL and PL/SQL logic to Databricks and verify results against the source system.
• Automate recurring reports on Databricks.
• Schedule and monitor jobs with Databricks Workflows, tuning them for performance, cost, and availability.
• Perform day-to-day Databricks workspace administration, including managing compute and cluster policies, users, groups, service principals, permissions, and secrets.
• Apply data governance and access controls using Unity Catalog within federal security boundaries.
• Follow DevSecOps practices, including using Git, implementing CI/CD for pipelines, and promoting changes through environments.
• Collaborate with developers, project managers, business analysts, and testers in an Agile team, and document work.

Benefits

• Health insurance
• Dental insurance
• Vision insurance
• 401(k) plan with matching contributions
• Company-paid life insurance
• Company-paid short-term disability coverage
• Company-paid long-term disability coverage
• Three weeks of paid time off annually
• Ten paid holidays
• Career development opportunities

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level