AWS Databricks Engineer

Credence•McLean, VA

About The Position

Credence is hiring an AWS Databricks Engineer to join its AI and Automation practice. The role is a technical contributor responsible for designing, building, and optimizing scalable cloud-based data solutions using AWS and Databricks. The engineer will develop data pipelines, implement data lakehouse architectures, and collaborate with engineering, data science, and stakeholder teams to deliver secure, high-performance data capabilities supporting federal missions. The position involves Databricks engineering, AWS data integration, data transformation and optimization, governance and quality, cloud automation and DevOps, collaborative engineering, and technical documentation. The engineer will build and maintain production data solutions and workflows, optimize performance and cost, and follow engineering, security, and federal development standards.

Requirements

U.S. citizenship and the ability to obtain and maintain a Secret clearance.
Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical field.
Hands-on experience in data engineering, cloud engineering, or related technical roles.
Hands-on experience building and maintaining production data solutions using Databricks, Apache Spark, and PySpark.
Strong experience with AWS data services such as S3, Glue, RDS, Redshift, Lambda, IAM, and CloudWatch.
Strong understanding of ETL/ELT processes, data lake and lakehouse architectures, data modeling, and large-scale data processing.
Hands-on experience with Delta Lake, SQL, Python, and Spark-based data processing.
Experience developing, scheduling, monitoring, and troubleshooting production data pipelines and workflows.
Experience with Git and CI/CD platforms such as GitLab, GitHub Actions, Jenkins, or AWS CodePipeline.
Understanding of cloud security, identity and access management, encryption, and secure data-handling practices.
Strong communication skills and a client-oriented mindset, with the ability to collaborate across technical and non-technical teams.

Nice To Haves

Experience implementing Databricks Lakehouse Architecture and medallion architecture using Bronze, Silver, and Gold data layers.
Experience with Databricks Unity Catalog, Delta Live Tables, Databricks Workflows, or similar data orchestration and governance capabilities.
Strong Python proficiency and experience with data engineering libraries and frameworks such as PySpark, Pandas, and Pydantic.
Experience with Infrastructure as Code tools such as Terraform, OpenTofu, AWS CDK, or CloudFormation.
AWS and/or Databricks certifications, such as AWS Certified Data Engineer - Associate, AWS Certified Solutions Architect, or Databricks Certified Data Engineer credentials.
Knowledge of federal cybersecurity, RMF, FedRAMP, or other government regulatory and compliance frameworks.
Experience delivering cloud-native data solutions within federal government or other highly regulated environments.

Responsibilities

Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake.
Design and implement modern lakehouse solutions using Databricks and AWS services.
Apply Bronze, Silver, and Gold medallion architecture layers to improve data quality, governance, lineage, performance, and accessibility.
Build and maintain ETL/ELT pipelines that integrate structured, semi-structured, and unstructured data across AWS services and related cloud data platforms.
Develop data transformation workflows using SQL, Python, and PySpark.
Optimize Spark workloads, queries, clusters, and storage configurations for performance, reliability, and cost efficiency.
Implement data quality, cataloging, lineage, governance, and access-control practices across Databricks and AWS environments, including Unity Catalog where applicable.
Automate data platform deployments and workflows using Infrastructure as Code, CI/CD pipelines, Git-based development practices, and AWS-native deployment tools.
Collaborate with data engineers, software engineers, data scientists, cloud engineers, and mission stakeholders to translate requirements into production-ready data solutions.
Develop clean, maintainable, well-documented code in accordance with engineering, security, and federal development standards.
Support code reviews, testing, troubleshooting, and technical design documentation.