Solutions/Databricks Architect

Agility Technologies Inc

1d • Remote

About The Position

The Databricks Architect will design, implement, and optimize our enterprise data lakehouse platform. The ideal candidate will be a technical leader with deep expertise in Databricks, cloud data architectures, and modern data engineering practices.

Requirements

• 5–10 years of experience in data engineering, data architecture, or distributed data processing, with at least 3+ years of hands-on experience with Databricks.
• Proficiency in Databricks, including Databricks SQL, Delta Lake, and the Lakehouse platform.
• Expertise in Apache Spark and distributed data processing frameworks.
• Advanced knowledge of Python, Scala, or Java for data engineering and analytics.
• Hands-on experience with cloud platforms designed for federal use, such as AWS GovCloud, Azure Government, or Google Cloud Public Sector.
• Strong knowledge of ETL processes, data workflows, and data transformation pipelines.
• Strong understanding of data lakes, data warehouses, data modeling, and schema design.
• Experience implementing data governance and security protocols within regulated environments.
• Knowledge of federal compliance standards, such as FISMA, FedRAMP, NIST 800-53, and CMMC (Cybersecurity Maturity Model Certification).
• Experience in securing data architecture for federal agencies, including encryption and access control best practices.
• Advanced expertise with SQL for querying and managing large datasets in relational databases.
• Familiarity with NoSQL databases and their use cases.
• Proven ability to lead cross-functional teams in the delivery of complex data solutions.
• Strong interpersonal and communication skills to engage with federal stakeholders and align technical solutions to mission objectives.
• Experience training and mentoring technical teams on Databricks best practices.

Nice To Haves

• Expertise in CI/CD pipelines and deployment of Databricks workflows and notebooks.
• Experience with Infrastructure-as-Code (IaC) tools such as Terraform, CloudFormation, or ARM templates.
• Strong understanding of machine learning pipelines and operationalizing models in Databricks MLflow.
• Familiarity with AI/ML frameworks like TensorFlow, PyTorch, or Scikit-learn.
• Experience with real-time data streaming using Kafka, Azure Event Hubs, or AWS Kinesis.
• Knowledge of other Big Data tools and frameworks such as Hadoop, Flink, or Elasticsearch.
• Experience with real-time analytics and event-driven architectures in mission-critical systems.
• Ability to work in highly regulated environments with federal stakeholders.
• Expertise in managing multiple priorities, meeting deadlines, and adapting to evolving client needs.
• Experience in preparing technical documentation, reports, and presentations tailored to federal audiences.
• Advanced Databricks certifications (e.g., Databricks Certified Professional Data Engineer).
• Certifications in federal cloud and cybersecurity standards (e.g., AWS Certified Security – Specialty, CCSP, CISSP, or CISM).
• Public Trust (or higher) clearance eligibility; prior federal client experience.

Responsibilities

• Design and implement scalable data lakehouse architectures using Databricks for federal data solutions.
• Develop and optimize end-to-end data pipelines, ensuring alignment with federal performance, security, and compliance requirements.
• Implement data governance and security protocols in accordance with federal standards (e.g., FedRAMP, FISMA, NIST, etc.).
• Collaborate closely with federal stakeholders to gather and analyze requirements and translate them into actionable data solutions.
• Lead efforts to integrate Databricks with federal cloud services, including AWS GovCloud, Azure Government, or other federal platforms.
• Provide subject matter expertise on Databricks capabilities, including Delta Lake, Databricks SQL, and Apache Spark.
• Optimize Databricks clusters for cost efficiency, high availability, and performance in a secure environment.
• Enable advanced analytics and machine learning workflows for federal use cases, leveraging tools such as MLflow and related frameworks.
• Develop and implement real-time data processing solutions for mission-critical federal applications.
• Stay updated on federal data policies and ensure solutions align with relevant standards and compliance requirements.

Benefits

• 401(k) matching
• Dental insurance
• Health insurance
• Paid time off
• Parental leave
• Vision insurance

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed