Data Platform Lead (Hybrid)

American Medical Association•Chicago, IL

4d•Hybrid

About The Position

The American Medical Association (AMA) is seeking a Data Platform Lead for their Information Technology team. This is a hybrid position based in Chicago, IL, requiring three days a week in the office. The role serves as the platform owner for AMA’s new enterprise Databricks data platform, leading its architecture, engineering, and daily operations. The primary responsibility is to ensure the secure, reliable, performant, and cost-effective operation of the platform. This role translates enterprise data platform strategy and governance into technical implementation, collaborating with IT, security, and business-unit engineering teams to enable responsible data and AI usage. The Data Platform Lead is the primary technical authority for AMA’s lakehouse environment and is accountable for platform capabilities, standards, and operations.

Requirements

Bachelor’s degree in computer science, engineering, information systems, or related field preferred or equivalent work experience and HS diploma/equivalent education required
7+ years of experience in data engineering, data platform engineering, or data architecture required.
Proven experience designing, building, and operating a large-scale data platform, preferably built on modern lakehouse architectures (Databricks preferred)
Proven expertise with Delta Lake, Unity Catalog, Apache Spark, and SQL-based transformation frameworks (dbt or equivalent)
Experience owning platform reliability, performance tuning, observability, and cloud cost optimization
Proven experience implementing data governance controls, including access, data quality, metadata, and lineage, within a data platform
Working knowledge of AI/ML workflows, MLOps practices, and model lifecycle management
Experience operating within a regulated data environment (healthcare / HIPAA, financial services, or similar) preferred
Demonstrated success leading technical teams and collaborating across engineering, security, and business disciplines
Strong ability to influence, negotiate, and drive alignment across stakeholders without direct authority
Ability to balance innovation and delivery speed with operational reliability, cost efficiency, and regulatory requirements

Responsibilities

Serve as the platform owner for AMA’s enterprise Databricks lakehouse, accountable for its technical direction, delivery, and operational health.
Translate the enterprise data platform strategy and multi-year roadmap into an actionable technical delivery plan and engineering backlog.
Define and maintain the platform’s technical architecture, including workspace topology, Delta Lake design, and Unity Catalog structure.
Maintain a prioritized platform engineering backlog using agile delivery methods, balancing new capability development, technical debt remediation, and operational work.
Evaluate emerging Databricks and data ecosystem capabilities and recommend adoption where they advance reliability, cost efficiency, or business value.
Contribute to platform budget planning and forecasting, and provide technical input into platform-related procurement and vendor management.
Own platform prioritization and backlog, including intake and sequencing of requests from business units and engineering teams.
Own the technical onboarding of new data sources and business-unit workloads onto the platform.
Collaborate with Sr Data Engineer to lead the design, build, and operation of data ingestion, transformation, and delivery pipelines across batch and streaming workloads.
Ensure adherence to engineering standards for data modeling, code quality, version control, testing, and CI/CD across the platform.
Design and maintain self-service analytics capabilities and reusable data products that reduce duplicated effort across business units.
Provide and operate shared MLOps platform capabilities (e.g., model registry, deployment frameworks), enabling data science and engineering teams to operationalize models.
Develop and maintain platform documentation, runbooks, and standard operating procedures.
Serve as the final decision authority on platform architecture and engineering standards; provide design approvals and guardrails for data engineering teams.
Contribute to the reliability, performance, scalability, and security of the enterprise data platform, in partnership with the Director II and both IT and federated business-unit engineering teams.
Define and operate observability tooling and monitoring practices for platform health, cost optimization, model performance, data quality, and governance compliance; drive continuous improvement initiatives.
Drive cloud cost optimization through cluster configuration, capacity planning, storage management, and cost allocation across business units.
Implement platform security controls, including encryption, access management, and audit logging, in alignment with IT Security policy and HIPAA and other regulatory requirements.
Define, configure, and enforce platform-level governance controls (e.g., Unity Catalog, access policies, lineage tooling, data quality monitoring, validation, and issue-escalation processes).
Partner with the Data Governance Lead to translate Enterprise Data Office policies into platform configuration and automated enforcement.
Implement technical controls supporting AI governance, including model lifecycle management, monitoring, and drift detection.