Data Platform Architect (Databricks)

Coastal•Indianapolis, IN

3d•Hybrid

About The Position

Coastal is seeking an experienced, hands-on Data Platform Architect and Integration Lead for a large-scale CRM modernization program. The role designs, builds, and delivers integrations that consume governed customer, account, and product master data products published in Databricks and reliably deliver them to an enterprise commercial CRM platform and related systems. The data products are curated Delta tables registered in Unity Catalog. The role owns their consumption and delivery while re-platforming existing integrations to modern, cloud-native patterns. This is a hands-on leadership position: the architect leads the integration workstream while personally contributing to coding, code review, testing, and deployment. The role owns the integration lifecycle from design specifications and build through testing, deployment across dev, sandbox, UAT, and production, runbooks, and knowledge transfer. The architect selects appropriate integration mechanisms—including Apache Iceberg interoperability, Spark/PySpark-based movement, or APIs—based on volume, latency, and directionality. They build transformation and mapping logic, handle master-data changes, preserve parity and prevent data loss during migration, and implement observability, data-quality checks, error handling, data-gap detection, and reprocessing. The role also leads client discussions, collaborates with a second system integrator on testing and defect resolution, defines reusable patterns and interface contracts, and partners with data-product owners and data-engineering teams on schemas, versioning, refresh frequency, access, and change management. Data integrity, security, privacy, and applicable regulated-industry compliance must be maintained. The position requires in-person presence in Indianapolis. If the employee is not working from Indianapolis, travel is required for important program meetings. Coastal describes its work as remotely flexible, with office locations in Palm Coast, Florida; Atlanta, Georgia; Tysons, Virginia; and Lexington, Kentucky, and travel as required to client locations.

Requirements

Direct, hands-on experience with Databricks, including Delta Lake, Unity Catalog, consuming and publishing data products, and lakehouse/medallion architecture.
Hands-on experience applying AI in a data, integration, or software-development context.
Hands-on experience building data and transformation services or pipelines on AWS.
Hands-on experience designing, building, and implementing REST APIs at scale.
Strong, hands-on PySpark and Spark SQL experience for data movement and transformation.
Hands-on experience integrating customer, account, and product master data into downstream systems, including schema mapping, reconciliation, and master-data change handling.
Demonstrated ability to lead a build team while personally performing build, review, and deployment work.
Ability to be present in Indianapolis or travel for important program meetings as required.

Responsibilities

Personally perform coding, code review, deployment, and other assigned transformation work while leading the integration team.
Own the end-to-end integration lifecycle, including design specifications, build, unit testing, deployment across dev, sandbox, UAT, and production, runbooks, and knowledge transfer.
Produce and deliver design specifications, deployment steps, and rollback approaches.
Consume Databricks data products, including Delta Lake tables through Unity Catalog, in accordance with their schema, refresh cadence, quality guarantees, and access model.
Move master data from Databricks into the CRM using an appropriate integration pattern, such as Apache Iceberg interoperability, Spark/PySpark, or APIs.
Build PySpark and Spark SQL transformation and mapping logic to reconcile data-product schemas with the target CRM data model.
Handle master-data changes, including deltas, data-change requests, and reference data.
Re-platform existing master-data integrations using Databricks data products and modern integration patterns while ensuring parity, reconciliation, and no data loss.
Implement observability, data-quality checks, error handling, data-gap detection, and reprocessing for Databricks-to-CRM flows.
Participate in and lead client design reviews, working sessions, governance forums, and status discussions.
Communicate technical decisions, risks, and dependencies clearly to client stakeholders and delivery teams.
Collaborate with the second system integrator responsible for independently testing and validating the build.
Review the second SI's test findings, triage them, and resolve defects, maintaining clean build-to-test handoffs and shared traceability.
Define reusable integration patterns and interface contracts for master-data flows.
Partner with data-product owners and data-engineering teams on schemas, versioning, refresh frequency, access, and change management.
Ensure data integrity, security, privacy, and applicable regulated-industry compliance across connected systems.
Be present in person in Indianapolis or travel for important program meetings as required.

Benefits

Flexible working hours with an emphasis on work-life balance.
Remote-flexible work and company offices in Palm Coast, Florida; Atlanta, Georgia; Tysons, Virginia; and Lexington, Kentucky, with travel as required to client locations.
Unlimited paid time off (RTO).
401(k) with company match.
Medical, vision, and dental coverage.
Competitive quarterly bonus opportunities.
Continuing education and certification reimbursements, specifically within the Salesforce and Snowflake ecosystems.
Occasional in-house competitions with spot bonuses.
Flexible team culture emphasizing transparency, support, flexibility, growth, teamwork, and fun.
Frequent virtual and in-person team and culture activities, including Lunch and Learns, Happy Hours, and team-building events.
Monthly All-Hands calls, an open-door leadership policy, and access to mentorship and guidance.
Opportunities for accelerated growth, networking, and career guidance and support.
Trust, transparency, and respect across all levels of the company.