Senior Data Platform Architect (Snowflake, Databricks, Open Platforms & Data Virtualization), AVP - State Street Investment Management

State Street•Boston, MA

36d•$90,000 - $142,500•Onsite

About The Position

The Senior Data Platform Architect is responsible for the end‑to‑end architecture, security design, and governance of SSIM's enterprise data platform spanning Snowflake, Databricks, open data platforms (Iceberg/Delta/Parquet), and data virtualization. This role requires previous experience supporting workloads in analytics, regulatory reporting, AI/ML, and batch applications. The individual will ensure the modern platform meets financial‑services regulatory expectations, enterprise security standards (GCS/IAM), and data protection requirements (NPPI/PII). They will enable scalable, interoperable delivery across Investment Management use cases. This role is hands‑on, decision‑critical, and risk‑bearing, with accountability for architectural choices that directly impact regulatory compliance, audit defensibility, and business continuity. Platform Designs must be Secure, Scalable, include Disaster Recovery, as well as Data Sharing capabilities. The role also involves designing and building monitoring and chargeback models for billing back to application teams.

Requirements

Snowflake – WH Sizing, Object Hierarchy, Snowsight, RBAC, Horizon Catalog, governance, performance, data sharing, External Tables, Stages, Loading, SPs, Scripts, Snowpark
Databricks – Spark, Unity Catalog, ML pipelines, Delta/Iceberg, Lakehouse Federation
Open platforms – Apache Iceberg, Delta Lake, Parquet, Polaris, Trino/Starburst
Data virtualization – Denodo, Starburst, Snowflake/Databricks federation, logical data fabric
Batch modernization – Migration from Oracle/PL‑SQL, mainframe, Unix ETL, Control‑M to cloud‑native orchestration
Multi‑cloud – AWS, Azure, GCP (storage, IAM, networking, private link)
Data modeling – Kimbal, Denormalization, Dimensional, medallion schema patterns, bridge tables, marts, PK, FK, Clustering, and Tuning Implications
Languages – SQL, Python, Java, Javascript, Scala, Spark, Notebooks, Snowpark
LINUX – Must Know Linux, comfortable with EC2 servers, VI, cat, etc.
Regulated industry experience – Financial Services, SOX, MRA, audit, NPPI/PII handling
Generative AI – Comfortable with Scripting and prompting. Understand when scripting is going to be more efficient and able to generate good scripts using well defined prompts and skill files.
Good Team and Communication Skills – Being Comfortable and willing to share your knowledge as well as learn from others. Coordinates well with other team members, project managers, and other teams.

Nice To Haves

Previous Redshift, DB2, SQL Server experience is a help for migrating off of these platforms and data conversion issues that arise.

Responsibilities

Define and govern the hybrid Snowflake–Databricks architecture to support classic Data Warehousing as well as Data Lake designs, aligning Snowflake's governed analytics layer with Databricks' engineering, streaming, and ML capabilities.
Establish reference blueprints and patterns using lakehouse and medallion patterns suited for enterprise scale and auditability.
Contribute Designs, Patterns, and Assets to SSIM Center of Excellence which establishes Best Practices for each platform as well as use cases for Reuse.
Lead Iceberg/Horizon/Unity Catalog interoperability across AWS, Azure, and GCP.
Architect open data platform strategy leveraging Apache Iceberg, Delta Lake, Parquet, and Polaris/Unity Catalog to avoid vendor lock‑in and enable cross‑engine query.
Define standards for open table format adoption, including: Business Tool Access Protocols, Drivers, Standards, Naming Standards, Folder Structures, Cloud Storage Security, Schema evolution, partitioning, and time‑travel.
Ensure cross‑engine compatibility (Snowflake, Databricks, Trino, Spark, Athena).
Establish catalog federation across Horizon, Unity, Glue, and Polaris.
Establish governance for open formats in regulated workloads (lineage, access control, encryption).
Drive interoperability between proprietary and open ecosystems to support modernization and exit strategies.
Capture feedback, findings, and suggestions from users and provide that distilled information back to vendors to improve their offerings.
Architect data virtualization patterns to enable logical access to data across Snowflake, Databricks, Oracle, SQL Server, and legacy NAS/file systems without physical movement.
Define use cases for: Federated queries (Snowflake external tables, Databricks Lakehouse Federation, Trino/Starburst, Denodo), Virtual data products for analytics and reporting, Logical data fabric spanning cloud and on‑prem sources.
Establish guardrails for performance, cost, security, and data classification in virtualized access patterns.
Reduce data duplication, copy proliferation, and storage cost through virtualization where appropriate.
Lead architecture and solutioning for batch applications migrating from legacy platforms (Oracle, mainframe, Unix/Linux ETL, file‑based workflows) to Snowflake/Databricks.
Define patterns for: Batch ingestion and orchestration (Airflow, Databricks Workflows, Snowflake Tasks/Streams, Control‑M integration), Window/SLA‑driven batch processing with monitoring and rerun safety, Migration of legacy stored procedures, PL/SQL, and shell‑based ETL to cloud‑native equivalents, Hybrid batch + streaming patterns (medallion, change data capture).
Establish batch performance, cost, and resiliency standards, including Reprocessing, Transaction Rollback, Event Logging, Query Timeout Limits, Warehouse Credit Limits, Finding/Flagging Expensive Processes from Account Usage Tables.
Partner with application owners to retire legacy infrastructure as part of data center exit and cloud migration programs.
Identify out-of-date drivers and application versions that could be security risks, and advise application teams on upgrades to modernize while migrating.
Architect and enforce data security by design, including: Design RBAC Roles at a Group Level for R, RW, Admin at DB, Schema, Object Levels, balancing security needs with reasonable administrative overhead.
Advise on Lower Level Row Access Policy, Column Masking, Secure Views so App Teams have row‑level and column‑level security where necessary.
Ensure compliance with SOX, MRAs, internal/external audits, and data protection controls.
Maintain defensible audit trails, lineage, and forensic traceability.
Align with GCS data residency, isolation, and classification standards.
Architect cloud‑native integrations across S3, ADLS, GCS, IAM, networking, and private link patterns.
Drive cross‑region/cross‑cloud HA/DR designs including Snowgrid replication and Iceberg federation.
Ensure vendor security, networking real‑time/streaming, and disaster recovery are embedded in design.
Act as SME for Snowflake, Databricks, open platforms, virtualization, and batch modernization during architecture reviews and escalations.
Partner with Security (GCS/GTS), Risk, Audit, Application, and Cloud Engineering teams.
Produce architecture diagrams, design artifacts, executive‑level presentations, and access models.

Benefits

retirement savings plan (401K) with company match
insurance coverage including basic life, medical, dental, vision, long-term disability, and other optional additional coverages
paid-time off including vacation, sick leave, short term disability, and family care responsibilities
access to our Employee Assistance Program
incentive compensation including eligibility for annual performance-based awards
eligibility for certain tax advantaged savings plans
inclusive development opportunities
flexible work-life support
paid volunteer days
vibrant employee networks