Data Platform Engineer

Greystar•Columbia, SC

13d•$125,000 - $150,000•Remote

About The Position

Greystar is seeking a senior, deeply technical platform operations engineer to join its Data team. The role is responsible for the reliability, governance, and cost efficiency of Greystar’s enterprise data infrastructure: a Databricks-native medallion architecture running on Microsoft Azure. The position works within the DataOps pod under the Analytics Engineering umbrella and owns Databricks workspace administration, Unity Catalog governance, Azure infrastructure operations, and multi-system platform support. The role focuses on Databricks workspace administration and the Azure data services stack, including ADLS Gen2, Azure Data Factory, Azure Monitor, Key Vault, Log Analytics, and Event Hub. Responsibilities also encompass support for EDP, DALDW, ADF, Synapse Analytics, Cosmos DB, and SQL Server; identity and access governance; compute and storage cost optimization; CI/CD and environment deployments; infrastructure as code; platform observability and self-healing; data pipeline standards; and disaster recovery. The position participates in on-call rotation and supports SLA commitments for business-critical data domains. The position may be performed remotely anywhere within the United States except Alaska. The posted annual salary range is $125,000–$150,000 USD. Corporate employees may be eligible for a quarterly or annual performance-based bonus. Benefits include medical, dental, vision, disability and life insurance; paid time off and holidays; a 401(k) match; paid parental leave; fertility benefit reimbursement; a sabbatical program; and other listed programs and insurance options.

Requirements

At least 5 years of platform engineering, infrastructure engineering, or data operations experience in a production environment.
Expert-level Databricks administration experience, including Unity Catalog, cluster policies, workspace management, compute configuration, and Delta Lake internals.
Strong command of Azure data services, including ADLS Gen2, ADF, Azure Monitor, Log Analytics, Key Vault, Cost Management, and Event Hub.
Identity and access management experience with Azure Active Directory, service principals, RBAC, and Unity Catalog governance policies.
Hands-on experience designing and orchestrating ADF pipelines at scale.
Spark performance tuning experience, including partitioning, Z-ordering, compute spend optimization, and bottleneck profiling.
Proven CI/CD experience using GitHub Enterprise, including environment promotion, rollback, and data-pipeline release management.
Python and/or SQL skills for automation scripts, compliance validation, and platform tooling.
Knowledge of medallion and lakehouse architecture patterns and multi-environment deployment discipline.
Terraform or other infrastructure-as-code experience provisioning and managing Databricks workspaces, Azure resources, Unity Catalog configurations, and environment-specific infrastructure through version-controlled IaC.
Azure DevOps Pipelines experience, including build and release pipeline configuration, YAML authoring, and GitHub Enterprise integration for end-to-end deployment automation.
Azure networking experience, including VNet configuration, private endpoints for Databricks and ADLS Gen2, NSG rules, firewall policies, and DNS resolution for secure data platform connectivity.
Azure Monitor and Log Analytics experience, including KQL authoring, alert rule configuration, diagnostic settings, workbook creation, and cluster event monitoring.
Knowledge of Delta Lake internals, including VACUUM, OPTIMIZE, ZORDER, time travel, table statistics, and transaction log management.

Nice To Haves

Experience with Synapse Analytics, Cosmos DB, or Azure SQL Server in production data environments.
Familiarity with Databricks Mosaic AI, Genie, or AI-native platform observability capabilities.
Background in data lineage tooling, metadata management, and Unity Catalog governance frameworks.
Experience with ERP integrations such as Yardi, Entrata, or RealPage in multi-tenant environments.
Background in legacy BI migration or platform modernization programs.
Experience using Linear for engineering sprint and release management.

Responsibilities

Administer Databricks workspaces, including upgrades, cluster policies, compute configurations, and Delta table lifecycle management.
Administer Unity Catalog access controls, service principals, groups, users, governance policies, and data lineage.
Tune Spark jobs for performance, reliability, and cost by profiling bottlenecks, optimizing partitioning, managing Z-ordering, and controlling compute spend.
Architect and enforce platform standards, including naming conventions, schema evolution policies, SLA tiers, and medallion layer contracts.
Use Databricks Mosaic AI and Genie to develop AI-native DataOps capabilities such as intelligent pipeline monitoring and anomaly detection.
Operate Azure data services including ADLS Gen2, Azure Data Factory, Azure Monitor, Log Analytics, Key Vault, and Event Hub.
Collaborate with Azure infrastructure and cloud engineering teams on networking, identity, security, and resource provisioning.
Drive cost governance through Azure Cost Management, Databricks DBU optimization, and storage lifecycle policies.
Provide platform support across systems and platforms, including access management, storage optimization, compliance, and resource provisioning.
Support ADF, Synapse Analytics, Cosmos DB, and Azure SQL Server.
Support compliance and audit readiness, including Key Vault secret rotation and management.
Own the deployment pipeline for DMP data workflows, promoting changes from development through staging to production.
Build and maintain GitHub Enterprise CI/CD workflows, including branch strategies, PR automation, environment-specific configuration management, and release gating.
Enforce deployment standards including automated testing gates, rollback procedures, change documentation, and environment parity controls.
Use Linear for sprint planning, release tracking, and issue management across deployment cycles.
Manage Azure Active Directory service principals, RBAC assignments, and Unity Catalog permission policies.
Partner with Data Governance to enforce data contracts, ownership standards, and quality SLAs within Unity Catalog.
Maintain data lineage, metadata management, and governance framework documentation.
Ensure platform configurations meet security, compliance, and audit requirements.
Maintain configuration parity across development, staging, and production Databricks environments, including cluster policies, Unity Catalog hierarchy, access controls, and compute configurations.
Define and enforce environment promotion hygiene, including configuration drift detection, environment-specific secret management, and pre-production validation gates.
Maintain infrastructure-as-code scripts and configuration templates for reproducible, auditable provisioning across environments.
Produce technical documentation, including runbooks, deployment playbooks, incident post-mortems, ADRs, and platform specifications.
Partner with analytics engineers, data governance, and product stakeholders on platform design and capacity planning.
Participate in the on-call rotation and support SLA commitments for business-critical DMP data domains.
Build and maintain AI-powered platform observability that uses LLMs and ML models to detect pipeline drift, classify anomalies, predict SLA risk, and generate incident summaries.
Architect and deploy production-grade self-healing pipeline capabilities, including automated retry logic, circuit breakers, and remediation triggers.
Integrate Databricks Mosaic AI and Genie into operational platform workflows for intelligent cluster management, anomaly detection, and natural-language platform diagnostics.
Contribute to Greystar’s 18-month agentic AI roadmap, leading near-term delivery of platform-level self-healing and proactive alerting capabilities.
Own infrastructure-oriented DMP pipeline architecture patterns, including Delta Live Tables design standards, schema evolution enforcement, and medallion layer contracts.
Define and enforce platform-wide naming conventions, partitioning strategies, Z-ordering policies, and Delta table optimization schedules such as VACUUM and OPTIMIZE.
Partner with analytics engineers to ensure pipeline designs conform to infrastructure standards, SLA tiers, and compute budget constraints.
Design and maintain disaster recovery strategies, including Delta table backup policies, ADLS Gen2 geo-redundancy configuration, and Databricks workspace recovery runbooks.
Define recovery time and recovery point objectives for business-critical DMP data domains and regularly test and validate recovery procedures.
Maintain business continuity plans aligned with Azure region failover capabilities and enterprise SLA commitments.
Manage ServiceNow or Asana ticket queues in accordance with team SLA commitments, track and close Linear or DevOps work items, and contribute to team knowledge bases, runbooks, and post-incident documentation.

Benefits

Competitive medical, dental, vision, disability, and life insurance, including low-cost or free basic employee-only medical coverage.
Paid time off, including 15 vacation days, 4 personal days, 10 sick days, and 11 paid holidays for new hires; birthday off after one year of service; and additional vacation accrued with tenure.
Onsite housing discounts at Greystar-managed communities for onsite team members, subject to discount and unit availability.
Six-week paid sabbatical after 10 years of service and every five years thereafter.
401(k) with company match up to 6% of pay after six months of service.
Paid parental leave.
Lifetime fertility benefit reimbursement up to $10,000, including adoption or surrogacy.
Employee Assistance Program.
Critical illness, accident, hospital indemnity, pet insurance, and legal plans.
Charitable giving program and benefits.
Potential eligibility for a quarterly or annual bonus based on individual and company performance.
Flexible work arrangements.