DataOps Engineer

Greystar•Columbia, SC

21h•$120,000 - $150,000•Remote

About The Position

Greystar is seeking a hands-on DataOps Engineer for its Data Marketplace (DMP) team. The role owns the reliability, scalability, and operational excellence of Greystar’s Databricks-native medallion data platform running on Microsoft Azure. It is part of the DataOps pod within the Analytics Engineering organization, with work spanning Databricks, Azure services, AI-powered observability, data quality, governance, and production deployment operations. The engineer will build and operate production-grade AI capabilities for pipeline diagnostics, anomaly detection, root cause analysis, and proactive monitoring, contributing to Greystar’s agentic AI roadmap and self-healing pipeline capabilities. The position also owns the deployment lifecycle for data pipeline changes and platform configurations across development, staging, and production, using GitHub Enterprise and Linear for release coordination. The role partners with analytics engineering, data governance, product stakeholders, source-system owners, and Azure infrastructure and cloud engineering teams. It includes producing technical documentation and participating in an on-call rotation to support service-level commitments for business-critical data domains. The position may be performed remotely anywhere in the United States except Alaska.

Requirements

At least 7 years of DataOps, data engineering, or platform engineering experience in a production environment.
Expert-level hands-on experience with Databricks, including Delta Live Tables, Jobs and Workflows, Unity Catalog, Spark performance tuning, and Delta Lake internals.
Strong command of Azure data services, including ADF, ADLS Gen2, Azure Monitor, Log Analytics, Key Vault, and related services.
Production use of AI tools in DataOps or data observability workflows, such as LLM-assisted diagnostics, intelligent alerting, or agentic monitoring.
CI/CD experience using GitHub Enterprise, including branch strategies, pull-request automation, environment promotion, and data-pipeline release management.
Solid Python and/or Scala skills for pipeline development and SQL fluency for Gold-layer transformation and data-quality validation.
Hands-on experience designing and orchestrating ADF pipelines at scale.
Experience with medallion or lakehouse architecture patterns and multi-environment deployment discipline.
Strong collaboration skills across engineering, governance, and business stakeholder teams.

Nice To Haves

Experience with Linear for engineering sprint management and release tracking.
Familiarity with Databricks Mosaic AI, Genie, or other AI-native Databricks capabilities.
Exposure to agentic AI frameworks or MCP (Model Context Protocol) server integrations.
Background in real estate, property management, or multi-source ERP data environments such as Yardi, Entrata, or RealPage.
Experience with Cosmos DB, Azure SQL, or similar operational data stores alongside lakehouse platforms.
Knowledge of data governance frameworks, data lineage tooling, and metadata management within Unity Catalog.
Background in legacy BI migration or platform modernization programs.

Responsibilities

Implement AI-powered observability using LLMs and ML models to detect pipeline drift, classify anomalies, predict SLA risk, and generate incident summaries.
Build agentic monitoring workflows to proactively identify data quality degradation, pipeline dropout, schema drift, and volume anomalies across DMP layers.
Integrate AI tools such as Databricks Mosaic AI, Genie, OpenAI APIs, or equivalent into production DataOps processes.
Develop and maintain AI-assisted root cause analysis tooling and feed structured learnings back into the platform.
Contribute to Greystar’s agentic AI roadmap and lead near-term delivery of self-healing pipeline capabilities.
Operate Azure data services supporting DMP, including ADLS Gen2, Azure Data Factory, Azure Monitor, Log Analytics, Key Vault, and Event Hub.
Design and maintain ADF pipelines for source-system ingestion, including orchestration for multi-tenant ERP environments.
Collaborate with Azure infrastructure and cloud engineering teams on networking, identity, security, and resource provisioning.
Manage costs through Azure Cost Management, Databricks DBU optimization, and storage lifecycle policies.
Design, build, and optimize Databricks pipelines using Delta Live Tables, PySpark, Workflows, and Jobs across the DMP medallion stack.
Administer and govern the Databricks workspace, including Unity Catalog, cluster policies, access controls, compute configurations, and Delta table lifecycle management.
Tune Spark jobs for performance, reliability, and cost, including profiling bottlenecks, optimizing partitioning, managing Z-ordering, and controlling compute spend.
Use Databricks Mosaic AI and Genie to build AI-native DataOps capabilities, including pipeline monitoring, anomaly detection, and natural-language data access.
Define and enforce platform standards for naming conventions, schema evolution policies, SLA tiers, and medallion-layer contracts.
Own the deployment pipeline for DMP data workflows, promoting changes through development, staging, and production environments.
Build and maintain GitHub Enterprise CI/CD workflows, including branch strategies, pull-request automation, environment-specific configuration, and release gating.
Use Linear for sprint planning, release tracking, and issue management, coordinating work with cross-functional stakeholders.
Enforce deployment standards including automated testing gates, rollback procedures, change documentation, and environment parity controls.
Align deployment cadences with analytics engineering and integration teams.
Implement data-quality checks across Bronze, Silver, and Gold layers for completeness, consistency, accuracy, uniqueness, and referential integrity.
Partner with the Data Governance team to enforce data contracts, ownership standards, and quality SLAs within Unity Catalog.
Build feedback loops between data-quality scoring, pipeline observability, and upstream source owners to improve data reliability.
Partner with analytics engineers, data governance, and product stakeholders to align pipeline and platform design with business requirements.
Create technical documentation, including runbooks, deployment playbooks, incident post-mortems, ADRs, and platform specifications.
Participate in the on-call rotation and support SLA commitments for business-critical DMP data domains.

Benefits

Competitive medical, dental, vision, disability, and life insurance benefits; low-cost (free basic) employee-only medical coverage.
Paid time off: 15 vacation days, 4 personal days, 10 sick days, and 11 paid holidays for new hires; birthday off after one year of service and additional vacation accrued with tenure.
Onsite housing discount at Greystar-managed communities for onsite team members, subject to discount and unit availability.
Six-week paid sabbatical after 10 years of service and every five years thereafter.
401(k) with company match up to 6% of pay after six months of service.
Paid parental leave and lifetime fertility benefit reimbursement up to $10,000, including adoption or surrogacy.
Employee Assistance Program.
Critical illness, accident, hospital indemnity, pet insurance, and legal plans.
Charitable giving program and benefits.
Potential eligibility for a quarterly or annual bonus program based on individual and company performance.
Flexible work arrangements.