DataOps Engineer

Greystar•Columbia, SC

5d•$120,000 - $150,000•Remote

About The Position

Greystar is seeking a DataOps Engineer to join the Data Marketplace (DMP) team. This is a deeply technical, hands-on platform engineering role at the core of Greystar’s enterprise data infrastructure — a Databricks-native medallion architecture (Bronze → Silver → Gold) running entirely on Microsoft Azure. You will own the reliability, scalability, and operational excellence of the DMP platform, working within DataOps pod inside the broader Analytics Engineering umbrella. This role is Databricks and Azure-heavy. Most of your day lives inside Databricks — Delta Live Tables, Unity Catalog, Jobs, Workflows — backed by the full Azure data services stack including ADF, ADLS Gen2, Azure Monitor, Key Vault, and more. Deep mastery of both platforms is a baseline expectation, not a differentiator. Critically, we expect this engineer to use AI as a first-class tool in their DataOps and observability practice — today, not eventually. That means AI-driven pipeline diagnostics, LLM-assisted root cause analysis, intelligent anomaly detection, and agentic observability agents that surface issues before they reach production. If you are still approaching DataOps the same way you did three years ago, this is not the right role. We are building self-aware, self-healing data infrastructure and need an engineer who is already operating that way. You will also own the full deployment lifecycle — promoting data pipeline changes and platform configurations across dev, staging, and production environments using GitHub Enterprise and Linear for structured release management. Strong CI/CD discipline, environment promotion hygiene, and release coordination are as important here as pipeline engineering craft.

Requirements

7+ years of DataOps, data engineering, or platform engineering experience in a production environment
Expert-level hands-on experience with Databricks: Delta Live Tables, Jobs/Workflows, Unity Catalog, Spark performance tuning, and Delta Lake internals
Strong command of the Azure data services ecosystem: ADF, ADLS Gen2, Azure Monitor, Log Analytics, Key Vault, and related services
Demonstrated, production use of AI tools in DataOps or data observability workflows — LLM-assisted diagnostics, intelligent alerting, agentic monitoring, or equivalent
Proven CI/CD experience using GitHub Enterprise — branch strategies, PR automation, environment promotion, and release management for data pipelines
Solid Python and/or Scala skills for pipeline development; SQL fluency for Gold layer transformation and DQ validation
Hands-on experience with ADF pipeline design and orchestration at scale
Experience with medallion / lakehouse architecture patterns and multi-environment deployment discipline
Strong collaborative skills across engineering, governance, and business stakeholder teams

Nice To Haves

Experience with Linear for engineering sprint management and release tracking
Familiarity with Databricks Mosaic AI, Genie, or other AI-native Databricks capabilities
Exposure to agentic AI frameworks or MCP (Model Context Protocol) server integrations
Background in real estate, property management, or multi-source ERP data environments (Yardi, Entrata, RealPage)
Experience with Cosmos DB, Azure SQL, or similar operational data stores alongside lakehouse platforms
Knowledge of data governance frameworks, data lineage tooling, and metadata management within Unity Catalog
Background in legacy BI migration or platform modernization programs

Responsibilities

Implement AI-powered observability — using LLMs and ML models to detect pipeline drift, classify anomalies, predict SLA risk, and generate automated incident summaries
Build agentic monitoring workflows that proactively surface data quality degradation, pipeline dropout, schema drift, and volume anomalies across all DMP layers
Integrate AI tooling (Databricks Mosaic AI, Genie, OpenAI APIs, or equivalent) into operational DataOps processes — not as experiments, but as production-grade capabilities
Develop and maintain AI-assisted root cause analysis tooling to reduce MTTR on pipeline failures, with structured learnings fed back into the platform
Contribute to Greystar’s 18-month agentic AI roadmap, leading near-term delivery of self-healing pipeline capabilities
Operate the full Azure data services stack supporting DMP: ADLS Gen2, Azure Data Factory (ADF), Azure Monitor, Log Analytics, Key Vault, and Event Hub
Design and maintain ADF pipelines for source system ingestion, including orchestration patterns for multi-tenant ERP environments (Yardi, Entrata, RealPage)
Collaborate with Azure infrastructure and cloud engineering teams on networking, identity, security, and resource provisioning
Drive cost governance through Azure Cost Management, Databricks DBU optimization, and storage lifecycle policies
Own the design, build, and optimization of data pipelines on Databricks using Delta Live Tables (DLT), PySpark, Workflows, and Jobs across the full DMP medallion stack
Administer and govern the Databricks workspace: Unity Catalog, cluster policies, access controls, compute configurations, and Delta table lifecycle management
Tune Spark jobs for performance, reliability, and cost — profiling bottlenecks, optimizing partitioning, managing Z-ordering, and controlling compute spend
Leverage Databricks Mosaic AI and Genie to build AI-native DataOps capabilities including intelligent pipeline monitoring, anomaly detection, and natural language data access
Architect and enforce DMP platform standards: naming conventions, schema evolution policies, SLA tiers, and medallion layer contracts
Own the full deployment pipeline for DMP data workflows — promoting changes from development through staging to production with rigor and minimal disruption
Build and maintain CI/CD workflows using GitHub Enterprise, including branch strategies, pull request automation, environment-specific configuration management, and release gating
Use Linear for sprint planning, release tracking, and issue management across deployment cycles; coordinate engineering work items with cross-functional stakeholders
Enforce deployment standards: automated testing gates, rollback procedures, change documentation, and environment parity controls
Partner with the analytics engineering and integration teams to align deployment cadences across the DMP stack
Instrument DQ checks across Bronze, Silver, and Gold layers covering completeness, consistency, accuracy, uniqueness, and referential integrity
Partner with Brett Finley’s Data Governance team to enforce data contracts, ownership standards, and quality SLAs within Unity Catalog
Build feedback loops between DQ scoring, pipeline observability, and upstream source owners to drive systemic data reliability improvements
Partner with analytics engineers, data governance, and product stakeholders to align pipeline and platform design with business requirements
Produce thorough technical documentation — runbooks, deployment playbooks, incident post-mortems, ADRs, and platform specs
Participate in on-call rotation and support SLA commitments for business-critical DMP data domains

Benefits

Competitive Medical, Dental, Vision, and Disability & Life insurance benefits.
Low (free basic) employee Medical costs for employee-only coverage; costs discounted after 3 and 5 years of service.
Generous Paid Time off. All new hires start with 15 days of vacation, 4 personal days, 10 sick days, and 11 paid holidays. Plus your birthday off after 1 year of service!
Additional vacation accrued with tenure.
For onsite team members, onsite housing discount at Greystar-managed communities are available subject to discount and unit availability.
6-Week Paid Sabbatical after 10 years of service (and every 5 years thereafter).
401(k) with Company Match up to 6% of pay after 6 months of service.
Paid Parental Leave and lifetime Fertility Benefit reimbursement up to $10,000 (includes adoption or surrogacy).
Employee Assistance Program.
Critical Illness, Accident, Hospital Indemnity, Pet Insurance and Legal Plans.
Charitable giving program and benefits.