DataOps Engineer

SBT Global•Englewood Cliffs, NJ

7d•Onsite

About The Position

We are looking for a mid‑level engineer to build and operate a data platform that uses Apache Iceberg as the lake‑house table format and Docker‑based micro‑services (Spark, Flink, Presto, etc.). You will own the end‑to‑end delivery pipeline, monitoring, security, and incident response, ensuring the platform runs reliably at scale.

Requirements

Bachelor’s degree in Computer Science, IT, Data Engineering, or a related field.
Approximately 5 years of hands-on experience building and operating large-scale data platforms (lake-house, data-warehouse, or big-data ecosystems).
Proven production experience with Apache Iceberg (table creation, partition management, schema evolution, catalog integration).
Strong Docker skills: multi-stage builds, Docker-Compose testing, routine image security scanning.
Experience with at least one major data-processing engine (Spark, Flink, or Presto/Trino) and its connection to Iceberg tables.
Proficiency in Python and/or Ansible for automating infrastructure and platform tasks.
Experience building CI/CD pipelines that include Docker linting, vulnerability scanning, and automated deployment of data-pipeline code.
Familiarity with observability tooling (Prometheus, Grafana, OpenTelemetry, Loki) and ability to create useful alerts and dashboards.
Ability to respond to incidents, write clear root-cause analysis reports, and contribute to post-mortem actions.
Willingness to participate in an on-call rotation as a first-line responder.
Availability to work on-site in New Jersey for the initial assignment and relocate to Dallas by October 2026.

Nice To Haves

Master’s degree.
Experience with cloud-native data services on AWS, Azure, or GCP (EMR, Dataproc, Synapse, etc.).
Familiarity with other lake-house formats such as Delta Lake or Apache Hudi and ability to evaluate trade-offs against Iceberg.
Knowledge of streaming platforms (Kafka, Pulsar, Kinesis) and real-time processing patterns.
Relevant certifications (Databricks Lakehouse Associate, Google Professional Data Engineer, AWS Certified Data Analytics – Specialty, etc.).
Background supporting data platforms in regulated industries (pharma, finance, healthcare) and understanding of associated compliance frameworks.

Responsibilities

Support Iceberg tables, manage schema changes, partitions, and snapshot retention, and keep the catalog synchronized.
Create and test Docker images for Spark/Flink/Presto using multi-stage Dockerfiles, local test environments with Docker-Compose, and vulnerability scans.
Build ETL/ELT jobs to ingest raw data and write to Iceberg tables, including simple streaming components.
Configure CI/CD pipelines for linting Dockerfiles, scanning images, versioning Iceberg metadata, and deploying pipelines without downtime.
Automate cluster provisioning, catalog configuration, vacuum/compaction, and other routine housekeeping tasks using Ansible/Python.
Instrument services with OpenTelemetry, Prometheus, Grafana, and Loki; create dashboards and set up alerts for pipeline latency, resource usage, table health, and error rates.
Monitor SLA metrics such as data freshness, job success rates, and query response times against agreed-upon targets.
Participate in the on-call rotation, perform first-line diagnosis and resolution of pipeline failures, Iceberg metadata issues, or container crashes, and write root-cause analyses.
Assist in enforcing security measures like image signing, mTLS, IAM roles, and bucket policies, and collaborate with the security team to meet compliance requirements (GDPR, HIPAA, ISO 27001).
Maintain internal documentation and conduct tech demos or brown-bag sessions on relevant technologies.