Senior Data Engineer (Databricks) - Contractor - US

Object Edge • Walnut Creek, CA

1d • Remote

About The Position

Object Edge is an AI-native, forward-deployed engineering firm that builds Hive, a company brain for revenue, embeds engineers to build agentic workflows on it, and provides management engineering to lead change. It is also an award-winning eCommerce consultancy delivering end-to-end design and development solutions centered around client goals and customer needs. Its global teams bring together curious, intellectual advisors who solve business problems and produce measurable results.

The Senior Data Engineer will be a technical leader in the Databricks practice, delivering innovative data solutions with minimal oversight. The role contributes to building modern data platforms for client organizations, working under the direction of a Solution Architect to design and implement components of client solutions. The engineer also participates in design sessions, breaks down complex development tasks, delivers development items on time, and contributes to Center of Excellence initiatives and Databricks solution accelerators. This position is remote within the United States.

Requirements

• At least 2 years of hands-on experience designing and developing data pipelines with Databricks, including Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
• Proficiency designing and building scalable, YAML configuration-driven pipelines for batch, micro-batch, and streaming ingestion and processing, including use of tools such as Auto Loader.
• Experience building complex job or workflow orchestration patterns using Databricks Jobs/Workflows or tools such as Airflow or dbt.
• Exposure to building data quality, pipeline audit, and observability solutions using Databricks native features or tools and frameworks such as Great Expectations, Collibra, or dbt.
• Proficiency with big data platforms including Apache Spark, Presto, and Amazon EMR.
• Experience with cloud data warehouses including Amazon Redshift, Snowflake, and Google BigQuery.
• Strong programming skills in SQL, stored procedures, and object-oriented programming languages such as Java, Python, or PySpark.
• Familiarity with building DevOps and CI/CD pipelines using Databricks Asset Bundles with automated validation and testing.
• Familiarity with NoSQL databases and container management systems.

Nice To Haves

• Exposure to Infrastructure as Code tools such as Terraform.
• Exposure to AI/ML tools such as MLflow, prompt engineering, and modern data and AI agentic workflows.
• Databricks Data Engineering Associate and/or Professional certification.
• Experience delivering multiple Databricks projects.

Responsibilities

• Develop and implement data solutions using Databricks, including features such as Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
• Collaborate with cross-functional teams to design and optimize batch and streaming data pipelines, ensuring data quality and efficiency.
• Stay current with emerging technologies and Databricks platform features, and continuously improve skills in Databricks and related tools.
• Provide technical leadership with minimal oversight to deliver solutions using cloud data lakehouse methodologies, distributed processing engines, event streaming platforms, and modern data technologies.
• Work under a Solution Architect's direction to help design and implement components of client data platform solutions.
• Participate in design sessions, break complex development tasks into deliverable work, and complete development items on time.
• Contribute to Databricks Center of Excellence initiatives and develop solution accelerators.

What This Job Offers

• Career Level: Senior
• Education Level: No Education Listed