Python Software Engineer (Data Engineering)

Critical Propulsion

1d

About The Position

This role involves owning production-grade Python ingestion pipelines for a global data marketplace that handles over 120 million files and 1.4 TB of data weekly, serving dozens of active data products. The engineer will design and build systems to reliably move data from various sources into a governed, high-availability estate, with code that runs directly in production.

Requirements

• Proficiency in writing production-grade Python code, utilizing concepts like class hierarchies, abstract base classes, dependency injection, and polymorphism.
• Experience architecting master controller/services/helpers structures and explaining design decisions.
• Ability to build RESTful API integrations that handle authentication, pagination, error handling, and retry logic.
• Proficiency in writing SQL for querying, transformation, and pipeline validation.
• Experience building fault-tolerant, resilient, and modular ingestion pipelines with a focus on observability and traceability.
• Experience working with Azure data infrastructure (ADLS, Azure SQL).
• Ability to operate autonomously and communicate directly.
• Experience with agentic development and critically reviewing AI-generated output.

Nice To Haves

• Azure Databricks and PySpark for high-volume distributed processing.
• Familiarity with XBRL and iXBRL financial data formats.
• Git/GitHub fluency and experience with trunk-based or short-lived branch workflows.
• Exposure to near real-time processing patterns.

Responsibilities

• Design and build production-grade Python ingestion pipelines for a global data marketplace.
• Develop systems that move data reliably from real-world sources into a governed, high-availability estate.
• Write Python code adhering to best practices, including class hierarchies, abstract base classes, dependency injection, and polymorphism.
• Architect master controller/services/helpers structures and articulate design choices.
• Build robust RESTful API integrations with proper authentication, pagination, error handling, and retry logic.
• Write SQL for querying, transformation, and pipeline validation.
• Build fault-tolerant, resilient, and modular ingestion pipelines with a focus on observability and traceability.
• Work with Azure data infrastructure, including ADLS and Azure SQL.
• Operate autonomously, minimizing unnecessary meetings and documentation.
• Communicate directly and provide evidence when issues arise.
• Embrace agentic development by delegating work to AI agents, critically reviewing their output, and iterating quickly.

Benefits

• A team where everyone builds, with no layers of management between individuals and the work.
• AI agents as teammates to accelerate development, testing, and documentation.
• Pulse cycles that provide rhythm without traditional sprint overhead.
• Direct client impact with pipelines moving real data to real consumers quickly.
• Competitive compensation sized for senior consultants.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed