Director – Data & Systems Engineering

Boston Energy Trading and Marketing•Boston, MA

7d•Hybrid

About The Position

The Director - Data & Software Engineering is responsible for leading BETM's core Data Engineering and Software Engineering functions. This role owns the architecture, delivery, and operational excellence of enterprise data platforms, APIs, backend services, software applications, and engineering practices that support trading, risk management, asset management, analytics, operations, and internal business platforms. The role is primarily focused on data platform leadership and software engineering execution. The successful candidate will guide engineering teams, establish technical standards, drive modernization, oversee delivery of business-critical systems, and ensure platforms are scalable, secure, reliable, maintainable, and cost-effective. This leader must be hands-on enough to review architecture, challenge design decisions, understand code-level tradeoffs, and coach engineers toward pragmatic, high-quality solutions. While this is not intended to be a dedicated AI Architect role, the Director must have a strong working understanding of AI Engineering and collaborate closely with the AI Architect to ensure BETM's data, API, and software platforms are ready for AI-enabled use cases. This includes supporting RAG patterns, semantic layers, governed data access, vector/search integrations, model-facing APIs, observability, evaluation workflows, and responsible AI controls in partnership with the AI Architect and broader technology leadership.

Requirements

Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or equivalent practical experience.
15+ years of experience in software engineering, data engineering, platform engineering, or architecture.
7+ years of engineering leadership experience managing teams of 10+ engineers across data, software, API, platform, or application engineering functions.
Deep hands-on expertise in Python, Snowflake, Databricks, REST API design, FastAPI or comparable API frameworks, distributed systems, and modern software architecture principles.
Strong production experience with Snowflake, including data modeling, workload design, query performance, governance, cost optimization, operational monitoring, and platform reliability.
Strong experience with PostgreSQL, Redis, Azure cloud services, DevOps practices, CI/CD pipelines, automated testing, observability, and production operations.
Proven experience designing and operating scalable data platforms, data pipelines, curated datasets, semantic layers, data quality frameworks, metadata/lineage practices, and governed analytical systems.
Proven experience leading software engineering teams delivering APIs, backend services, integrations, internal applications, automation platforms, and business-critical systems.
Strong architectural judgment with the ability to review solution designs, evaluate technical tradeoffs, challenge assumptions, and guide teams toward maintainable and cost-effective implementation choices.
Working understanding of AI Engineering concepts, including LLMs, RAG, embeddings, vector databases, semantic search, model-facing APIs, AI evaluation, observability, and responsible AI controls.
Ability to collaborate effectively with an AI Architect and translate AI platform needs into data, API, security, integration, and software engineering requirements.
Experience supporting high-availability, business-critical platforms requiring reliability, scalability, security, auditability, monitoring, and disciplined production support.
Strong communication skills with the ability to engage executives, business stakeholders, architects, product owners, analysts, and engineering teams.

Nice To Haves

Preferred experience in energy trading, commodities, utilities, financial services, or other data-intensive industries.
Preferred experience with Snowflake Cortex, Databricks, Azure OpenAI, Azure AI Services, LangGraph, Semantic Kernel, MCP, or comparable AI/data platform ecosystems, with emphasis on integration and platform readiness rather than pure AI research.

Responsibilities

Lead and develop engineers across Data Engineering, Software Engineering, API development, platform services, and production support.
Own the strategic direction, architecture, delivery, and operational health of BETM's enterprise data platforms and major software systems.
Architect and develop latest trading and asset management platforms for management of client's assets and trade in the ISO markets.
Establish engineering standards, development practices, documentation expectations, delivery discipline, code quality, testing practices, and operational readiness expectations.
Define architecture standards for data platforms, APIs, backend services, business applications, distributed systems, and cloud-native engineering solutions.
Provide technical leadership across Python, SQL, FastAPI, REST APIs, PostgreSQL, Redis, Snowflake, Azure services, event-driven patterns, and distributed systems.
Lead design and modernization of software applications, internal platforms, data-backed products, and legacy systems while balancing delivery urgency with long-term maintainability.
Own data engineering strategy, including ingestion pipelines, curated datasets, analytical data models, semantic layers, metadata, lineage, governance, data quality, testing, monitoring, and documentation.
Drive Snowflake architecture, performance optimization, workload management, cost governance, access patterns, and platform reliability.
Oversee software engineering delivery across APIs, backend services, integrations, internal applications, automation workflows, and business-facing platforms.
Partner with the AI Architect to ensure data, API, and software platforms support AI Engineering patterns such as RAG, semantic search, embeddings, vector databases, AI evaluation workflows, and model-facing service integration.
Collaborate with the AI Architect on AI governance, data access controls, auditability, observability, hallucination-risk mitigation, and responsible AI practices where platform and data engineering decisions are involved.
Partner with stakeholders across trading, risk, asset management, operations, analytics, IT, and executive leadership to translate business objectives into actionable engineering roadmaps and delivery plans.
Manage prioritization, capacity planning, staffing plans, delivery commitments, stakeholder expectations, and tradeoff decisions across multiple engineering workstreams.
Own production supports routines, incident management, monitoring, alerting, runbooks, operational readiness, reliability improvements, and continuous improvement through automation.
Drive CI/CD, secure deployment practices, automated testing, observability, service-level expectations, engineering KPIs, and platform health metrics.