Technical Director, Data Engineering

Diligent Corporation•Vancouver, BC

7d•CA$180,000 - CA$210,000•Hybrid

About The Position

Diligent is looking for a Technical Director of Data Engineering to help design and build the next generation of data and AI infrastructure powering our products and internal platforms. This is not a traditional management role. We are looking for an experienced builder; someone who still enjoys writing code, debugging distributed systems, evaluating frameworks, and getting hands-on with architecture and implementation. You will work across large-scale ingestion pipelines, search systems, AI/LLM infrastructure, event-driven architectures, vector databases, analytics platforms, and real-time data processing. You should be equally comfortable discussing high-level architecture with senior leadership and diving into a failing Kubernetes pod or optimizing a Spark job. The ideal candidate has strong opinions informed by real-world experience, understands tradeoffs deeply, and can move quickly without creating unnecessary complexity.

Requirements

Significant real-world experience building and operating production systems using many of the following:
Data & Distributed Systems
Airflow
Elasticsearch / OpenSearch
Vector databases and semantic retrieval systems
MongoDB, PostgreSQL, DynamoDB, or similar platforms
Cloud & Infrastructure
AWS
AWS CDK
Serverless architectures
Distributed observability and monitoring stacks
AI / Search / Modern Data Applications
LLM integration patterns
RAG architectures
Embeddings and vector search
MCP servers and AI orchestration frameworks
LangChain, LlamaIndex, DSPy, or similar ecosystems
AI evaluation, tracing, and observability tooling
Search relevance and ranking systems
Backend Engineering
Python strongly preferred
Experience with Java, Go, or TypeScript is a plus
API design and distributed service architectures
Event-driven and asynchronous systems
Still enjoys building and debugging systems directly
Has strong technical depth, not just architectural vocabulary
Comfortable operating in ambiguity and fast-moving environments
Understands how to simplify systems instead of endlessly abstracting them
Has experience modernizing legacy platforms and evolving architectures incrementally
Can distinguish between engineering fundamentals and hype cycles
Values shipping working systems over theoretical perfection

Nice To Haves

Experience with large-scale news, document, or regulatory data pipelines
Experience with search relevancy tuning and semantic retrieval
Exposure to governance, compliance, risk, or enterprise SaaS platforms
Familiarity with modern AI engineering workflows and agentic systems
Experience supporting both startup-speed execution and enterprise-scale operations

Responsibilities

Design and build scalable data platforms and distributed processing systems
Develop modern ingestion, transformation, and retrieval pipelines for structured and unstructured data
Build systems supporting AI/LLM applications, semantic search, RAG pipelines, vector search, and agentic workflows
Work hands-on with engineering teams to implement production-grade solutions rather than producing slideware
Evaluate and standardize frameworks, tooling, and infrastructure patterns across teams
Improve performance, reliability, observability, and cost efficiency of data systems
Partner with product and platform engineering teams to accelerate delivery of AI-native capabilities
Drive pragmatic engineering decisions balancing speed, maintainability, and operational simplicity
Mentor engineers technically through design reviews, architecture guidance, and pair debugging
Help establish engineering standards around CI/CD, testing, data quality, monitoring, and operational excellence