Senior Data Engineer

The Motley Fool

9d•$90 - $125

About The Position

The Motley Fool is seeking a Senior Data Engineer to design, build, and take full ownership of the data infrastructure powering their investment operations. This role involves owning the complete data lifecycle, from ingestion to transformation, warehousing, analytics, and machine learning. The engineer will be responsible for ensuring trustworthy, timely data for the Investment Committee and business partners. Key responsibilities include architecting ETL/ELT pipelines with Apache Airflow, building and optimizing a Snowflake data warehouse, developing analytical models and dashboards, and introducing machine learning capabilities for forecasting and anomaly detection. This position will drive the transformation of data infrastructure from manual processes to scalable, governed, automated systems, acting as the connective tissue between raw data and decision-making.

Requirements

5+ years of professional Python development, including object-oriented design and data manipulation libraries (pandas, NumPy).
Familiarity with financial research data vendors and feed/API products (e.g., CapIQ Xpressfeed, FactSet, Bloomberg, Thomson Reuters/Refinitiv/LSEG, Russell or MSCI).
Familiarity with financial business data and feed/API products from Broadridge, Morningstar, custodian banks, and fund administrators.
Proven experience designing and operating ETL/ELT pipelines.
Deep expertise in Snowflake architecture (clustering keys, micro-partitions, Snowpipe, Stages).
Ability to write complex analytical SQL, including window functions, CTEs, and recursive queries, and optimize them for cost and performance.
Hands-on experience with AWS CDK or Terraform, defining infrastructure in code.
Exposure to LLM integration patterns (markdown files, prompt engineering).

Nice To Haves

Apache Airflow and Lambda experience.
Experience with data visualization tools (Tableau, Streamlit, or similar).
Background in data governance, data cataloging, or data lineage tooling.
Demonstrated experience maintaining CI/CD pipelines for automated testing and deployment using Github actions and Terraform.
Experience profiling and optimizing queries across OLAP (Snowflake) and OLTP (PostgreSQL/Aurora) systems.
Familiarity with EXPLAIN plans, indexing strategies, and database-level performance tuning.
Practical experience building, evaluating, and deploying ML models.
Familiarity with common ML frameworks (scikit-learn, XGBoost).
Understanding of when and how to apply ML to business problems.
Experience with RAG, knowledge bases, and embeddings.

Responsibilities

Design, build, and maintain robust ETL/ELT pipelines using Apache Airflow (MWAA), handling complex dependencies.
Ingest data from diverse sources (SFTP, REST APIs, flat files, third-party financial data providers) and normalize it into a consistent analytical model.
Build automated data quality checks and monitoring systems to detect anomalies and alert the team.
Implement AWS Lambda functions for event-driven tasks like triggering ingestion or validating data.
Maintain and document the data catalog.
Serve as the subject-matter expert for Snowflake, designing schemas, managing data loading, and implementing access controls.
Write and optimize advanced analytical SQL queries for reporting, performance attribution, and ad-hoc analysis.
Profile and optimize slow-running queries in Snowflake using various techniques.
Define and deploy cloud resources using Terraform or AWS CDK, treating infrastructure as code.
Help design and maintain CI/CD workflows for automated testing and deployment.
Partner with teams to translate questions into data models, dashboards, and reports using Tableau.
Design and build automated pipelines to render data into production-ready marketing outputs (one-pagers, pitch decks, email campaigns, social content).
Act as a resource for software engineers to build an AI layer, enabling LLMs to query data and answer natural-language questions.