Lead Data Engineer (New York)

Fitch Group • Chicago, IL

5d • $140,000 - $160,000 • Hybrid

About The Position

Fitch Group is seeking a Lead/Principal Data Engineer to join their New York office. As a global financial information services provider, Fitch Group offers credit and risk insights, data, and tools to enhance financial markets. The Technology & Data Team is an innovative department focused on cutting-edge technologies like AI and cloud solutions, recognized as a "Best Place to Work in Technology" for three consecutive years. The team provides a supportive environment for professional growth and impact.

Requirements

• 8+ years of data engineering experience, including 3+ years in a lead role architecting large-scale data platforms.
• Expert-level proficiency in Java, Springboot for building cloud-native data processing solutions running on Docker/Kubernetes.
• Deep hands-on experience with Apache Airflow, Snowflake (data warehousing, modeling, optimization), and Databricks.
• Strong AWS expertise including S3, Lambda, Glue, EMR, Kinesis, EKS, RDS etc.
• Production database experience with PostgreSQL (design, optimization, replication) and MongoDB (document modeling, sharding, replica sets).
• Proven CI/CD and GitOps experience using GitHub, GitHub Actions, and ArgoCD for automated deployments and multi-environment management.
• Proficient with agile tools such as JIRA for sprint management and Confluence for technical documentation and knowledge sharing.
• Excellent analytical, problem-solving, and communication skills, with the ability to explain complex concepts to non-technical stakeholders and drive initiatives in complex environments.
• Working knowledge of AI/ML frameworks (LangChain, LlamaIndex, AutoGen, etc.) and understand how Agentic AI can enhance data engineering workflows through automated data validation, intelligent orchestration, and self-healing pipelines.
• Practical understanding of AI integration patterns in data platforms, including prompt engineering, RAG architectures, and vector database implementations.
• Familiarity with Model Context Protocol (MCP) or similar frameworks for enabling AI agents to interact securely and efficiently with data sources, APIs, and tools.
• Experience with AI-powered development tools such as GitHub Copilot and Amazon Q.

Nice To Haves

• Experience with code quality metrics and shift-left principles.
• Experience testing container resiliency (Docker/Kubernetes).
• Experience designing large end-to-end performance scenarios.
• Experience building large and high-performing data pipelines.
• Exposure to Playwright and BDD for automated testing.
• Exposure to the financial industry and data platforms (data warehouses, data lakes).
• Experience with modern data stack tools, data mesh/fabric architectures, and streaming platforms (Kafka, Kinesis).
• Proficiency with observability tools (Datadog) and data quality/governance frameworks.
• Understanding of data security and compliance standards (GDPR, SOC 2, CCPA) and contributions to open-source data projects.
• Relevant certifications (AWS Data Analytics/Solutions Architect, Databricks/Snowflake Data Engineer, CKA).
• Hands-on experience building production Agentic AI systems that operate on data platforms, including multi-agent orchestration and intelligent pipeline optimization.
• Deep expertise with Model Context Protocol (MCP) implementation, including building custom MCP servers or integration patterns for enterprise data platforms.

Responsibilities

• Lead the design and architecture of end-to-end data pipelines and solutions on modern cloud-based platforms, including Snowflake, Databricks, and AWS.
• Lead the design and architecture of end-to-end data pipelines and solutions on platforms including Java, Springboot, Docker/Kubernetes, Snowflake, Databricks, and AWS.
• Design and implement data solutions using PostgreSQL for relational data and MongoDB for NoSQL requirements, ensuring optimal performance and scalability.
• Architect and deploy containerized data applications using Docker, Kubernetes, and AWS EKS, incorporating GitHub Actions for automated deployments.
• Design and implement CI/CD pipelines using GitHub Actions, establish branching strategies, and ensure automated testing, code quality checks, and security scanning.
• Collaborate with cross-functional teams—including Data Scientists, Analytics teams, and business stakeholders—to translate requirements into scalable technical solutions.
• Mentor and guide data engineers by promoting technical excellence, establishing coding standards, and conducting architecture reviews.
• Drive data platform modernization initiatives and ensure data quality, reliability, and governance across all data systems.
• Design and implement AI-enhanced data pipelines that leverage LLMs and Agentic AI frameworks to automate data quality checks, anomaly detection, and intelligent data transformation workflows.
• Architect data infrastructure to support AI/ML workloads, including feature stores, vector databases, and real-time inference pipelines integrated with cloud-native services.
• Leverage established standards and best practices to integrate AI agents into data engineering workflows, including context management protocols (MCP) for seamless AI-to-data-platform communication.

Benefits

• Hybrid Work Environment: On-site presence required two days per week.
• A Culture of Learning & Mobility: Access to dedicated training, leadership development, and mentorship programs to support continuous learning.
• Investing in Your Future: Retirement planning and tuition reimbursement programs to help you meet your short- and long-term goals.
• Promoting Health & Wellbeing: Comprehensive healthcare offerings that support physical, mental, financial, social, and occupational wellbeing.
• Supportive Parenting Policies: Family-friendly policies, including a generous global parental leave plan, designed to help you balance work and family life.
• Inclusive Work Environment: A collaborative workplace where all voices are valued, supported by Employee Resource Groups that unite and empower colleagues worldwide.
• Dedication to Giving Back: Paid volunteer days, matched donation programs, and ample opportunities to volunteer in your community.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed