Senior Associate - Elasticsearch Engineer

New York Life•New York, NY

3d•Hybrid

About The Position

As part of Technology, you'll have the opportunity to contribute to groundbreaking initiatives that shape New York Life's digital landscape. Leverage cutting-edge technologies like Generative AI to increase productivity, streamline processes, and create seamless experiences for clients, agents, and employees. Your expertise fuels innovation, agility, and growth — driving the company's success. The Elasticsearch Engineer will work on a team supporting the day-to-day operation, performance, and reliability of the enterprise Elasticsearch platform that serves as the core of our security data lake. This is a mid-level engineering role (3–5 years of experience) embedded within the Security Data Platform team, focused on Elasticsearch cluster operations, ingest pipeline development, and data architecture.

Requirements

Hands-on experience operating and troubleshooting multi-node Elasticsearch clusters (40+ nodes) including shard allocation, recovery tuning, backpressure diagnosis, and node-level resource management
Strong understanding of Index Lifecycle Management (ILM) policies across hot/warm/cold/frozen tiers, including searchable snapshots and frozen-tier index restoration workflows
Experience building and maintaining ingest pipelines using native Elasticsearch processors (grok, set, rename, convert, script, pipeline chaining) with a preference for processor-based approaches over Painless where possible
Working knowledge of Painless scripting for ingest-time field transformations, conditional logic, and data normalization
Proficiency with index templates, component templates, and data stream architecture — including understanding of mapping conflicts, dynamic templates, and failure store indices
Familiarity with Elastic Common Schema (ECS) field mapping conventions and how to apply them to security log sources during ingest
Experience with data stream rollovers, reindexing operations, and mapping migration strategies for live production data
Ability to write and optimize ES|QL and KQL queries for security use cases, and build/maintain Kibana dashboards and data views
Experience monitoring and tuning search performance including slow query log analysis, shard sizing strategies, query profiling, and understanding the impact of mapping choices (keyword vs text, doc_values, subobjects) on query efficiency
Familiarity with cluster health and performance monitoring via Kibana Stack Monitoring and Devtools for diagnosing allocation and performance issues

Nice To Haves

Experience with cross-cluster search (CCS) and remote cluster configuration in multi-cluster architectures
Familiarity with Terraform-managed Elasticsearch resources (roles, API keys, index templates, data views)
Exposure to Cribl Stream or similar log routing/transformation platforms feeding into Elasticsearch via HEC or Elasticsearch output
Understanding of compliance-driven data retention requirements (e.g., NY DFS, NAIC) and how they map to ILM/tier policies
Experience with Elastic Security app, detection rules, or security-focused Kibana content
Experience with Elastic Cloud cost management including deployment sizing, autoscaling behavior, data tier cost optimization (hot vs frozen storage economics), and identifying savings opportunities through shard consolidation, ILM tuning, or field reduction at ingest
Understanding of capacity planning — forecasting storage and compute needs based on ingest rates, retention requirements, and query workload patterns

Responsibilities

Managing a large Elasticsearch 9.x cluster environment
Building and maintaining ingest pipelines that normalize high-volume security log sources to Elastic Common Schema (ECS)
Designing and tuning index templates and data stream lifecycles across hot/warm/cold/frozen tiers
Ensuring the platform meets performance SLAs and compliance-driven retention requirements