Lead Data Engineer
About The Position
The Data and Analytics team is part of Network and Collaboration Services (NCS) and plays a critical role in enabling informed, evidence-based decision-making across the organization. We are responsible for collecting and integrating network assets and network data from various sources and tools, then transforming this data into reliable insights and analytical solutions that drive organizational strategy and operations. We are seeking a skilled Lead Logstash/Elasticsearch Engineer to design, implement, and support enterprise-scale log management, data ingestion, and search analytics solutions. The ideal candidate must have strong hands-on experience with the Elastic Stack, solid Linux administration skills, Python scripting capabilities, and a foundational understanding of networking concepts.
Requirements
- 5+ years of experience working with Elasticsearch, Logstash, or related technologies
- Experience in supporting production environments with high availability requirements
- Bachelor's degree in Computer Science, Information Technology, Engineering, or related field (or equivalent practical experience).
- Proficiency in creating and managing Logstash pipelines, filters, and data transformations
- Knowledge of Elasticsearch indexing, mappings, queries, and cluster management
- Strong knowledge of Linux OS (RHEL, CentOS, Ubuntu, or similar)
- Expertise with Linux command-line tools, process management, file systems, and shell scripting
- Ability to troubleshoot application and system-level issues in Linux environments
- Proficiency in Python for automation, data processing, and operational tooling
- Experience developing scripts for monitoring, log analysis, and system administration
- Understanding of networking fundamentals like TCP/IP, DNS, HTTP/HTTPS, load balancing, and firewalls
- Ability to troubleshoot network connectivity and firewall issues
- Capability to analyze connectivity and data flow in distributed systems
Nice To Haves
- Knowledge of T-SQL. Experience writing queries, stored procedures, and troubleshooting database-related issues
- Familiarity with Microsoft SQL Server environments
- Experience with Kibana dashboards and visualizations
- Exposure to CI/CD pipelines and DevOps practices
- Knowledge of monitoring tools such as Prometheus, Grafana, or similar
- Experience with cloud platforms (AWS, Azure, or GCP)
- Familiarity with security monitoring, SIEM, or observability solutions
Responsibilities
- Design, develop, and maintain data ingestion pipelines using Logstash
- Configure, administer, and optimize Elasticsearch clusters for performance, scalability, and reliability
- Monitor and troubleshoot data flows, indexing processes, and search performance issues
- Develop and maintain automation scripts using Python
- Manage and support Linux-based environments hosting Elastic Stack components
- Analyze system logs and operational data to identify issues and improve platform stability
- Collaborate with infrastructure, application, and security teams to onboard new data sources
- Implement data retention, indexing, and lifecycle management strategies
- Document configurations, operational procedures, and troubleshooting guides
- Participate in incident response and root cause analysis activities.
Benefits
- bonuses
- flexible benefits
- competitive compensation
- commissions
- stock where applicable
- Leaders who support your development through coaching and managing opportunities
- Ability to make a difference and lasting impact
- Work in a dynamic, collaborative, progressive, and high-performing team
- A world-class training program in financial services
- Flexible work/life balance options
- Opportunities to do challenging work
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
