Splunk Production Services Engineer

Bank of America•Pennington, AL

22h•Onsite

About The Position

At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. We do this by driving Responsible Growth and delivering for our clients, teammates, communities and shareholders every day. Being a Great Place to Work and providing a culture of caring is core to how we drive Responsible Growth. We are intentional about fostering an inclusive workplace where every teammate has the opportunity to succeed, build a career and contribute to our shared success. This includes attracting and developing exceptional talent, recognizing and rewarding performance, and supporting our teammates’ physical, emotional, and financial wellness through affordable, competitive and flexible benefits. We value the unique perspectives individuals bring from all backgrounds and career paths - whether shaped by military service, community college education, or a wide range of work and life experiences. These journeys foster resilience, leadership and innovation, strengthening our workforce and positively impact the communities we serve. Bank of America is committed to an in-office culture that supports collaboration, engagement, and career development. Our approach includes clear in-office expectations, while providing an appropriate level of flexibility based on role-specific responsibilities and business needs. At Bank of America, you can build a successful career with opportunities to learn, grow, and make an impact. Join us! Position Summary: We are seeking a highly skilled Splunk Production Services Engineer to support and operate a large-scale, business‑critical Splunk Enterprise and Splunk Cloud platform within a financial services environment. Splunk is a foundational capability for the Information Security organization, enabling real-time security monitoring, threat detection, investigations, and regulatory reporting. This role is accountable for production stability, performance, data integrity, and security log readiness, requiring deep technical expertise and a strong operational ownership mindset. The engineer will act as a trusted platform owner, ensuring Splunk availability, scalability, and reliability while partnering closely with Information Security, SOC, architecture, engineering, and operations teams.

Requirements

5+ years of hands-on experience administering large-scale Splunk Enterprise or Splunk Cloud environments
Strong expertise in: Indexer clustering and search head clustering
Universal and heavy forwarder architectures
SmartStore / S3-compatible object storage
SPL, search optimization, summary indexing, data model acceleration
Deep experience with security log ingestion and SIEM use cases
Proven ability to lead production incidents, perform RCA, and drive preventive solutions
Strong Linux administration skills and experience managing Splunk configuration and apps
Experience working in 24x7 production environments with high availability expectations
Excellent written and verbal communication skills, with the ability to engage senior technical and business stakeholders
A production owner’s mindset
Deep technical credibility in Splunk and data pipelines
Ability to operate calmly and decisively during high‑severity security and platform incidents
Strong partnership with Information Security, where Splunk availability and data quality are mission‑critical to protecting the bank

Nice To Haves

Splunk certifications such as Enterprise Admin or Enterprise Architect
Experience with Splunk Enterprise Security (ES) and SOAR (Phantom or equivalent)
Exposure to cloud logging and security architectures (AWS, Azure, GCP)
Knowledge of Red Hat Enterprise Linux and Windows Server administration
Experience with monitoring, APM, and event management tools
Strong understanding of security, network, system, and database operations
Ability to balance multiple priorities in a fast-paced, enterprise production environment

Responsibilities

Own end-to-end production support for a highly distributed Splunk Enterprise and Splunk Cloud environment, including search head clusters, indexer clusters, deployers, deployment servers, and forwarders
Ensure high availability, performance, and resiliency of the Splunk platform supporting security and operational use cases
Lead incident response, troubleshooting, root cause analysis (RCA), and service restoration for Splunk and Cribl platforms
Proactively identify risks, capacity constraints, and performance bottlenecks; implement preventive and tuning measures
Serve as a key technical enabler for Information Security and SOC teams, ensuring timely, accurate, and reliable ingestion of security logs
Onboard and normalize new data sources, supporting CIM compliance, field normalization, and SIEM best practices
Tune ingestion pipelines using props.conf and transforms.conf, index-time and search-time optimizations
Build and support dashboards, searches, and alerts that enable threat detection, investigations, and reporting
Administer and support the Cribl environment for data routing, filtering, enrichment, and cost optimization
Ensure data integrity, reliability, and performance across Splunk ingestion pipelines
Collaborate with architecture teams on data flow strategies and onboarding standards
Develop and maintain runbooks, SOPs, installation guides, and operational documentation
Adhere to change management, incident management, and SLA commitments using ITSM tools
Operate effectively in a regulated banking environment, supporting auditability and compliance requirements