SPLUNK ENGINEER

Zermount, Inc•Arlington, VA

About The Position

Zermount is seeking an experienced Splunk Engineer to support our client's enterprise security, operations, and monitoring environment. This role is responsible for the engineering, ongoing administration, maintenance, and enhancements of our client's Splunk environment, ensuring performance, scalability, and operational effectiveness. The ideal candidate brings proven operational experience in Splunk engineering and data ingestion, strong experience working within structured change management environments, and the ability to collaborate across infrastructure, network, and security teams.

Requirements

5+ years of hands-on experience engineering and administering enterprise Splunk environments, including multi-site clustered and distributed architectures.
Demonstrated expertise in log ingestion, data normalization, field extractions, and custom parser development across diverse data sources (network, application, database, cloud).
Proficient with Splunk Search Processing Language (SPL), including development of complex searches, alerts, reports, and dashboards.
Experience installing, configuring, upgrading, and performance tuning Splunk Enterprise in Linux environments (RHEL), including direct configuration of Splunk .conf files.
Experience integrating Splunk with enterprise security and operational tools, including: Splunk DB Connect and custom SQL queries syslog-ng configuration on RHEL (SELinux environments) Custom integrations using Python, Bash, or PowerShell
Experience supporting and optimizing distributed data pipelines, including administration of Cribl deployments and strategies to manage and reduce Splunk license consumption.
Experience performing major version upgrades and lifecycle management activities within production environments.
Experience operating within formal change management and ticket-driven workflows.
Ability to produce technical documentation, architecture diagrams, and implementation artifacts.
Required: A minimum of Splunk Certified Administrator Certification or higher AND at least one IT Security certification reflected on the DOD 8140 IAT level II baseline.

Nice To Haves

The following are additional certifications that are preferred but not required: Splunk Certified Architect or Splunk Core Consultant; Linux Administration, and Cribl Certification

Responsibilities

Engineer, implement, configure, administer, maintain, upgrade, patch, and troubleshoot the Splunk Enterprise platform in accordance with client policies
Design and continuously evaluate Splunk architecture to ensure scalability, performance, and alignment with current and future operational requirements; assess existing implementations and recommend enhancements or redesigns
Onboard, ingest, parse, normalize, and troubleshoot new and existing data sources, including network traffic, application logs, databases, and cloud platforms
Develop and maintain custom data parsers, field extractions, and data models to ensure accurate and efficient data integration across enterprise systems
Install, configure, upgrade, and maintain Splunk Apps, Add-ons, and knowledge objects; extend platform functionality to meet operational needs.
Develop and maintain custom searches, alerts, reports, and dashboards to support internal stakeholders, SOC, leadership, and external users; review and enhance detection and reporting capabilities.
Monitor and optimize Splunk system performance, connectivity, license utilization, and overall platform health; conduct system tuning and capacity planning. Provide daily health check reports to management and stakeholders.
Perform major version upgrades and support full platform lifecycle management, including patching, backup validation, restoration testing, and decommissioning activities
Administer and troubleshoot Splunk infrastructure hosted on RHEL servers, including user account management, access controls, certificate maintenance, logging configuration, and configuration backups
Troubleshoot ingestion failures, platform issues, and integration challenges; coordinate with internal teams and external vendors through issue resolution
Develop technical documentation, architecture and data flow diagrams, and implementation strategies; participate in design reviews, testing cycles, and change management processes
Collaborate with stakeholders and management to define requirements, translate business needs into technical deliverables, and provide accurate status updates
Track, manage, and report on work through schedules, tickets (service, request, incident), workflows, status reports, dashboards, etc.
Provide engineering, administrative and technical support as required to other team members or tools as a member of a cross functional security engineering team.