Senior Splunk Engineer

Leidos•Arlington, VA

About The Position

Leidos has an immediate need for a Splunk Engineer SME for a new customer on a highly-visible and strategic Cybersecurity Task Order. The Splunk Engineer SME will install and maintain Splunk infrastructure, gather requirements from customers, onboard data, and assist end users with searches, dashboards, reports, and knowledge objects. The Splunk Engineer SME may be required to interact with senior management, as necessary. Conceptualize, design, build, and maintain current and future customer-supported tools and platforms. Manage multiple assignments, changing priorities, and work independently with little oversight. Develop data storage, access, and retention strategies for a large IT enterprise using industry standards and best practices to advise Customer executive-level stakeholders. Design, build, implement, and administer Splunk infrastructure in on-prem and cloud environments. Create, manage, and support automation solutions for Splunk deployment and orchestration in on-prem and cloud environments. Work with existing and custom Splunk applications and add-ons to fulfill customer needs. Provide overall engineering and design support for a distributed Splunk environment consisting of heavy forwarders, indexers, and search head servers, spanning security, performance, and operational roles. Onboard data to Splunk via forwarder, scripted inputs, TCP/UDP, and modular inputs from various sources. Normalize data to ensure CIM compliance, and develop data models to accelerate queries, dashboards, and correlation searches. Work closely with all relevant stakeholders to solve technical problems at the network, system, and application levels. Conduct periodic architectural reviews of Splunk and related systems to assess effectiveness and propose optimal installation alternatives as required. Develop and manage comprehensive documentation, artifacts, procedures, and processes for the optimal management of the Splunk infrastructure.

Requirements

Bachelor's Degree and 12-15 years of experience in system administration, database administration, network engineering, software engineering, or software development, with a concentration in Cybersecurity. Additional years of experience and Cyber certifications may be considered in lieu of a degree.
12-15 years of experience with Linux and Windows system administration or an expert understanding of operating systems and common operating environments
Ten (10) years of experience with Splunk in distributed deployments
Current Splunk Enterprise Certified Architect certification
Excellent written and oral skills, ability to work closely with multiple customers, manage expectations, and track engagement scope
Experience implementing FISMA, NIST, NSA, and other information security, cybersecurity and CDM-related industry policies, procedures, guidelines, standards, and best practices
Expert-level knowledge and ability with Splunk Enterprise Security or integration with other Security Information and Event Management (SIEM) platforms
Extensive experience with advanced configuration of Splunk including Indexer Clustering and Search Head Clustering
Proficient at data on-boarding activities including routing, parsing, and normalizing events to the Splunk Common Information Model (CIM)
Proficiency onboarding data using Splunk-developed add-ons for Windows, Linux, and common third-party devices and applications
Experience onboarding data into Splunk via forwarder, scripted inputs, TCP/UDP, and modular inputs from a variety of sources
Proficiency managing Splunk using the Splunk command-line interface
Proficiency managing Splunk using configuration files
Experience collaborating with separate engineering teams to configure data sources for Splunk integration
Intermediate understanding of SQL and common SQL dialects
Proficiency implementing and onboarding data in Splunk DB Connect
Experience with Splunk performing systems administration, including performing installation, configuration, monitoring system performance and availability, upgrades, and troubleshooting
General networking and security troubleshooting (firewalls, routing, NAT, etc.)
Splunk architecture/design, implementation, and troubleshooting experience
Experience in managing, maintaining, and administering multi-site indexer cluster
Proficiency developing log ingestion and aggregation strategies per Splunk best practices
Perform integration activities to configure, connect, and pull data with 3rd party software APIs
Experience implementing and optimizing Splunk data models
Proficient in regular expressions
Scripting and development skills (Bash, Python, or PowerShell)
Ability to autonomously prioritize and successfully deliver across a portfolio of projects
Department of Homeland Security ESOC employees are required to obtain an Entry on Duty (EOD) clearance to support this program

Nice To Haves

Splunk Certified Consultant
Splunk Certified Architect
CASP
GCWN
GISF
GSSP
GICSP
CCNP
CCNP Security
CCIE Security
CEH
ENSA
ECSP
MCSE
VCP
BCAP
VCIX
VCDX
RHCA
RHCE

Responsibilities

Install and maintain Splunk infrastructure
Gather requirements from customers
Onboard data to Splunk
Assist end users with searches, dashboards, reports, and knowledge objects
Interact with senior management
Conceptualize, design, build, and maintain current and future customer-supported tools and platforms
Manage multiple assignments, changing priorities, and work independently with little oversight
Develop data storage, access, and retention strategies for a large IT enterprise using industry standards and best practices to advise Customer executive-level stakeholders
Design, build, implement, and administer Splunk infrastructure in on-prem and cloud environments
Create, manage, and support automation solutions for Splunk deployment and orchestration in on-prem and cloud environments
Work with existing and custom Splunk applications and add-ons to fulfill customer needs
Provide overall engineering and design support for a distributed Splunk environment consisting of heavy forwarders, indexers, and search head servers, spanning security, performance, and operational roles
Onboard data to Splunk via forwarder, scripted inputs, TCP/UDP, and modular inputs from various sources
Normalize data to ensure CIM compliance, and develop data models to accelerate queries, dashboards, and correlation searches
Work closely with all relevant stakeholders to solve technical problems at the network, system, and application levels
Conduct periodic architectural reviews of Splunk and related systems to assess effectiveness and propose optimal installation alternatives as required
Develop and manage comprehensive documentation, artifacts, procedures, and processes for the optimal management of the Splunk infrastructure