Splunk Administrator (Site Reliability Engineer)

SchoolsFirst Federal Credit Union•Sacramento, CA

6d•$47 - $75•Onsite

About The Position

The Splunk Administrator (Site Reliability Engineer) is responsible for deploying, managing, and optimizing the Splunk platform. This role involves onboarding machine data, building queries using Search Processing Language (SPL), and designing dashboards to support IT Enterprise Applications, Security Operations, and observability practices. The position requires proactive monitoring of application health, efficient resolution of critical system outages through triage, analysis, debugging, and problem-solving, and serving as a subject matter expert for the Splunk Platform. The goal is to ensure enterprise applications and supporting platforms are fully functional, reliable, available, performant, and secure.

Requirements

High School Diploma or GED required
Bachelor's Degree in a related field or equivalent years of experience required
5-7 years of prior relevant experience required
Knowledge of Java Enterprise Edition and .NET frameworks.
Knowledge of standard java/.NET tools to debug application and system performance issues.
Knowledge of tuning application serving environments.
Knowledge of software release management and the SDLC.
Knowledge of Redhat Linux and Microsoft Windows Server 2008/2012.
Knowledge of security best practices for multi-tiered operating systems as well as application security practices.
Disaster recovery experience

Nice To Haves

Application Performance Management, such as Application Dynamics, Splunk and/or SNMP monitoring tools
IBM Security Access Manager (ISAM), IBM/Apache HTTP Server, IBM Secure Directory Server
Scripting skills for automating tasks
Experience with Docker, Kubernetes, Openshift, Python, Jenkins, Git/Gitflow, Ansible, Quay and Artifactory
Managing IBM AIX, Redhat, Linux and/or Microsoft Windows Servers, process development, support and implementation of systems architecture
Solid oral and written communication, project management, and interpersonal skills.

Responsibilities

Deploying, managing, and optimizing the Splunk platform.
Onboarding machine data, building queries using Search Processing Language (SPL), and designing dashboards.
Proactively monitoring and reporting on the environmental health of applications.
Engaging in critical system outages (triage) resolution efficiently to identify, review, analyze, debug and resolve issues.
Serving as a subject matter expert for the Splunk Platform.
Ensuring enterprise applications and supporting platforms are fully functional, reliable, available, performant, and secure.
Ingesting and onboarding new operational data sources, ensuring data is reliable, appropriately normalized, accurately timestamped, correctly categorized, and aligned to the Splunk Common Information Model where applicable.
Monitoring and maintaining Splunk platform health, including indexers, search heads, data flows, storage utilization, clustered or distributed components, internal logs, capacity trends, and service availability.
Resolving ingestion bottlenecks, performance issues, and platform outages.
Creating, maintaining, and improving real-time Splunk dashboards, visualizations, alerts, reports, metrics, logs, traces, and event views.
Crafting, tuning, and optimizing Splunk Processing Language (SPL) searches, scheduled searches, alert logic, reports, dashboards, and resource-intensive queries.
Configuring automated alerts and triggers for anomaly detection, system downtime, performance degradation, ingestion issues, and cyber or operational threats.
Alerting and accurately reporting KPIs on systems status with tuning recommendations at regular intervals.
Providing detailed analysis using APM, Splunk, and other monitoring or observability solutions.
Supporting a 24x7 production environment, including on-call rotation, deployment support, and timely response to critical incidents.
Responding quickly to incidents, investigating triggered alerts, isolating performance bottlenecks, and fixing issues.
Serving as an escalation point for complex application, infrastructure, observability, security, and platform issues.
Coordinating resolution across various organizational teams.
Supporting incident triage, root cause analysis, post-incident reviews, postmortems, and corrective action planning.
Identifying recurring issues and recommending preventive improvements.
Developing solutions to meet technical and business requirements, creating detailed design documents.
Demonstrating application environment tuning abilities and providing solutions to capacity requirements.
Contributing to automation and efficiency improvements for application and infrastructure buildout components, operational tasks, monitoring coverage, and deployment readiness.
Ensuring effective release management, change management, quality assurance, rollback planning, peer review, and deployment readiness in project lifecycles using an ITIL-centric approach.
Supporting production and non-production environments, accounting for differences in each environment for deployments and testing.
Peer reviewing changes within the Enterprise Applications team.
Maintaining and improving configuration documentation, diagrams, SOPs, runbooks, work/incident ticketing systems, knowledge articles, and training materials.
Ensuring adherence to configuration, operating, security, compliance, and governance best practices and standards.
Deploying and troubleshooting applications on both Java and .NET platforms.
Ensuring the highest levels of service for Members and Team Members.
Performing other duties as assigned.
Complying with regulatory compliance and assigned training requirements.