Observability Engineer (Splunk)

SS&C • Jacksonville, FL

6d • Hybrid

About The Position

As a leading financial services and healthcare technology company based on revenue, SS&C is headquartered in Windsor, Connecticut, and has 27,000+ employees in 35 countries. Some 20,000 financial services and healthcare organizations, from the world's largest companies to small and mid-market firms, rely on SS&C for expertise, scale, and technology.

Requirements

• Splunk (On‑Prem Splunk Enterprise) Hands-on experience administering Splunk Enterprise in on‑prem environments (not exclusively Splunk Cloud).
• Strong understanding of configuration layering, knowledge objects, and Splunk’s data pipeline (input → parsing → indexing → search).
• Experience with distributed Splunk (SHC and indexer clustering), including troubleshooting and performance tuning.
• Working knowledge of SPL, dashboards, and alerting suitable for production operations.
• Linux / CLI Fluency (Non‑Negotiable) Comfortable operating primarily from the command line; can inspect and manipulate files and system state quickly and safely.
• Bash, systemd, process troubleshooting, and /proc familiarity.
• Text and data tooling: vi (or equivalent), regex, sed/awk, jq, pipelines.
• SSH identities & certificates, rsync, RPM packages/repos, tar/zip, DNS basics, curl.
• Filesystem and storage competence: mounts, disk space management, LVM concepts, growth/resize patterns.
• Practical Git skills (beyond the happy path).
• Operational Expectations Plan changes with impact analysis, communications, validation, and rollback.
• Demonstrate production rigor: backups, change windows, and post-change verification.
• Maintain clear documentation and handoffs; leave systems better than you found them.
• Degree in Computer Science or related degree
• 5+ years of experience

Nice To Haves

• Splunk Enterprise Certified Admin (or equivalent experience).
• SmartStore configuration and operational experience.
• AWS fundamentals (EC2, EBS, VPC) and hybrid connectivity patterns.
• Telemetry pipelines from containerized applications; OpenTelemetry familiarity.
• OpenShift/Kubernetes exposure; configuration management tools (Salt/Ansible); containers/K3S.
• Ability and willingness to teach and uplift team practices.

Responsibilities

• Assist in the day-to-day health, reliability, and performance of Splunk Enterprise (on‑prem), including installs, upgrades, monitoring, backups, and recovery.
• Design, operate, and evolve distributed Splunk architectures, including Search Head Clusters (SHC) and Indexer Clusters.
• Manage ingestion and indexing pipelines end-to-end, including forwarders, parsing rules, and index design/retention.
• Develop and maintain SPL searches, dashboards, alerts, and data models that are operationally meaningful and actionable.
• Operate storage and retention strategies; contribute to SmartStore planning/implementation where appropriate.
• Partner with infrastructure and application teams to onboard telemetry correctly and sustainably (logs, metrics, traces, and events).
• Define and enforce observability standards (naming, tagging, retention, alert thresholds, and dashboard patterns).
• Improve signal-to-noise: reduce alert fatigue and increase actionable, well-contextualized alerts.
• Execute production changes independently and safely: instance resizing/replacement, storage migrations, network/IP cutovers, and maintenance-window execution.
• Apply sound engineering judgment to minimize downtime and preserve rollback options.
• Create and improve runbooks, automation, and post-change verification checklists.

Benefits

• Flexibility : Hybrid Work Model and Business Casual Dress Code, including jeans
• Your Future: 401k Matching Program, Professional Development Reimbursement
• Work/Life Balance: Flexible Personal/Vacation Time Off, Sick Leave, Paid Holidays
• Your Wellbeing: Medical, Dental, Vision, Employee Assistance Program, Parental Leave
• Wide Ranging Perspectives: Committed to Celebrating the Variety of Backgrounds, Talents and Experiences of Our Employees
• Training: Hands-On, Team-Customized, including SS&C University
• Extra Perks: Discounts on fitness clubs, travel and more!

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Number of Employees: 1-10 employees