Principal Engineer - Observability Events Management

Wells Fargo & Company•Iselin, NJ

21h•Onsite

About The Position

Wells Fargo is seeking a Principal Engineer in Technology as part of their CTO organization. This role is key in driving technology-led innovation at an enterprise scale, specifically transforming Telemetry, Monitoring, and Observability across on-premises, private, and public cloud environments. The vision is to deliver best-in-class observability capabilities with consumer-grade insights at a competitive cost, leveraging open-source technologies and open standards. This modernization aims to reduce failure costs, improve platform availability, and optimize software engineering capacity across the enterprise.

Requirements

7+ years of Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
3+ years of experience with distributed streaming and event driven architectures, using technologies such as Kafka
3+ years of experience in database design and data architecture, including relational and/or NoSQL systems, data modeling, and high throughput data access patterns
7+ years of programming experience in Java, C# and/or Python, building scalable, production grade backend systems and APIs

Nice To Haves

AIOps expertise spanning alert ingestion, correlation, enrichment, and AI‑driven incident analysis within large‑scale enterprise observability platforms.
Experience in AI/ML technologies, including Large Language Models (LLMs), prompt engineering, model integration, or applied machine learning in production environments
Experience designing and integrating APIs and microservices, including REST/gRPC services and service orchestration patterns
Knowledge of API security, OAuth, authentication/authorization platforms, and reverse proxy or gateway patterns
Experience working with cloud native platforms like OCP, including containerization, distributed systems, and scalable deployment models
Experience incident & problem management, technology change management and performance testing and tuning
Experience with system design and architecture for high scale applications, including performance optimization, fault tolerance, and resiliency
Strong verbal, written, and interpersonal communication skills
Ability to influence and build relationships with LOB stakeholders, technology leadership, external service providers, and architecture teams
Experience of modern software development lifecycle, including TDD, CI/CD, Pairing, Build Automations, Automated Testing, Agile Games, Chaos Engineering
Knowledge of Agile methodologies, and product operating model
Knowledge and understanding of application or software security: web application penetration testing, secure code review, secure static code analysis
Knowledge and understanding of technology architecture: solutions development
Knowledge and understanding of complex enterprise systems and frameworks including frontends, middleware, services layer, database, backend, and downstream interfaces

Responsibilities

Act as an advisor to leadership to develop or influence applications, network, information security, database, operating systems, or web technologies for highly complex business and technical needs across multiple groups
Lead the strategy and resolution of highly complex and unique challenges requiring in-depth evaluation across multiple areas or the enterprise, delivering solutions that are long-term, large-scale and require vision, creativity, innovation, advanced analytical and inductive thinking
Translate advanced technology experience, an in-depth knowledge of the organizations tactical and strategic business objectives, the enterprise technological environment, the organization structure, and strategic technological opportunities and requirements into technical engineering solutions
Provide vision, direction and expertise to leadership on implementing innovative and significant business solutions
Maintain knowledge of industry best practices and new technologies and recommends innovations that enhance operations or provide a competitive advantage to the organization
Strategically engage with all levels of professionals and managers across the enterprise and serve as an expert advisor to leadership