Principal Engineer Automation

Wells Fargo•Irving, TX

About The Position

This role involves acting as a trusted technical advisor to senior leadership, influencing the architecture and development of applications, platforms, APIs, network automation services, information security capabilities, data systems, operating environments, and cloud-native technologies. The engineer will lead the strategy and resolution of complex engineering challenges, design and develop production-grade software platforms, APIs, microservices, SDKs, libraries, automation frameworks, and reusable components. They will define software architectures for distributed, event-driven, and asynchronous systems, and establish software engineering standards for application structure, API design, code quality, automated testing, secure development, and production readiness. The role also includes developing reference implementations, contributing to technically complex portions of the platform, and leading technical design reviews, architecture reviews, code reviews, failure-mode analysis, and production-readiness assessments.

Requirements

7+ years of Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education
7+ years of software engineering, platform engineering, cloud engineering, infrastructure engineering, network automation, SRE, NRE, or equivalent experience demonstrated through one or a combination of work experience, training, military experience, or education.
5+ years of experience designing, developing, testing, and supporting production-grade software, distributed systems, automation platforms, or enterprise integration services.
3 plus years of Advanced software development experience using Python and at least one additional modern programming language such as Go, Java, C#, .NET, or TypeScript.
3+ years experience designing and developing APIs, microservices, asynchronous services, event-driven systems, workflow services, or comparable distributed software components.
3 plus years of experience developing software using Git-based workflows, pull requests, peer review, versioning, package management, CI/CD, automated testing, and release management.
3 plus years of experience deploying or supporting containerized applications using Docker, Kubernetes, OpenShift, or comparable container orchestration platforms.
3 plus years of experience integrating multiple systems through REST APIs, webhooks, messaging, events, SDKs, or service interfaces.
3 plus years of experience designing for scalability, availability, resiliency, observability, security, and operational support.

Nice To Haves

Strong knowledge of software engineering fundamentals, including data structures, algorithms, modular design, object-oriented or functional design, design patterns, concurrency, error handling, automated testing, and performance optimization.
Demonstrated experience leading technical design or architecture across multiple engineering teams, platforms, or organizational boundaries.
Strong analytical and systems-thinking skills with the ability to decompose ambiguous, enterprise-scale problems into executable engineering solutions.
Strong written, verbal, and visual communication skills, including the ability to explain complex software and architecture decisions to engineering teams, technology leaders, and executive stakeholders.
Prior experience in a Principal Engineer, Staff Engineer, Senior Staff Engineer, Software Architect, Platform Architect, or comparable technical leadership capacity.
Demonstrated experience serving as the technical lead for a large-scale software platform and driving architecture and engineering strategy across multiple teams without direct management authority.
Experience creating reference architectures, reusable software frameworks, engineering standards, or platform services adopted by multiple teams.
Experience leading architectural reviews, software design reviews, code-quality initiatives, production-readiness reviews, or cross-organizational technical programs.
Demonstrated ability to balance hands-on engineering with strategic influence, mentoring, governance, and long-term technical planning.
Deep understanding of distributed systems concepts, including consistency, availability, partition tolerance, concurrency, state management, failure recovery, idempotency, backpressure, retries, timeouts, and compensating transactions.
Experience designing high-volume, highly available, or mission-critical services.
Experience with event-driven architecture, message brokers, streaming platforms, service discovery, API gateways, caching, distributed data systems, and asynchronous processing.
Experience defining versioned APIs, service contracts, domain models, and backward-compatible integration strategies.
Experience with performance engineering, load testing, capacity analysis, profiling, and software optimization.
Experience building internal developer platforms, self-service engineering portals, platform-as-a-product capabilities, engineering enablement services, or reusable delivery frameworks.
Experience creating paved roads, golden paths, project templates, reusable pipelines, service catalogs, SDKs, or automated onboarding capabilities.
Experience measuring platform adoption, developer productivity, delivery lead time, reliability, support demand, or operational toil.
Experience with developer portals or service catalogs such as Backstage or comparable technologies.
Hands-on experience with Temporal, Celery, Argo Workflows, Apache Airflow, Dagster, or another durable workflow or asynchronous execution framework.
Understanding of workflow activities, workers, task queues, signals, queries, timers, retries, workflow versioning, replay behavior, and durable state.
Experience designing idempotent, fault-tolerant, long-running workflows with human approvals and external system dependencies.
Experience separating orchestration logic from execution logic, policy decisions, and user-facing intake processes.
Experience with Docker, Kubernetes, OpenShift, Helm, Harness, GitHub Actions, Azure DevOps, Argo CD, or comparable build and deployment technologies.
Experience with Terraform, OpenTofu, Ansible, Pulumi, Kubernetes operators, or comparable Infrastructure as Code technologies.
Experience operating software services in Azure, AWS, Google Cloud, private cloud, or hybrid-cloud environments.
Understanding of cloud well-architected practices, identity and access management, secrets management, networking, scalability, resiliency, and cost-conscious engineering.
Experience with service meshes, ingress platforms, API management, cloud-native networking, or policy-as-code technologies.
Experience developing software for network automation, infrastructure automation, configuration management, software lifecycle management, or infrastructure orchestration.
Knowledge of enterprise networking concepts and technologies, including routing, switching, wireless, firewalls, load balancing, DNS, IP address management, or network security.
Experience with network automation interfaces and technologies such as RESTCONF, NETCONF, gNMI, gRPC, YANG, SNMP, streaming telemetry, or vendor APIs.
Experience with Nautobot, NetBox, or another source-of-truth and network state-management platform.
Experience with Ansible, Nornir, pyATS, Terraform, or comparable infrastructure and network automation frameworks.
Familiarity with Cisco, Juniper, Arista, Palo Alto, Fortinet, F5, or comparable enterprise infrastructure platforms.
Experience implementing observability using OpenTelemetry, Prometheus, Grafana, ELK, Splunk, distributed tracing, structured logging, streaming telemetry, or flow analytics.
Experience establishing SLIs, SLOs, error budgets, health indicators, alerting standards, or production-readiness criteria.
Familiarity with incident management, root-cause analysis, automated remediation, resilience testing, chaos engineering, capacity planning, and operational toil reduction.
Experience correlating application behavior, infrastructure state, network telemetry, workflow execution, and business outcomes.
Experience building production AI, machine learning, generative AI, or AI-assisted engineering solutions.
Experience with retrieval-augmented generation, embeddings, vector search, semantic retrieval, knowledge graphs, structured model outputs, tool calling, or prompt orchestration.
Experience developing AI evaluation frameworks addressing groundedness, source coverage, retrieval quality, unsupported claims, classification performance, and human acceptance.
Understanding of responsible AI, model governance, explainability, prompt and model versioning, access controls, sensitive data protection, and human review.
Experience building data pipelines and working with structured and unstructured data, including JSON, YAML, logs, telemetry, documents, API payloads, and software repository content.
Experience with information security and technology risk management, including secure development, security architecture, threat modeling, policy and standards, security assessments, mitigation design, and control implementation.
Experience incorporating identity, authorization, least privilege, secrets management, data protection, auditability, and policy enforcement into software platforms.
Financial services experience or experience within another highly regulated industry.
Bachelor’s or advanced degree in Computer Science, Software Engineering, Computer Engineering, Information Systems, Data Science, or a related technical field.
Relevant certifications such as CCIE, CCDE, JNCIE, Microsoft Azure Solutions Architect Expert, AWS Certified Solutions Architect Professional, Google Professional Cloud Architect, Certified Kubernetes Administrator, Certified Kubernetes Application Developer, or comparable certifications.

Responsibilities

Act as a trusted technical advisor to senior leadership, influencing the architecture and development of applications, platforms, APIs, network automation services, information security capabilities, data systems, operating environments, and cloud-native technologies for highly complex business and technical needs across multiple organizations.
Lead the strategy and resolution of highly complex and unique engineering challenges requiring evaluation across multiple technology domains, delivering solutions that are long-term, large-scale, secure, resilient, and maintainable.
Design and develop production-grade software platforms, APIs, microservices, SDKs, libraries, automation frameworks, and reusable components that enable network and infrastructure engineering capabilities across the enterprise.
Define software architectures for distributed, event-driven, and asynchronous systems, including service boundaries, data contracts, workflow states, integration patterns, failure handling, concurrency, consistency, and recovery models.
Establish software engineering standards for application structure, API design, code quality, automated testing, secure development, dependency management, versioning, release engineering, observability, and production readiness.
Develop reference implementations and contribute directly to high-value or technically complex portions of the platform, particularly where new patterns, technologies, or engineering standards must be proven.
Lead technical design reviews, architecture reviews, code reviews, failure-mode analysis, and production-readiness assessments for critical platform capabilities.
Build and evolve internal engineering platforms that provide self-service automation, standardized workflows, reusable services, governed execution paths, and consistent developer experiences.
Treat shared engineering platforms as products, with clearly defined users, service contracts, roadmaps, adoption measures, documentation, support models, and reliability objectives.
Create paved roads and golden paths that enable engineering teams to develop, test, certify, release, and operate automation through approved patterns rather than one-off implementations.
Improve developer productivity through reusable APIs, templates, software development kits, CI/CD pipelines, test harnesses, local development environments, documentation, and automated onboarding.
Reduce duplicated engineering effort and operational toil by converting common functions into reusable platform services, shared libraries, automation modules, and supported integration patterns.
Establish appropriate boundaries among platform ownership, application ownership, production execution, operational support, and risk decision-making.
Design durable workflows for long-running, failure-prone, approval-dependent infrastructure and network processes using Temporal, Celery, or comparable workflow and asynchronous execution technologies.
Define patterns for workflows, activities, workers, task queues, events, signals, timers, retries, timeouts, compensating actions, versioning, idempotency, replay safety, and human approval gates.
Build workflow capabilities that preserve state across failures, support controlled resumption, provide complete execution history, and maintain alignment among technical validation, business approval, and production execution.
Create reusable workflow components for intake, validation, certification, release approval, change alignment, deployment, verification, rollback, evidence collection, exception handling, and closeout.
Define clear execution boundaries among orchestration services, CI/CD platforms, approval systems, source-of-truth platforms, AI advisory services, and automation execution engines.
Design and implement RESTful, event-driven, streaming, and standards-based integrations among enterprise platforms, network infrastructure, source-of-truth systems, workflow engines, observability services, artifact repositories, and change-management systems.
Define stable, versioned service contracts and data models that allow platform components to evolve independently while maintaining compatibility, security, and traceability.
Build integrations using technologies and protocols such as REST, RESTCONF, NETCONF, gRPC, webhooks, message queues, event streams, OpenAPI specifications, and structured data formats.
Develop data pipelines and services that collect, validate, normalize, correlate, and expose network state, software lifecycle data, workflow execution data, telemetry, release evidence, and operational outcomes.
Establish patterns for data quality, lineage, ownership, freshness, access control, retention, reconciliation, and authoritative-source designation.
Integrate network source-of-truth platforms such as Nautobot or NetBox with automation services, workflow orchestration, intended-state models, actual-state telemetry, compliance checks, and drift-detection processes.
Design, build, and operate containerized services using Docker, Kubernetes, OpenShift, Helm, and comparable cloud-native technologies.
Develop CI/CD and GitOps capabilities that automate build, testing, security validation, policy enforcement, artifact promotion, environment deployment, and release verification.
Establish engineering patterns for promoting software and configuration safely across development, test, UAT, and production environments.
Implement Infrastructure as Code and configuration automation using technologies such as Terraform, OpenTofu, Ansible, Helm, and Kubernetes manifests.
Define standards for source control, branching, pull requests, protected branches, release tags, artifact integrity, dependency controls, and environment-specific configuration.
Build automated test capabilities across unit, component, contract, integration, regression, performance, resiliency, and end-to-end testing.
Design highly available, fault-tolerant, scalable systems that can support mission-critical engineering and infrastructure workflows.
Establish service-level indicators, service-level objectives, error budgets, availability targets, capacity expectations, and production-readiness requirements for platform services.
Design observability architectures using metrics, logs, traces, events, flow data, streaming telemetry, and business-level workflow indicators.
Implement dashboards, alerts, service health indicators, dependency views, and diagnostic capabilities using technologies such as OpenTelemetry, Prometheus, Grafana, ELK, Splunk, and distributed tracing platforms.
Lead the development of automated detection, diagnostics, remediation, rollback, and evidence-capture patterns.
Apply failure-mode analysis, resilience testing, controlled fault injection, capacity testing, and performance engineering to identify risks before production adoption.
Ensure platform teams receive actionable health information rather than relying solely on infrastructure alerts or manual investigation.
Design and develop governed AI-assisted engineering services that improve knowledge discovery, software development, test generation, release analysis, operational diagnostics, evidence summarization, and engineering decision support.
Build AI application capabilities using retrieval-augmented generation, semantic search, structured outputs, tool integration, model evaluation, and human-in-the-loop approval patterns.
Develop knowledge services that use approved and authoritative engineering sources, including standards, documentation, software repositories, release evidence, test results, network state, telemetry, and operational history.
Establish clear boundaries between AI recommendation and production execution. AI-generated recommendations must remain explainable, traceable, reviewable, and subject to approved policy or human controls.
Define evaluation methods for source grounding, retrieval quality, response accuracy, unsupported claims, recommendation usefulness, and operational risk.
Partner with security, data, risk, legal, compliance, and AI governance teams to ensure AI-enabled capabilities meet enterprise requirements.
Embed security, resiliency, auditability, and policy enforcement into software architecture and platform design rather than treating them as post-development activities.
Design identity, authorization, secrets management, service account, certificate, data protection, and least-privilege patterns for platform services and integrations.
Establish guardrails that prevent unauthorized execution, bypass of required approvals, unverified artifact promotion, uncontrolled configuration changes, or use of untrusted data.
Ensure platform actions and workflow transitions produce sufficient evidence for troubleshooting, compliance, risk review, and audit.
Partner with Information Security and Risk organizations to translate policies and standards into enforceable software controls and measurable engineering requirements.
Identify technical risks proactively and lead the design and implementation of appropriate mitigation strategies.
Translate leadership vision, business priorities, and enterprise technology objectives into executable software strategies, architectural roadmaps, platform capabilities, and engineering initiatives.
Provide vision, direction, and technical expertise to leadership on major engineering investments, modernization opportunities, platform decisions, and emerging technologies.
Influence engineering direction across multiple teams and organizations without relying on direct management authority.
Define reusable architecture patterns and drive consistent implementation across software, infrastructure, network, reliability, and automation teams.
Mentor senior engineers, technical leads, architects, and emerging principal engineers through design collaboration, code review, technical forums, and structured knowledge sharing.
Maintain knowledge of industry engineering practices, software frameworks, cloud-native platforms, AI technologies, observability methods, and network automation developments.
Evaluate new technologies through prototypes and evidence-based assessments, then define adoption guidance, implementation patterns, and production-readiness criteria.
Promote a culture of engineering ownership, automation, measurement, experimentation, documentation, and continuous improvement.

Benefits

Wells Fargo is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other legally protected characteristic.
Employees support our focus on building strong customer relationships balanced with a strong risk mitigating and compliance-driven culture which firmly establishes those disciplines as critical to the success of our customers and company. They are accountable for execution of all applicable risk programs (Credit, Market, Financial Crimes, Operational, Regulatory Compliance), which includes effectively following and adhering to applicable Wells Fargo policies and procedures, appropriately fulfilling risk and compliance obligations, timely and effective escalation and remediation of issues, and making sound risk decisions. There is emphasis on proactive monitoring, governance, risk identification and escalation, as well as making sound risk decisions commensurate with the business unit’s risk appetite and all risk and compliance program requirements.
Accommodation for applicants with disabilities is available upon request in connection with the recruitment process.
To request a medical accommodation during the application or interview process, visit Disability Inclusion at Wells Fargo.
Wells Fargo maintains a drug free workplace. Please see our Drug and Alcohol Policy to learn more.

Stand Out From the Crowd

Upload your resume and get instant feedback on how well it matches this job.

Upload and Match Resume