Principal Engineer Automation

Wells Fargo•Irving, TX

About The Position

The Principal Engineer Automation serves as a senior technical advisor and engineering leader, shaping architecture and development of enterprise software, platforms, APIs, network automation services, information security capabilities, data systems, operating environments, and cloud-native technologies. The role leads resolution of complex engineering challenges and translates business priorities into secure, resilient, maintainable, large-scale solutions. The engineer designs and develops production-grade platforms, microservices, APIs, SDKs, libraries, automation frameworks, and reusable components. They establish engineering standards and reference architectures, lead technical and production-readiness reviews, and contribute hands-on to complex platform work. The role treats internal platforms as products, creating self-service capabilities, standardized workflows, developer enablement, and reusable delivery patterns. Responsibilities span durable workflow orchestration; API, integration, and data engineering; cloud-native software and CI/CD; reliability and observability; governed AI-assisted engineering; and security, risk, and governance. The role also provides technical direction across teams without relying on direct management authority, mentors engineers and technical leaders, evaluates emerging technologies, and promotes engineering ownership and continuous improvement. Success is measured by wider adoption of reusable platform capabilities, less duplicated work and operational toil, improved software quality and reliability, faster and safer delivery, stronger alignment of systems and infrastructure state, controlled adoption of AI-assisted engineering, and effective technical influence across the organization.

Requirements

At least 7 years of engineering experience, or equivalent demonstrated through work experience, training, military experience, or education.
At least 7 years of software engineering, platform engineering, cloud engineering, infrastructure engineering, network automation, SRE, NRE, or equivalent experience.
At least 5 years designing, developing, testing, and supporting production-grade software, distributed systems, automation platforms, or enterprise integration services.
At least 3 years of advanced software development using Python and at least one additional modern language such as Go, Java, C#, .NET, or TypeScript.
At least 3 years designing and developing APIs, microservices, asynchronous services, event-driven systems, workflow services, or comparable distributed components.
At least 3 years developing software with Git-based workflows, pull requests, peer review, versioning, package management, CI/CD, automated testing, and release management.
At least 3 years deploying or supporting containerized applications using Docker, Kubernetes, OpenShift, or comparable orchestration platforms.
At least 3 years integrating multiple systems through REST APIs, webhooks, messaging, events, SDKs, or service interfaces.
At least 3 years designing for scalability, availability, resiliency, observability, security, and operational support.

Nice To Haves

Strong knowledge of software engineering fundamentals, including data structures, algorithms, modular design, object-oriented or functional design, design patterns, concurrency, error handling, automated testing, and performance optimization.
Experience leading technical design or architecture across multiple engineering teams, platforms, or organizational boundaries.
Strong analytical and systems-thinking skills, including decomposing ambiguous enterprise-scale problems into executable solutions.
Strong written, verbal, and visual communication skills for explaining complex software and architecture decisions to engineers, technology leaders, and executives.
Experience as a Principal Engineer, Staff Engineer, Senior Staff Engineer, Software Architect, Platform Architect, or in a comparable technical leadership role.
Experience leading a large-scale software platform and driving strategy across teams without direct management authority.
Experience creating reference architectures, reusable frameworks, engineering standards, or platform services adopted by multiple teams.
Experience leading architecture, software design, code-quality, production-readiness reviews, or cross-organizational technical programs.
Ability to balance hands-on engineering with strategic influence, mentoring, governance, and long-term technical planning.
Deep understanding of distributed systems, including consistency, availability, partition tolerance, concurrency, state, recovery, idempotency, backpressure, retries, timeouts, and compensating transactions.
Experience designing high-volume, highly available, or mission-critical services.
Experience with event-driven architecture, message brokers, streaming platforms, service discovery, API gateways, caching, distributed data systems, and asynchronous processing.
Experience defining versioned APIs, service contracts, domain models, and backward-compatible integration strategies.
Experience with performance engineering, load testing, capacity analysis, profiling, and optimization.
Experience building internal developer platforms, self-service portals, platform-as-a-product capabilities, engineering enablement services, or reusable delivery frameworks.
Experience creating paved roads, golden paths, templates, pipelines, service catalogs, SDKs, or automated onboarding capabilities.
Experience measuring platform adoption, developer productivity, delivery lead time, reliability, support demand, or operational toil.
Experience with Backstage or comparable developer portal or service catalog technologies.
Hands-on experience with Temporal, Celery, Argo Workflows, Apache Airflow, Dagster, or another durable workflow or asynchronous execution framework.
Understanding of workflow activities, workers, task queues, signals, queries, timers, retries, versioning, replay behavior, and durable state.
Experience designing idempotent, fault-tolerant, long-running workflows with human approvals and external dependencies.
Experience separating orchestration logic from execution logic, policy decisions, and user-facing intake.
Experience with Docker, Kubernetes, OpenShift, Helm, Harness, GitHub Actions, Azure DevOps, Argo CD, or comparable build and deployment technologies.
Experience with Terraform, OpenTofu, Ansible, Pulumi, Kubernetes operators, or comparable Infrastructure as Code technologies.
Experience operating services in Azure, AWS, Google Cloud, private cloud, or hybrid-cloud environments.
Understanding of cloud well-architected practices, identity and access management, secrets, networking, scalability, resiliency, and cost-conscious engineering.
Experience with service meshes, ingress, API management, cloud-native networking, or policy-as-code.
Experience developing software for network or infrastructure automation, configuration management, software lifecycle management, or infrastructure orchestration.
Knowledge of enterprise networking, including routing, switching, wireless, firewalls, load balancing, DNS, IP address management, or network security.
Experience with RESTCONF, NETCONF, gNMI, gRPC, YANG, SNMP, streaming telemetry, or vendor APIs.
Experience with Nautobot, NetBox, or another source-of-truth and network state-management platform.
Experience with Ansible, Nornir, pyATS, Terraform, or comparable infrastructure and network automation frameworks.
Familiarity with Cisco, Juniper, Arista, Palo Alto, Fortinet, F5, or comparable enterprise infrastructure platforms.
Experience implementing observability with OpenTelemetry, Prometheus, Grafana, ELK, Splunk, distributed tracing, structured logging, streaming telemetry, or flow analytics.
Experience establishing SLIs, SLOs, error budgets, health indicators, alerting standards, or production-readiness criteria.
Familiarity with incident management, root-cause analysis, automated remediation, resilience testing, chaos engineering, capacity planning, and operational toil reduction.
Experience correlating application behavior, infrastructure state, network telemetry, workflow execution, and business outcomes.
Experience building production AI, machine learning, generative AI, or AI-assisted engineering solutions.
Experience with retrieval-augmented generation, embeddings, vector search, semantic retrieval, knowledge graphs, structured model outputs, tool calling, or prompt orchestration.
Experience developing AI evaluation frameworks for groundedness, source coverage, retrieval quality, unsupported claims, classification performance, and human acceptance.
Understanding of responsible AI, model governance, explainability, prompt and model versioning, access controls, sensitive data protection, and human review.
Experience building data pipelines and working with structured and unstructured data, including JSON, YAML, logs, telemetry, documents, API payloads, and software repositories.
Experience with information security and technology risk management, including secure development, security architecture, threat modeling, policies, standards, assessments, mitigation, and controls.
Experience incorporating identity, authorization, least privilege, secrets management, data protection, auditability, and policy enforcement into software platforms.
Financial services experience or experience in another highly regulated industry.
Bachelor’s or advanced degree in Computer Science, Software Engineering, Computer Engineering, Information Systems, Data Science, or a related technical field.
Relevant certifications such as CCIE, CCDE, JNCIE, Microsoft Azure Solutions Architect Expert, AWS Certified Solutions Architect Professional, Google Professional Cloud Architect, Certified Kubernetes Administrator, Certified Kubernetes Application Developer, or comparable certifications.

Responsibilities

Advise senior leadership and influence architecture and development across software, platform, network automation, security, data, operating environment, and cloud-native domains.
Lead the strategy and resolution of complex engineering challenges and deliver secure, resilient, maintainable, large-scale solutions.
Design and develop production-grade platforms, APIs, microservices, SDKs, libraries, automation frameworks, and reusable components.
Define distributed, event-driven, and asynchronous system architectures, including service boundaries, contracts, state, integration, failure handling, and recovery models.
Establish software engineering standards for architecture, code quality, testing, security, dependency management, releases, observability, and production readiness.
Develop reference implementations and contribute directly to technically complex platform capabilities.
Lead technical design, architecture, code, failure-mode, and production-readiness reviews.
Build internal engineering platforms with self-service automation, standardized workflows, reusable services, governed execution, and consistent developer experiences.
Manage shared platforms as products, with defined users, contracts, roadmaps, adoption measures, documentation, support, and reliability objectives.
Create paved roads and golden paths for developing, testing, certifying, releasing, and operating automation using approved patterns.
Improve developer productivity through APIs, templates, SDKs, CI/CD pipelines, test harnesses, local environments, documentation, and onboarding automation.
Reduce duplicated effort and operational toil through reusable platform services, libraries, modules, and integration patterns.
Define ownership boundaries among platforms, applications, production execution, operational support, and risk decisions.
Design durable workflows for long-running, failure-prone, approval-dependent infrastructure and network processes.
Define workflow patterns for activities, workers, queues, events, signals, timers, retries, timeouts, compensating actions, versioning, idempotency, replay safety, and human approvals.
Build workflows that preserve state through failures, support controlled resumption, retain execution history, and align validation, approval, and production execution.
Create reusable workflow components for intake, validation, certification, approvals, deployment, verification, rollback, evidence, exception handling, and closeout.
Define execution boundaries among orchestration services, CI/CD, approval systems, source-of-truth platforms, AI advisory services, and automation engines.
Design and implement RESTful, event-driven, streaming, and standards-based integrations among enterprise platforms, infrastructure, workflow, observability, artifact, and change-management systems.
Define stable, versioned service contracts and data models that support independent evolution, compatibility, security, and traceability.
Build integrations using REST, RESTCONF, NETCONF, gRPC, webhooks, message queues, event streams, OpenAPI, and structured data formats.
Develop data pipelines and services to collect, validate, normalize, correlate, and expose network state, lifecycle data, workflow execution, telemetry, release evidence, and operational outcomes.
Establish data quality, lineage, ownership, freshness, access, retention, reconciliation, and authoritative-source patterns.
Integrate network source-of-truth platforms with automation, workflows, intended state, telemetry, compliance checks, and drift detection.
Design, build, and operate containerized services using Docker, Kubernetes, OpenShift, Helm, or comparable technologies.
Develop CI/CD and GitOps capabilities for builds, testing, security validation, policy enforcement, artifact promotion, deployment, and release verification.
Establish safe software and configuration promotion patterns across development, test, UAT, and production.
Implement Infrastructure as Code and configuration automation using Terraform, OpenTofu, Ansible, Helm, and Kubernetes manifests.
Define standards for source control, branching, reviews, protected branches, release tags, artifact integrity, dependencies, and environment-specific configuration.
Build automated unit, component, contract, integration, regression, performance, resiliency, and end-to-end testing capabilities.
Design highly available, fault-tolerant, scalable systems for mission-critical engineering and infrastructure workflows.
Establish service indicators, objectives, error budgets, availability targets, capacity expectations, and production-readiness requirements.
Design observability using metrics, logs, traces, events, flow data, streaming telemetry, and workflow indicators.
Implement dashboards, alerts, health indicators, dependency views, and diagnostics using observability and tracing technologies.
Lead development of automated detection, diagnostics, remediation, rollback, and evidence-capture patterns.
Apply failure-mode analysis, resilience testing, fault injection, capacity testing, and performance engineering to identify production risks.
Ensure platform teams receive actionable health information rather than relying only on infrastructure alerts or manual investigation.
Design governed AI-assisted engineering services for knowledge discovery, development, test generation, release analysis, diagnostics, evidence summarization, and decision support.
Build AI capabilities using retrieval-augmented generation, semantic search, structured outputs, tool integration, evaluation, and human approval.
Develop knowledge services using approved, authoritative engineering sources.
Keep AI recommendations explainable, traceable, reviewable, and subject to policy or human controls; separate recommendations from production execution.
Define evaluation methods for grounding, retrieval quality, accuracy, unsupported claims, usefulness, and operational risk.
Partner with security, data, risk, legal, compliance, and AI governance teams on enterprise requirements.
Embed security, resiliency, auditability, and policy enforcement into software architecture and platform design.
Design identity, authorization, secrets, service account, certificate, data protection, and least-privilege patterns.
Establish guardrails against unauthorized execution, approval bypass, unverified artifact promotion, uncontrolled changes, and untrusted data.
Ensure platform actions and workflow transitions produce evidence for troubleshooting, compliance, risk review, and audit.
Partner with Information Security and Risk organizations to turn policies into enforceable controls and measurable engineering requirements.
Identify technical risks proactively and lead mitigation design and implementation.
Translate leadership vision and business priorities into software strategies, architectural roadmaps, platform capabilities, and engineering initiatives.
Advise leadership on engineering investments, modernization, platform decisions, and emerging technologies.
Influence engineering direction across teams without direct management authority.
Define reusable architecture patterns and drive consistent implementation across software, infrastructure, network, reliability, and automation teams.
Mentor senior engineers, technical leads, architects, and emerging principal engineers through design collaboration, reviews, forums, and knowledge sharing.
Evaluate new technologies through prototypes and evidence-based assessments and define adoption and production-readiness guidance.
Promote engineering ownership, automation, measurement, experimentation, documentation, and continuous improvement.