GenAI Platform Engineering Lead

M&T Bank•Buffalo, NY

4d•$116,400 - $194,000•Onsite

About The Position

Manages the activities of Engineering Team Leaders, Engineering Supervisors and/or engineering units responsible for the reliability, observability and operational readiness of the Bank’s AI platform. Provides day-to-day direction for the teams and applications in alignment with departmental goals and the needs of the clients they support. Serves as the technical lead for AI operations and service assurance, with accountability for the operational layer of the AI platform. Works closely with the Platform Engineering Manager, who owns the broader platform, to ensure production systems are secure, resilient, observable, cost-effective and operationally ready. Oversees incident response practices, runbooks, monitoring, infrastructure automation and production support. Responsible for managing client relationships and expectations, prioritizing the project queue and achieving individual and organizational objectives at minimum cost.

Requirements

A combined minimum of 9 years’ higher education and/or work experience, including a minimum of 4 years’ engineering and/or architecture experience and 3 years leadership experience
Minimum of 5 years’ experience in site reliability engineering, platform operations, infrastructure engineering or a related discipline, including experience supporting production systems through an on-call model
Hands-on experience building, operating and troubleshooting Azure infrastructure and services, including Azure API Management, Azure Monitor, Log Analytics, Microsoft Entra ID and Terraform
Experience building and supporting observability pipelines using OpenTelemetry, Prometheus, Grafana, Azure-native technologies and/or comparable tools
Strong understanding of metrics, traces and logs, including when and how each should be used to monitor and troubleshoot production systems
Experience implementing continuous integration and continuous delivery practices for infrastructure using Terraform, GitHub Actions, GitLab CI, infrastructure-as-code patterns and/or comparable technologies
Strong scripting and automation skills using Python, Bash, PowerShell and/or comparable languages
Experience leading or supporting production incident response, including troubleshooting, escalation, root-cause analysis and service restoration
Capable of working on multiple projects of a complex nature
Proficiency with project management, word processing and spreadsheet applications
Complete understanding of the system development life cycle
Excellent problem-solving skills to assist in issue resolution
Familiarity with application development software, cloud infrastructure and hardware platforms
Excellent verbal and written communication skills
Excellent analytical and decision-making skills
Strong project management and presentation skills
Experience encouraging teamwork and serving as a role model when leading and directing others
Understanding of the technical, business, operational, risk and cost impacts of a project, platform or production issue

Nice To Haves

Bachelor’s degree
Minimum of 10 years’ technology management, site reliability engineering, platform operations or large program leadership experience
Experience operating AI or large language model platforms in a production environment
Experience with large language model operational concerns, including token cost tracking, model latency monitoring, provider failover and rate limiting by caller identity
Experience integrating application infrastructure with Security Information and Event Management platforms and security event feeds
Familiarity with AI gateway patterns, including prompt logging, personally identifiable information interception and infrastructure-layer content filtering
Experience working in a regulated environment where audit trails, access controls and change management practices are required
Extensive application and product knowledge within AI platforms, cloud infrastructure, site reliability engineering, observability and/or service assurance
Subject matter expert understanding of supported applications, with advanced knowledge of interfacing and integrated applications
Understanding of multiple business areas and their functions
Proven mentoring and leadership capabilities
Experience with the technologies, applications and functions of the area being led
Good understanding of the Bank’s application framework
Awareness of the Bank’s business plan and strategic objectives, with the ability to help shape direction
Self-motivated with the ability to motivate and develop others
Understanding of the supported businesses and their terminology

Responsibilities

Lead the reliability, observability and operational readiness of the AI platform, including production monitoring, incident response, service assurance, infrastructure automation and operational controls.
Establish and maintain operational runbooks, escalation procedures, service-level indicators, service-level objectives and incident response practices.
Measure operational performance through platform availability, incident response time, mean time to recovery, model latency, token consumption and infrastructure cost.
Provide leadership during production incidents. Coordinate troubleshooting, communication, escalation, root-cause analysis and corrective actions.
Build and maintain observability pipelines using OpenTelemetry, Prometheus, Grafana, Azure Monitor, Log Analytics and/or comparable technologies.
Oversee the deployment and operation of Azure infrastructure and services, including Azure API Management, Azure Monitor, Log Analytics, Microsoft Entra ID and infrastructure managed through Terraform.
Promote infrastructure-as-code and automated deployment practices using Terraform, GitHub Actions, GitLab CI and/or comparable tools.
Ensure platform changes are deployed through controlled pipelines rather than manual processes.
Drive automation of recurring operational activities using Python, Bash, PowerShell and other appropriate scripting technologies.
Oversee AI-specific operational capabilities, which may include token cost tracking, model latency monitoring, provider failover, caller-level rate limiting, prompt logging, personally identifiable information interception and infrastructure-layer content filtering.
Partner with cybersecurity and risk teams to support Security Information and Event Management integrations and security event feeds from application infrastructure.
Manage and participate in consultations with client management to analyze short-range business requirements and recommend innovations that anticipate the future impact of changing business and technology needs.
Build and maintain positive client relationships.
Monitor technology direction, industry trends and vendor applications related to AI platforms, site reliability engineering, cloud infrastructure, observability and service assurance.
Research and initiate changes to existing processes, technologies and operating models when necessary.
Lead vendor and product analysis and provide recommendations.
Oversee application development support, testing efforts, technology infrastructure, project management and other assigned technology domains.
Serve as a subject matter expert for AI platform operations, production reliability, observability and service assurance.
Build rapport across the organization and maintain a professional level of communication and cooperation with technology, business, risk, cybersecurity and vendor partners.
Maintain relationships with vendors and professional organizations.
Direct team activities, assign personnel to projects and provide technical and operational guidance.
Ensure schedules and commitments are completed.
Lead short-term staffing and capacity planning.
Implement technology consistent with Division standards and long-range plans.
Ensure adherence to Department and Technology standards and procedures, including documentation, audit trail and change management requirements.
Translate business and operational requirements as needed to assist staff in preparing detailed specifications for system enhancements.
Evaluate and manage recommended designs based on business, operational and technology requirements.
Identify, communicate and resolve issues and concerns.
Manage project plans and coordinate major project and production-readiness activities.
Remain current on work outside the team that may affect the team, platform or client environment.
Develop and manage multiple cost center budgets, including cloud infrastructure and AI platform operating costs.
Recommend and implement policies and procedures that improve the performance, reliability and effectiveness of the Department.
Exercise the usual authority of a manager concerning staffing, performance appraisals, promotions, salary recommendations, performance management and terminations.
Understand and adhere to the Company’s risk and regulatory standards, policies and controls in accordance with the Company’s Risk Appetite.
Design, implement, maintain and enhance internal controls to mitigate risk on an ongoing basis.
Identify risk-related issues requiring escalation to management.
Promote an environment that supports belonging and reflects the M&T Bank brand.
Maintain M&T internal control standards, including the timely implementation of internal and external audit points and resolution of issues raised by external regulators, as applicable.
Complete other related duties as assigned.