Senior Engineer – AI Monitoring & Control Capabilities
About The Position
This is a critical-delivery, hands-on individual contributor role responsible for engineering and operating vendor applications end to end in a fully on-premises OpenShift environment. The position supports enterprise AI monitoring, controls, and governance commitments by developing and integrating application components, deploying and scaling containerized services, and providing production support to keep services reliable, performant, and audit-ready. The role focuses on AI Agent operations. Supported applications provide guardrails and observability for enterprise LLM-powered systems, including model observability, traceability, and evaluation needed to meet risk management, compliance, and regulatory expectations. The engineer is expected to learn tools and environments quickly, apply a pragmatic build-and-operate mindset, and lead cross-functional initiatives without formal authority. The position also includes mentoring engineers, establishing development standards, improving delivery processes, and contributing to financial planning and budget management.
Requirements
- Bachelor's or Master's degree in Computer Science, Data Science, MIS, a related field, or equivalent practical experience.
- 10+ years of software engineering experience designing, building, and scaling production applications.
- Strong proficiency in Python, including common data and ML libraries.
- Hands-on experience deploying and operating containerized workloads on OpenShift or Kubernetes.
- Proven experience taking LLM- or RAG-based applications into production and maintaining them at scale.
- Experience building APIs and integrating third-party platforms.
- Working experience with AI Agent development, including OpenAI APIs and LangChain.
- Practical understanding of inference performance optimization.
- Experience applying guardrails, observability, and model-based evaluation to AI applications.
- Experience delivering within Agile and Agile-at-scale environments against firm timelines.
- Proven ability to influence outcomes and meet delivery commitments in a matrixed organization without direct authority.
- Strong problem-solving, communication, and stakeholder management skills.
Nice To Haves
- Experience delivering technology solutions within a large, complex, highly regulated enterprise, preferably financial services.
- Ability to ramp quickly on new technologies and tools.
- Experience with annotation pipelines, feedback loops, fine-tuning, and model alignment techniques.
- Familiarity with prompt versioning and lifecycle management.
- Experience with vendor-provided guardrail and observability platforms such as Galileo, LangSmith, Arize, WhyLabs, Braintrust, Agenta, Fiddler, MLflow, Weights & Biases, LangFuse, and RAGAS.
- Working knowledge of additional AI Agent frameworks such as LlamaIndex or Haystack.
- Familiarity with evaluation and guardrails for agentic, multi-step AI workflows.
- Collaborative and self-directed work style supporting engineering teams.
Responsibilities
- Own delivery of critical vendor applications from build through production support in a fully on-premises OpenShift environment, meeting committed milestones and quality expectations.
- Build and maintain APIs and integrations connecting vendor guardrail and observability platforms to enterprise AI Agent solutions.
- Establish and manage workflows that monitor performance, safety, latency, throughput, quality, and guardrail effectiveness in support of AI risk and control objectives.
- Implement and maintain model evaluation frameworks using LLM-as-a-Judge and specialized language models.
- Develop observability dashboards and alerting using Grafana or similar tools, producing audit-friendly evidence and traceability.
- Troubleshoot production issues and improve scalability, resilience, and operational efficiency to sustain availability for critical business services.
- Contribute to evaluation methodologies and guardrail controls for agentic and multi-step AI systems.
- Lead and mentor engineers, establish development standards, and drive process improvements across cross-functional delivery.
- Provide input into financial planning, budget tracking, forecasting, and management of supported applications and services.
Benefits
- Annual discretionary incentive plan eligibility.
- Benefits eligibility.
- Paid time off.
- Resources and support for employees.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
