Principal Backend Engineer (Forward Deployed)

Invisible Technologies•San Francisco, CA
•$250,000 - $300,000•Hybrid

About The Position

Invisible is hiring a Principal Forward Deployed Engineer to lead high-impact AI-powered solutions for clients’ critical workflows. The role combines hands-on AI engineering, software development, and technical consulting. The engineer will shape strategy, architecture, and execution for end-to-end systems, including production LLM-powered and agentic applications, while working directly with clients and internal teams. The position involves turning ambiguous workflows and messy data into repeatable systems, leading projects from discovery through deployment and post-launch iteration, and making architecture trade-offs across performance, scalability, cost, and reliability. The engineer will also build shared infrastructure and reusable components, establish internal playbooks and MLOps practices, mentor engineers, and help shape Invisible’s forward-deployed AI engineering culture. This is a hybrid role. Onsite expectations depend on team placement; some NYC-based teams require three days per week in the office. The role may require travel for onsite client engagements and being on call for customers when situations arise.

Requirements

  • 10+ years of software engineering experience, including significant experience building data, ML, or backend systems.
  • Deep proficiency in Python and hands-on experience with Hugging Face, LangChain, OpenAI, Pinecone, and related LLM and AI agent ecosystems.
  • Experience with full-stack and API-based deployment patterns, including Docker, FastAPI, Kubernetes, and cloud environments such as GCP, AWS, or Azure.
  • Experience with workflow orchestration libraries, pub/sub systems such as Kafka, and schema governance.
  • Expertise in data governance and operations, including Unity Catalog and policy management, cluster and job orchestration, data contracts and quality enforcement, Delta/ETL pipelines, and replay processes.
  • Strong product and system design instincts, with the ability to translate business needs into technical architecture.
  • Experience building usable systems from messy data and ambiguous requirements.
  • Excellent communication and client-facing skills, including leading conversations with technical and non-technical stakeholders.
  • Proven experience owning projects from scoping through deployment in ambiguous, high-stakes environments.
  • A bachelor's degree in Data Science, Computer Science, or a related field, or equivalent professional experience.
  • Willingness to be on call for customers when situations arise.
  • Willingness to travel for onsite client engagements as needed.
  • Availability to work from the office a few days per week, depending on team needs; some NYC-based teams require three days per week onsite.

Nice To Haves

  • For teams with an applied AI/ML focus, experience with MLOps frameworks and best practices, including model versioning, monitoring, and lifecycle management.
  • Experience designing scalable ML systems such as RAG pipelines.

Responsibilities

  • Partner with delivery and executive stakeholders to scope, design, and lead implementation of AI-driven solutions, including LLM and agentic systems.
  • Identify transformational opportunities in ambiguous workflows and turn them into repeatable systems.
  • Lead architecture design and trade-off discussions involving performance, scalability, cost, and reliability, including for RAG pipelines and production AI applications.
  • Own projects from discovery through deployment, including client-facing delivery, internal coordination, and post-launch iteration.
  • Build shared infrastructure, reusable components, and internal playbooks to improve the team.
  • Develop MLOps practices for model versioning, monitoring, and lifecycle management.
  • Coach and mentor mid-level engineers and help shape the culture of forward-deployed AI engineering at Invisible.

Benefits

  • Bonus and equity are included in all full-time, non-commission offers.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service