Lead AI Engineer || Immediate Joiner

Enable Dataorporated

About The Position

This role involves significant work in solution and data architecture, focusing on migrating authentication methods, configuring Azure resources, and preparing technical documentation. A key part of the role is designing and building data ingestion pipelines into Databricks using PySpark and PySQL, with a strong emphasis on testing and stakeholder collaboration. The position requires a deep understanding of Databricks Unity Catalog, OBO tokens, Service Principals, and Databricks Apps. Additionally, the role involves developing minimal Single Page Applications (SPAs) with AI integration using FastAPI/React and Claude Models/APIs, including prompt engineering and understanding token economics. Experience with AI safety, caching mechanisms, Claude Code, Genie, and full-stack development is essential. The role also includes DevOps responsibilities such as creating and automating CI/CD pipelines for various deployments using ADO, incorporating security scans, and configuring environments. Strong communication skills are required to collaborate with toxicologists and project managers, delivering work as an individual contributor in short sprint cycles.

Requirements

Core technical skill will be PySpark and PySQL
Understanding of the Databricks Unity Catalog permissions model and how to use OBO tokens and configure Service Principals for Databricks Agents,Genie Spaces, Machine Learning and Foundation Models.
Knowledge of Databricks Apps.
Ability to build and Deploy Databricks Apps using Visual Studio Code.
Sending custom notifications from Databricks using APIs to custom Team Channel Webhooks
Ability to use FastAPI /React front end and build minimal Single Page Applications(SPAs) using Chat UI interfaces and Claude Models/APIs
Ability to use defined System Prompts, User Prompts etc..(i..e,) skills in prompt engineering.
Basic Knowledge of Token Economics and ability use rule enforcement in prompts in addition to standard NLP language.
Ability to reengineer code and see what it does and map it to the requirement specifications from users
Ability to use Graphs to detect inter dependencies between rules and circular references.
Ability to use both Claude Vision API in addition to standard text processing.
Understand and develop Retrieval Augmented Generation specifically to Databricks like AI Search indexes, Hierarchical Chunking etc..
Know how to implement basic guard rails related to AI safety
Understand various caching mechanisms
Ability to work with Claude Code and Genie.
Ability to understand how a full stack works frontend, backend, middle ware data layers, routes etc.. and troubleshoot issues if any.
Correct and rewrite code if any related to security findings in addition to standard development.
Ability to Document code through various techniques like TDD,FDD,DFD and SD’s
Create and Automate deployment pipelines(CI/CD) for Teams Bot Deployment ,Web Application Deployments, Databricks Bundle/notebook Deployments through ADO.
Add various validation steps like SnykScans, Linting.
Configuring various environments based on CI/CD pipelines.
Ability to work with short descriptions of requirements and collaborating closely with Toxicoloigists, Project Managers from client team.
Ability to deliver work as an Individual Contributor in short sprint cycles.
Only immediate Joiners

Responsibilities

Solution Architecture/Data Architecture
Change source code to accommodate for various architectural considerations e.g.: Moving MSAL based authentication to JWT based authentication to Databricks X-forwarded headers while using Apps Sit
Technical reviews for configuring necessary Azure resources for deploying the application.
Prepare Technical/Network Architecture Documentation and send it reviews and approvals for cloud and security teams before going live.
Design and build pipelines for ingesting data into Databricks from various sources like SAP, website scraping.
Write and do scenario-based testing, edge cases and discuss with the stake holders to finalize the necessary code changes and acceptance criteria.
Understanding of the Databricks Unity Catalog permissions model and how to use OBO tokens and configure Service Principals for Databricks Agents,Genie Spaces, Machine Learning and Foundation Models.
Knowledge of Databricks Apps.
Build and Deploy Databricks Apps using Visual Studio Code.
Send custom notifications from Databricks using APIs to custom Team Channel Webhooks
Use FastAPI /React front end and build minimal Single Page Applications(SPAs) using Chat UI interfaces and Claude Models/APIs
Use defined System Prompts, User Prompts etc..(i..e,) skills in prompt engineering.
Use rule enforcement in prompts in addition to standard NLP language.
Reengineer code and see what it does and map it to the requirement specifications from users
Use Graphs to detect inter dependencies between rules and circular references.
Use both Claude Vision API in addition to standard text processing.
Understand and develop Retrieval Augmented Generation specifically to Databricks like AI Search indexes, Hierarchical Chunking etc..
Implement basic guard rails related to AI safety
Understand various caching mechanisms
Understand how a full stack works frontend, backend, middle ware data layers, routes etc.. and troubleshoot issues if any.
Correct and rewrite code if any related to security findings in addition to standard development.
Document code through various techniques like TDD,FDD,DFD and SD’s
Create and Automate deployment pipelines(CI/CD) for Teams Bot Deployment ,Web Application Deployments, Databricks Bundle/notebook Deployments through ADO.
Add various validation steps like SnykScans, Linting.
Configuring various environments based on CI/CD pipelines.
Collaborate closely with Toxicologists, Project Managers from client team.
Deliver work as an Individual Contributor in short sprint cycles.