Senior Data Engineer
About The Position
The Senior Data Engineer designs and builds governed data foundations, pipelines, APIs, and a Databricks semantic layer to make enterprise data trustworthy, AI-ready, and accessible to non-technical colleagues. The role combines hands-on Databricks and PySpark engineering with partnership across Finance and IT to advance a modern, agile target-state data architecture. The position develops AI-ready data pipelines and governed access foundations for AI agents, and establishes quality, lineage, observability, evaluation, and security practices. It also designs and maintains a governed Databricks Unity Catalog semantic layer, configures Genie spaces for natural-language querying, and validates semantic outputs with business owners. The role is based on-site in Westmont, Illinois. The position does not provide visa sponsorship; candidates must be authorized to work in the United States without sponsorship.
Requirements
- Hands-on Databricks experience with Delta Lake, Unity Catalog, Spark SQL, PySpark, and Workflows.
- Strong SQL and Python skills.
- Demonstrated experience building a semantic layer in Databricks, such as Unity Catalog views, metric views, and/or Genie configuration.
- Experience building batch and streaming pipelines and lakehouse/medallion data models.
- Experience developing REST APIs using FastAPI or Flask, with OAuth-based authentication and authorization.
- Working knowledge of AI data engineering concepts, including embeddings, vector search, RAG, LLM agents, context engineering, and AI evaluation.
- Experience with data governance, lineage, quality, and security in an enterprise environment.
- Experience with cloud platforms and CI/CD.
- Must be authorized to work in the United States without visa sponsorship.
Nice To Haves
- Azure cloud platform experience is preferred.
- Experience with Genie or similar text-to-SQL tools, including accuracy benchmarking.
- Experience with MCP or agent frameworks for enterprise data access.
Responsibilities
- Contribute to information infrastructure and data management processes that advance an agile, robust, AI-ready target-state architecture.
- Build systems and pipelines to ingest, cleanse, normalize, and structure internal and external datasets for analytical and AI use.
- Partner with colleagues and external professionals to determine requirements, anticipate future needs, and identify data development opportunities.
- Develop an understanding of data flows across CRM, Broker & Sales, Finance, HR, MDM, ODS, Data Lake, and EDW.
- Deliver data and API solutions, including natural-language and AI-assisted access to governed data, to support data-driven decisions by non-technical staff.
- Develop proofs of concept to validate proposals and influence platform architects, product managers, and software engineers.
- Design and develop data lake and API solutions for structured and unstructured data, and guide colleagues migrating to modern platforms.
- Build AI-ready pipelines that curate and version data for retrieval, evaluation, and agentic workflows, including document parsing, chunking, embeddings, and vector search for RAG.
- Develop API and MCP-based foundations that give AI agents governed access to enterprise data.
- Implement data quality, lineage, observability, and evaluation frameworks to assess AI-generated answers against trusted data.
- Apply governance and access controls, including row- and column-level security and masking, to respect data classification and privacy requirements.
- Partner with data science and business teams to move AI prototypes into monitored, production-grade solutions.
- Design, build, and maintain a governed semantic layer in Unity Catalog, including curated views, certified metric definitions, and centralized business logic.
- Configure and tune Databricks Genie spaces, including instructions, sample queries, entity matching, and naming conventions, for accurate natural-language querying.
- Establish semantic-layer standards, documentation, and versioning, and validate outputs against source-of-truth systems with Finance, HR, and other business owners.
Benefits
- 401(k) plan with matching company contributions
- Comprehensive medical, dental, and vision care
- Paid parental leave at 100% of salary
- Paid time off and company holidays
- Early access to earned wages through Daily Pay
- Supportive culture and comprehensive benefits package prioritizing mental, physical, and emotional health
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
