Lead AI Engineer with Databricks, Agentic AI

We are seeking a Lead AI Engineer with expertise in Databricks and Databricks Agentic AI processes to join our team.

The successful candidate will play a crucial role in delivering agentic solutions within our data platform for the PLM use cases. This position requires a seasoned professional with a strong background in AI engineering and a track record of leveraging Databricks for building agents.

Two AI Engineers are required to design, build and operate the core intelligence layer of the Product Lifecycle Management (PLM) solution. Working within an approved AI pattern on Databricks (Azure), you will be fully embedded within the PLM team to develop a deep understanding of the domain, processes and data before building the end-to-end pipeline from raw supplier document ingestion through to structured AI-generated recommendations surfaced for human review. You will also build and maintain the Databricks Apps front-end (React/Python) that exposes this capability to approximately 30 business users.

Responsibilities

• Embed fully within the PLM team from day one to understand supplier specification workflows, test method structures and properties management before writing a line of code
• Design and implement the document ingestion pipeline using Databricks ai_parse_document for PDF, Excel and Word supplier specification files, storing artifacts in Unity Catalog Volumes
• Build and maintain Agent Bricks (Information Extraction) pipelines to extract structured fields from supplier documents against the canonical PLM data model
• Develop and optimize LLM-powered grouping and recommendation logic using ai_query and Databricks Model Serving, ensuring outputs are explainable and auditable
• Create the Databricks Apps front-end (React + Python backend): file upload, metadata grids, dropdown-driven forms, human-in-the-loop review/approval screens and write-back to Lakebase
• Integrate the Graph Database layer to store and traverse relationships between specifications, properties, test methods and AI recommendations
• Configure and maintain Azure Private Link / Private Endpoint connectivity, including DNS forwarding for databricksapps.com, to ensure all traffic stays within the corporate network perimeter
• Implement structured logging, lineage tracking in Unity Catalog, and audit trails to support Responsible AI governance requirements
• Contribute to ARB artifacts and technical design documentation as required
• Collaborate daily with the Business Analyst to translate PLM domain requirements into working data models and pipeline logic, and with the Delivery Manager to surface risks and effort estimates

Requirements

• 5+ years of experience in AI/data engineering roles
• At least 1 year of relevant leadership experience
• Expertise in Databricks (Unity Catalog, Volumes, Model Serving)
• Proficiency in ai_parse_document and Agent Bricks
• Skills in Python (PySpark, pandas, FastAPI)
• Background in React front-end development (hooks, REST integration)
• Competency in Azure (Private Link, Private Endpoints, VNet)
• Knowledge of LLM prompt engineering and ai_query
• Familiarity with Graph Databases (Neo4j or equivalent)
• Understanding of Delta Lake and Lakebase
• Capability to design and integrate REST APIs
• Experience with Azure DevOps (ADO)
• Proficiency in English at an Upper-Intermediate level (B2) or higher

Nice to have

• Experience with GxP or regulated manufacturing data environments
• Exposure to PLM or ERP systems (SAP, Teamcenter, Enovia)
• Databricks certification (Data Engineer Professional or ML Professional)
• Background in graph data modeling and lineage use cases

Benefits

We connect like-minded people

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

We cover it all

• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Comprehensive medical & family care package
• Five trust days per year (sick leave without a medical certificate)
• Benefits package (sports activities, a variety of stores and services)