Lead Software Engineer

JPMorganChase•Plano, TX

About The Position

As a Lead Software Engineer at JPMorganChase within the Corporate Sector – Corporate Technology, you are an integral part of an agile team enhancing, building, and delivering trusted, secure, stable, and scalable technology products. As a core technical contributor, you conduct critical technology solutions across multiple technical areas and business functions in support of the firm’s business objectives. The role leads the architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark, and establishes Lakehouse patterns with Delta Lake. It also drives responsible use of enterprise-authorized AI-assisted engineering practices, with consistent validation standards for correctness, performance, security, and reliability. The position owns Databricks cluster strategy, pipeline orchestration, secure ingestion and transformation frameworks, data quality and governance, performance engineering, reusable software libraries, and CI/CD for data projects.

Requirements

Formal training or certification in software engineering concepts and at least 5 years of applied experience.
Advanced software engineering and data engineering experience, including significant production delivery with Apache Spark on Databricks and/or AWS EMR.
Experience leading effective use of approved AI-assisted software development tools for coding, code review, test acceleration, and troubleshooting, and setting expectations for validating AI outputs for correctness, performance, and security.
Understanding of responsible AI use in engineering workflows, including data sensitivity, secure handling of inputs and outputs, and security and resiliency expectations; experience coaching engineers on safe, compliant adoption.
Advanced hands-on Databricks expertise across Delta Lake, Unity Catalog, Workflows, Repos/notebooks, and SQL Warehouses, including cluster configuration and optimization.
Ability to architect, build, and operate reliable batch and streaming ETL/ELT pipelines, including schema design and evolution, SLAs, and reliability engineering practices.
Deep Spark performance-tuning skills, including diagnosing bottlenecks and optimizing jobs for scalability, cost, and runtime efficiency.
Strong Python programming proficiency for data processing, platform tooling, and automation.
Strong SQL and analytics data-modeling expertise, including dimensional/star schema design and Lakehouse best practices.
Security-first engineering mindset covering data sensitivity, secure input/output handling, roles and instance profiles, secrets management, encryption at rest and in transit, network controls, and resiliency and security expectations.

Nice To Haves

Experience with Delta Live Tables and advanced Databricks governance, including catalogs, grants, and auditing.
AWS networking knowledge, including VPCs, subnets, routing, security groups, and data egress controls.
Experience with Terraform for infrastructure deployments.
Cost optimization experience, including autoscaling, spot versus on-demand instances, auto-termination, storage layouts, and compaction.
Familiarity with Airflow, Genie, Streamlit, and React.
Experience with data-system observability, including freshness and completeness metrics, lineage, SLAs, and alerting.
Demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship, with strong stakeholder communication.

Responsibilities

Lead the architecture and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark, driving performance, reliability, and scalability.
Establish and evolve Delta Lake Lakehouse patterns, including ACID transactions, schema evolution, time travel, Z-ordering, and compaction.
Drive team adoption of enterprise-authorized AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes, while establishing validation standards and promoting reuse.
Apply software development life-cycle toolchain capabilities, including authorized AI-assisted development and automation, to improve the value realized by automation.
Own Databricks cluster strategy and configuration, including runtime selection, autoscaling, driver and executor sizing, Spark configurations, init scripts, cluster policies, pools, and instance profiles.
Orchestrate and automate pipelines and jobs using Databricks Workflows, integrating AWS eventing and orchestration services as needed.
Design secure ingestion and transformation frameworks using Databricks services, Delta or unmanaged table designs, ingestion tasks, and Airflow DAGs to produce trusted and refined datasets.
Enforce data quality, lineage, and governance using Unity Catalog and/or AWS Glue Catalog, embedding expectations and validation in pipelines.
Drive Spark and Databricks performance engineering and tuning, including partitioning, file sizing, AQE, joins, shuffle, caching, memory and spill control, job sizing, and clustering or partitioning keys.
Build and maintain reusable libraries, frameworks, and APIs in Python and/or Java, with strong unit, integration, and data validation test coverage.
Implement CI/CD for data projects using Git-based workflows, Terraform-based infrastructure deployments and environment promotion, and automated releases.
Champion engineering standards, code reviews, testing, and AI-assisted engineering practices with consistent validation and reuse of proven patterns.