Senior Data Engineering Support Engineer
About The Position
NTT DATA is seeking a Senior Data Engineering Support Engineer to serve as the primary technical anchor for enterprise big data operations. The role focuses on high availability, performance efficiency, and cost optimization across production data pipelines in a Databricks and PySpark Delta Lakehouse environment. The engineer resolves complex production incidents, leads root-cause analysis, develops automated self-healing solutions, and mentors junior support staff. The position is listed as remote, with the starting pay range of $92,160.00 to $138,240.00 for U.S.-based positions. Actual compensation depends on work location, experience, technical skills, and other qualifications; the position may also be eligible for incentive compensation.
Requirements
- At least 7 years of total experience in Data Engineering.
- At least 3 years supporting or optimizing large-scale Databricks and PySpark systems in production.
Nice To Haves
- Advanced PySpark and Python engineering skills and explicit understanding of Spark architecture, including lazy evaluation, DAG execution, and execution versus storage memory.
- High proficiency in advanced SQL for complex analytical troubleshooting.
- Expert understanding of Delta Lake internals, including ACID transactions, Time Travel, Liquid Clustering, and Optimize/Z-Order commands.
- A proven track record diagnosing and resolving large-scale distributed data pipeline issues, including thread dumps, memory leaks, and serialization errors.
- Strong architectural experience deploying and managing resources in AWS, Azure, or GCP.
- Robust understanding of Git-based workflows, automated testing, and secure hotfix deployment through CI/CD pipelines.
- Active Databricks certification, such as Databricks Certified Data Engineer Professional.
- Deep experience managing production environments using Delta Live Tables (DLT) and advanced Unity Catalog rule engines.
Responsibilities
- Act as the highest tier of internal escalation for complex, high-priority failures in large-scale batch and streaming PySpark jobs.
- Diagnose and resolve distributed computing bottlenecks, including data skew, out-of-memory errors, improper partition pruning, and costly shuffle operations, using Spark UI, logs, and Ganglia metrics.
- Analyze Databricks cluster utilization and reduce cloud compute costs through strategies such as autoscaling, spot instances, and selecting suitable driver and executor node types.
- Oversee and troubleshoot data access control, schema evolution, and data lineage within Databricks Unity Catalog.
- Build automated Python/PySpark frameworks and scripts for data reconciliation, system health checks, and self-healing pipelines.
- Write technical runbooks and post-mortem root-cause analysis documents, and establish standard operating procedures to reduce incident resolution time.
- Provide technical guidance to junior support engineers, conduct code reviews for hotfixes, and collaborate with development teams to connect engineering and operations.
Benefits
- Medical insurance with an employer contribution
- Dental insurance with an employer contribution
- Vision insurance with an employer contribution
- Flexible spending or health savings account
- Life and AD&D insurance
- Short- and long-term disability coverage
- Paid time off
- Employee assistance
- 401(k) program with company match
- Additional voluntary or legally required benefits
- Potential eligibility for incentive compensation based on individual and/or company performance
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
