Lead Data Engineer - Finance Technology (Hadoop, PySpark, Scala/Java)
About The Position
The Lead Data Engineer role within Finance Technology Solutions at Target is a technical leadership position focused on designing and developing scalable, high-performance data solutions. This role is crucial for evolving the company's data architecture to meet business requirements for reliability, efficiency, and scalability. The engineer will leverage expertise in big data technologies, distributed systems, and cloud platforms to shape the engineering roadmap and establish best practices for data processing, analytics, and real-time data services. Key responsibilities include architecting and optimizing data pipelines using Hadoop, Spark, Scala/Java, and cloud technologies to support enterprise-wide data initiatives, as well as developing APIs and data services for low-latency data access.
Requirements
- BS or MS in Computer Science, Machine Learning, Artificial Intelligence, Applied Mathematics or a related technical field preferred
- 7plus years of experience in data engineering, software development, or distributed systems
- Expertise in Big Data technologies such as Hadoop, Spark, and distributed processing frameworks
- Strong programming skills in PySpark, Scala, and/or Java
- Experience with cloud platforms (AWS, GCP, or Azure) and associated data services (e.g. S3, BigQuery, Databricks, EMR, Snowflake)
- Proficiency in API development using REST, GraphQL, or gRPC to support real-time and batch data access
- Experience with real-time and streaming data architectures like Kafka, Flink or Kinesis
- Strong knowledge of data modeling, ETL pipeline design and performance optimization
- Understanding of data governance, security, and compliance in large-scale data environments
- Strong problem-solving skills and the ability to thrive in complex, evolving environments
- Excellent communication and collaboration skills, with experience working across cross-functional teams
Responsibilities
- Architect and build scalable, high-performance data pipelines and distributed data processing solutions using Spark, Scala/Java and cloud platforms (AWS, GCP, or Azure)
- Design and implement real-time and batch data processing solutions, ensuring data is efficiently processed and readily available for analytical and operational use
- Develop APIs and data services that provide low-latency, high-throughput access to data for downstream applications, enabling real-time decision-making
- Optimize and enhance data models, workflows, and processing frameworks to improve performance, scalability, and cost efficiency
- Drive data governance, security, and compliance best practices
- Collaborate with product teams and business stakeholders to understand requirements and deliver data-driven solutions
- Lead the design, implementation, and lifecycle management of data services and platforms
- Stay current with emerging technologies and promote the adoption of best practices in big data engineering, cloud computing, and API development
- Provide technical leadership and mentor engineers to foster engineering excellence and best practices
Benefits
- Comprehensive health benefits and programs (medical, vision, dental, life insurance)
- 401(k)
- Employee discount
- Short term disability
- Long term disability
- Paid sick leave
- Paid national holidays
- Paid vacation
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
Associate degree
