Senior Data Engineer

Bank of America•Charlotte, NC

8d•Onsite

About The Position

This job is responsible for driving data engineering efforts to deliver enterprise-wide capabilities and complex data solutions. Key responsibilities include directing code design and delivery tasks associated with the integration, cleaning, transformation, and control of data in operational and analytical data systems and working with the Project Management team to define outcomes and inform work structures. Job expectations include providing technical thought leadership by implementing complex data solutions and interactions across multiple systems and domains. Overview: Lead requirement analysis, high-level and low-level design, development, and implementation of enterprise-scale data platforms and integration solutions leveraging Spark/Scala, Kafka, Hadoop, Starburst, Cloud technologies, Object Storage, and Informatica. Lead developer experience with ability to conduct design reviews, mentor / coach development teams, and produce comprehensive yet flexible solution designs. Ensures that the design and engineering approach for complex features are consistent with the larger portfolio solution Key experience developing and delivering complex software requirements to accomplish business goals Capability to ensure software is developed to meet functional, non-functional, and compliance requirements Successful background ensuring solutions can be integrated successfully into the overall application/system with clear, robust and well-tested interfaces Should be able to lead multiple projects concurrently each with competing deadlines. Should be knowledgeable in Treasury Payments and Deposits domains. Serves as a fully seasoned/proficient technical resource; provides technical knowledge and capabilities as team member and individual contributor. Typically 10 or more years of data delivery/integration experience Works under minimal supervision Has the knowledge of Hadoop Spark/Spark Streaming best practices and standards as well as standard software development lifecycle experience. Must have an understanding of our ability to quickly grasp technology concepts Solid quantitative, analytical, process development, facilitation skills, and organizational skills required Strong communication, presentation, interpersonal, software development and work management skills are essential Coordinate with deployment and support teams and ensure projects are successfully deployed into testing and Production environments. Development experience using Agile process Experience in designing, developing and deploying Real time and Near real time & batch data integration solutions Work with people in different time zones & teams

Requirements

12+ years of experience with Data Engineering
Experience with large-scale analytical platforms, metadata-driven architectures, and distributed data processing frameworks is preferred.
Design and implement cloud-native data platforms utilizing Object Storage, Data Lakehouse architecture, distributed query engines, and streaming technologies to support real-time, near real-time, and batch processing requirements.
Hands on experience in Scala, Spark, Kafka Streaming
Very good understanding of Hadoop components, architecture & tool sets: Spark and Scala, Informatica, Starburst, Object Storage in enterprise data platforms
Must have Architected, developed & implemented solutions using real time streaming technologies
Good knowledge of performance tuning
Experience working with Exadata & Oracle SQL Tuning
Experience working with Starburst and Object Storage
Experience developing Streaming/Near Real time and Batch data integration solutions.
Hands on experience in UNIX Shell Scripting
Working experience with scheduling tools

Nice To Haves

CICD Build & Deployment tools
API development experience
Financial Services / Banking experience
Experience in distributed computing
Experience with open table formats such as Apache Iceberg, Delta Lake, or equivalent technologies
Experience with enterprise metadata management, data lineage, governance, and data quality frameworks

Responsibilities

Assembles large, complex data sets that meet functional and non-functional requirements, ensuring that the design and engineering approach is consistent across multiple systems
Maintains, improves, cleans, and manipulates large data for operational and analytics data systems, builds complex processes supporting data transformation, data structures, metadata, data quality controls, dependency, and workload management, and communicates required information for deployment, maintenance, and support of business functionality
Utilizes multiple architectural components in the design and development of client requirements and collaborates with development teams to understand data requirements and ensure the data architecture is feasible to implement
Defines and builds data pipelines to enable data-informed decision making, ensuring adherence to release processes and risk management routines
Contributes to existing test suites including integration, regression, and performance, analyzes test reports, identifies any test issues and errors, and leads triage of underlying causes
Leads the identification of gaps in data management standards adherence and works with appropriate partners to develop plans to close gaps, leading concept testing and conducting research to prototype toolsets and improve existing processes
Mentors Data Engineers in the delivery and release of continuous integration and continuous delivery events and defines key performance indicators and internal controls