Data Engineer 4, Real-time Data Platform

Capital One • Plano, TX

7d

About The Position

Capital One is seeking a Data Engineer to help drive a major transformation by combining data with emerging technologies. The role leads the design and development of an enterprise-scale, low-latency, fault-tolerant streaming data platform and distributed streaming pipelines and analytics data stores. It uses streaming frameworks such as Flink and Spark Streaming, programming languages such as Java, Scala, and Python, and big data technologies including Spark, Flink, Kafka, Snowflake, AWS Big Data Services, and Redshift.

The Data Engineer leads a group of engineers building data pipelines on medium- to large-scale datasets, influences best practices for pipeline design and data architecture, and works with structured and unstructured data. The position operates in a collaborative Agile environment focused on CI/CD, application resiliency, and partnership with Cyber and Security teams. It also collaborates across Agile teams and with digital product managers to design, develop, test, implement, and support technical solutions, including cloud-based solutions. The role includes unit testing, code reviews, performance tuning, mentoring, and keeping current with technology trends.

The position is listed for Plano, Texas, with a full-time annual salary range of $179,400 to $204,700. It is eligible for performance-based incentive compensation, which may include cash bonuses and/or long-term incentives. Capital One describes its benefits as a comprehensive, competitive, and inclusive set of health, financial, and other benefits; eligibility varies by employment status and management level.

Requirements

• Bachelor's degree or higher in Computer Science or a related quantitative field, such as Statistics, Economics, Operations Research, Analytics, Mathematics, or Engineering.
• At least 4 years of application development experience; internship experience does not apply.
• At least 2 years of experience in distributed data.
• At least 2 years of SQL experience.
• At least 2 years of experience with at least one programming language: Python, Java, or Scala.
• At least 2 years of data pipeline design and development experience.
• At least 1 year of experience in data modeling and designing end-to-end data solutions using relational and non-relational database systems.

Nice To Haves

• 7 or more years of application development experience with demonstrated proficiency in Python, SQL, Scala, or Java.
• 4 or more years of hands-on experience designing, deploying, and operating data workloads in a public cloud environment such as AWS, Microsoft Azure, or Google Cloud.
• 4 or more years of experience building or supporting distributed data or compute workloads using tools such as EMR, Spark, Glue, or Databricks.
• 4 or more years of experience designing, implementing, and operating real-time or streaming data pipelines.
• 2 or more years of experience with data observability tools such as Monte Carlo or Splunk, or data orchestration tools such as Airflow or Dagster.
• 4 or more years of experience working with unstructured or semistructured data using NoSQL databases such as MongoDB, Cassandra, or DynamoDB.
• 4 or more years of experience designing and supporting data warehousing solutions such as Snowflake or Redshift.
• 2 or more years of experience working in an Agile development environment.
• 2 or more years of experience developing user-centric reusable data products.

Responsibilities

• Lead the design and build of an enterprise-scale, scalable, low-latency, fault-tolerant streaming data platform.
• Build distributed streaming data pipelines and analytics data stores using frameworks such as Flink and Spark Streaming and languages such as Java, Scala, and Python.
• Lead a group of engineers building data pipelines with big data technologies on medium- to large-scale datasets.
• Influence best practices for data pipeline design, data architecture, and processing structured and unstructured data.
• Work collaboratively in an Agile environment with a focus on CI/CD, application resiliency standards, and partnership with Cyber and Security teams.
• Stay current with technology trends, experiment with and learn new technologies, participate in technology communities, and mentor other engineers.
• Collaborate across Agile teams to design, develop, test, implement, and support technical solutions.
• Collaborate with developers experienced in machine learning, distributed microservices, and full-stack systems.
• Collaborate with digital product managers to deliver cloud-based solutions.
• Perform unit tests and conduct code reviews to ensure code is rigorously designed, well-coded, and tuned for performance.

Benefits

• Comprehensive health, financial, and other benefits supporting total well-being; eligibility varies by employment status and management level.
• Performance-based incentive compensation, potentially including cash bonuses and/or long-term incentives.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior