Data Engineer 4, Real-time Data Platform

Capital One • Plano, TX

8d • $179,400 - $204,700

About The Position

Capital One is seeking a Data Engineer to help build and lead a scalable, low-latency, fault-tolerant real-time streaming data platform. The role involves developing distributed streaming data pipelines and analytics data stores using frameworks such as Flink and Spark Streaming, programming languages such as Java, Scala, and Python, and technologies including Kafka, Snowflake, Redshift, and AWS big data services.

The engineer will guide a group of engineers, influence data pipeline and architecture practices, and work with structured and unstructured data. The role collaborates across Agile teams and with product managers, security partners, and engineers working in machine learning, distributed microservices, and full-stack systems. It also includes designing, developing, testing, implementing, and supporting cloud-based solutions; performing unit tests and code reviews; and mentoring engineers while keeping current with technology trends.

Requirements

• Bachelor's degree or higher in Computer Science or a related quantitative field, such as Statistics, Economics, Operations Research, Analytics, Mathematics, or Engineering.
• At least 4 years of application development experience; internship experience does not apply.
• At least 2 years of experience in distributed data.
• At least 2 years of experience with SQL.
• At least 2 years of experience with Python, Java, or Scala.
• At least 2 years of experience in data pipeline design and development.
• At least 1 year of experience in data modeling and designing end-to-end data solutions using relational and non-relational database systems.

Nice To Haves

• 7+ years of application development experience with demonstrated proficiency in Python, SQL, Scala, or Java.
• 4+ years of hands-on experience designing, deploying, and operating data workloads in a public cloud environment such as AWS, Microsoft Azure, or Google Cloud.
• 4+ years of experience building or supporting distributed data or compute workloads using tools such as EMR, Spark, Glue, or Databricks.
• 4+ years of experience designing, implementing, and operating real-time or streaming data pipelines.
• 2+ years of experience with data observability tools such as Monte Carlo or Splunk, or data orchestration tools such as Airflow or Dagster.
• 4+ years of experience working with unstructured or semistructured data using NoSQL databases such as MongoDB, Cassandra, or DynamoDB.
• 4+ years of experience designing and supporting data warehousing solutions such as Snowflake or Redshift.
• 2+ years of experience working in an Agile development environment.
• 2+ years of experience developing user-centric reusable data products.

Responsibilities

• Lead the design and build of an enterprise-level scalable, low-latency, fault-tolerant streaming data platform.
• Build distributed streaming data pipelines and analytics data stores using frameworks such as Flink and Spark Streaming and languages such as Java, Scala, and Python.
• Lead engineers building data pipelines with technologies including Spark, Flink, Kafka, Snowflake, AWS big data services, and Redshift.
• Influence best practices for data pipeline design, data architecture, and processing structured and unstructured data.
• Work in a collaborative Agile environment focused on CI/CD, application resiliency standards, and partnerships with Cyber and Security teams.
• Keep current with technology trends, experiment with and learn new technologies, participate in technology communities, and mentor engineering colleagues.
• Collaborate across Agile teams to design, develop, test, implement, and support technical solutions.
• Collaborate with developers experienced in machine learning, distributed microservices, and full-stack systems.
• Work with digital product managers to deliver cloud-based solutions.
• Perform unit tests and review code with team members to ensure quality and performance.

Benefits

• Comprehensive health, financial, and other benefits supporting employee well-being.
• Eligibility for performance-based incentive compensation, which may include cash bonuses and/or long-term incentives.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior