Data Engineer II - Getting Customers Ready for AI

Microsoft•Redmond, WA

1d•$102,100 - $219,200

About The Position

Microsoft Security's Getting Customers Ready for AI team is seeking a Data Engineer II to help build the data foundations that power AI-native security systems. In this role, you will design and develop scalable data platforms, pipelines, and telemetry systems that transform raw signals across identity, devices, data, applications, and infrastructure into high-quality, trusted datasets for analytics and AI-driven insights. You will work in a collaborative engineering environment to deliver reliable, secure, and performant data systems, enabling downstream AI models, security detections, and customer-facing experiences. This role is ideal for engineers looking to deepen their expertise in large-scale data systems, distributed processing, and modern data platform architectures for AI and Agents. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Requirements

Bachelor's Degree in Computer Science or related technical field AND 2+ years of software, data, or related engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
Ability to meet Microsoft, customer, and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years.

Nice To Haves

Hands-on experience building or maintaining data pipelines and distributed data systems.
Programming experience in Python, Scala, or SQL for data processing and pipeline development.
Understanding of data modeling, ETL processes, and large-scale data processing concepts.
Experience with big data frameworks (e.g., Spark, Flink) or streaming systems (e.g., Kafka/Event Hub).
Familiarity with cloud-based data platforms (Azure preferred, e.g., ADLS, Synapse, Databricks).
Exposure to data governance, security, and compliance practices in enterprise environments.
Experience supporting AI/ML workloads through data pipelines, feature stores, or training datasets.
Developed problem-solving skills with the ability to operate in ambiguity and deliver incrementally.

Responsibilities

Design and build scalable data pipelines (batch and streaming) to process large volumes of security and operational data.
Develop and optimize ETL/ELT workflows that transform raw telemetry into structured, consumable datasets.
Implement data ingestion frameworks to integrate multi-source data from services, APIs, and event streams.
Improve pipeline performance, reliability, and efficiency through partitioning, indexing, and optimization techniques.
Design and evolve data models, schemas, and storage strategies for analytics and AI use cases.
Work with distributed storage systems (e.g., data lakes, warehouses) to ensure scalability and cost efficiency.
Maintain data partitioning, retention, and lifecycle strategies aligned to business and compliance needs.
Ensure data is structured for downstream ML pipelines, feature engineering, and analytics workloads.
Implement data validation, quality checks, and observability frameworks to ensure data accuracy and reliability.
Apply best practices for data governance, lineage, and auditing across pipelines and datasets.
Ensure compliance with security, privacy, and regulatory requirements when handling sensitive data.
Contribute to standardization of data contracts and schemas across services.
Enable high-quality datasets for AI/ML teams, including support for feature pipelines and training data preparation.
Collaborate with AI engineers to ensure data is optimized for RAG pipelines, model training, and evaluation workflows.
Build and maintain telemetry pipelines and metrics systems that provide insights into AI system performance and usage.
Support development of data-driven signals and insights that improve customer security posture.
Partner with engineering, data science, and product teams to deliver end-to-end data solutions.
Contribute to system design discussions, architecture reviews, and cross-team integration efforts.
Work across teams to ensure consistent data definitions and interoperability across platforms.
Write clean, maintainable, and well-tested code for data pipelines and platform services.
Build monitoring and alerting mechanisms for pipeline health, latency, and data quality issues.
Document data architecture, pipeline designs, and operational practices for knowledge sharing and scalability.
Follow best practices for performance optimization, cost management, and reliability engineering.
Embody our culture and values.