Senior Software Engineer
About The Position
The Data and Insight team in Microsoft’s Search Foundation organization processes petabyte-scale Bing and MSN logs using open-source technologies and provides insights into experiments. The team also supports the grounding business with real-time signals and insights, and protects Bing Search through offline bot-protection analysis and model training. In this role, you will lead the design and development of scalable data analytics and insights platforms, handling petabyte-scale data processing and real-time insights. You will build and own data-processing architectures, core datasets, and fault-tolerant pipelines using Azure and open-source technologies. The work includes ensuring data quality, detecting anomalies, optimizing performance and storage, meeting SLAs, and understanding data compliance and governance. You will collaborate with data scientists, product managers, and other partners to create end-to-end data assets that improve products, and leverage AI across coding, testing, data processing, and validation.
Requirements
- Bachelor's degree in Computer Science or a related technical field and 4+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
- At least 6 years of experience in one or more coding languages, such as C#, C++, or Java.
- At least 6 years of backend service development experience.
- At least 2 years of batch or streaming data-processing experience at petabyte scale.
- Domain expertise in Spark or Flink for processing petabyte-scale data daily.
- Experience optimizing data-pipeline performance.
- Fluent verbal and written English communication.
Nice To Haves
- Familiarity with data-processing technology such as Spark or Flink and messaging technology such as Kafka or EventHub.
- Experience with a cloud environment such as Azure.
- Experience with data storage technology such as HDFS or Azure Blob.
- Experience leveraging AI in coding or testing.
- Experience using AI in data processing and data validation.
- Data compliance or governance experience.
Responsibilities
- Design, build, and own data-processing architectures and systems for data and analytics pipelines, dashboards, and metrics using Azure and open-source technologies.
- Build core datasets and scalable, fault-tolerant pipelines.
- Implement data anomaly detection and data quality checks.
- Optimize pipelines for compute and storage while ensuring high-quality datasets and SLA compliance.
- Understand and account for data compliance and governance.
- Collaborate with data scientists, product managers, and cross-functional partners to design and develop end-to-end data assets that improve products.
- Work hands-on with data-processing technologies such as Spark or Flink and messaging technologies such as Kafka or EventHub.
- Work hands-on with data storage technologies such as HDFS or Azure Blob.
- Leverage AI technology in coding, testing, data processing, and data validation.
Benefits
- The posting states that certain roles may be eligible for benefits and other compensation and links to additional benefits and pay information.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
