About the position
Independent consultants only. The role of Data Engineer - AI at MSRcosmos involves working remotely from the USA. The position is contract-based and requires a strong proficiency in programming languages such as Python, Scala, or similar. The candidate will be expected to have a solid understanding of machine learning frameworks like TensorFlow and PyTorch, along with strong experience in data classification, particularly in identifying PII data entities. Knowledge and experience with retrieval-augmented generation (RAG) and agent-based workflows are essential. The role also requires a deep understanding of how to re-rank and improve LLM outputs using Index and Vector stores. Additionally, the candidate should be able to leverage AWS services (e.g., SageMaker, Comprehend, Entity Resolution) to solve complex data and AI-related challenges, manage and deploy machine learning models and frameworks at scale using AWS infrastructure, and possess strong analytical and problem-solving skills to innovate and develop new approaches to data engineering and AI/ML. Experience with AWS ETL services (such as AWS Glue, Lambda, and Data Pipeline) for effective data processing and integration tasks is also required, along with experience in core AWS Services including AWS IAM, VPC, EC2, S3, RDS, Lambda, CloudWatch, and CloudTrail.
Requirements
- Proficiency in programming languages such as Python, Scala, or similar.
- Solid understanding of machine learning frameworks such as TensorFlow and PyTorch.
- Strong experience in data classification, including the identification of PII data entities.
- Knowledge and experience with retrieval-augmented generation (RAG) and agent-based workflows.
- Deep understanding of how-to re-rank and improve LLM outputs using Index and Vector stores.
- Ability to leverage AWS services (e.g., SageMaker, Comprehend, Entity Resolution) to solve complex data and AI-related challenges.
- Ability to manage and deploy machine learning models and frameworks at scale using AWS infrastructure.
- Strong analytical and problem-solving skills, with the ability to innovate and develop new approaches to data engineering and AI/ML.
- Experience with AWS ETL services (such as AWS Glue, Lambda, and Data Pipeline) to handle data processing and integration tasks effectively.
- Experience in core AWS Services including AWS IAM, VPC, EC2, S3, RDS, Lambda, CloudWatch, CloudTrail.
Nice-to-haves
- Experience with data privacy and compliance requirements, especially related to PII data.
- Familiarity with advanced data indexing techniques, vector databases, and other technologies that improve the quality of AI/ML outputs.
