Role overview

Scale AI is seeking a Machine Learning Research Scientist to join the evaluation pod within the GenAI Research Organization. The role focuses on building benchmarks and diagnostic methods for frontier language models and multimodal systems. Scale partners with leading AI labs to provide high-quality data and accelerate progress in generative AI research.

Key responsibilities

Analyze the behavior of frontier large language models and AI agents to identify, characterize, and diagnose failure modes across capability gaps, reasoning errors, robustness issues, and alignment concerns.
Design and construct benchmarks and evaluation methodologies that assess LLM capabilities in both text and multimodal domains.
Apply post-training expertise in supervised fine-tuning, reinforcement learning from human feedback, and reward modeling to connect observed model failures to specific data and training interventions.
Collaborate with researchers and engineers to establish best practices for evaluation-driven AI development.
Partner with foundation model research labs to translate failure analysis into technical and strategic guidance for next-generation generative AI systems.
Publish research findings at top-tier AI conferences.

Requirements

Ph.D. or Master’s degree in Computer Science, Machine Learning, AI, or a closely related discipline.
Deep expertise in deep learning, reinforcement learning, and large-scale model fine-tuning techniques.
Demonstrated experience with post-training methods including RLHF, preference modeling, or instruction tuning, as well as LLM evaluation or benchmark development.
Published research in machine learning at major conferences such as NeurIPS, ICML, ICLR, ACL, EMNLP, or CVPR, or in leading journals.
Strong written and verbal communication skills.
Prior experience in a customer-facing role.

What makes this role worth considering

This position offers the opportunity to work on foundational challenges in AI evaluation and safety at a company partnering with industry leaders including Meta, Mayo Clinic, and U.S. government agencies. You will directly influence how the next generation of AI systems are developed and improved through rigorous diagnostic research.