Azalia Mirhoseini

Anthropic and Stanford University

Title: Pushing the Limits of Scaling Laws in the Age of Large Language Models

Abstract

The recent success of large language models has been characterized by scaling laws – the power law relationship between performance and training dataset size, model parameter size, and training compute. In this talk, we will discuss ways to push the scaling laws even further by innovating across data, models, software and hardware. This includes reinforcement learning from human and AI feedback to improve learning efficiency, sparse and dynamic mixture-of-experts neural architectures for better performance, an automated framework for co-designing custom AI accelerators, and a deep RL method for chip floorplanning used in multiple generations of Google AI’s accelerator chips (TPU). Through these cutting-edge examples, we will outline a full-stack approach that leverages AI to overcome the next set of scaling challenges.

---

Bio

Azalia Mirhoseini is an assistant professor of computer science at Stanford University. Her research interest is developing capable and efficient AI systems that can solve high-impact, real-world problems. Before joining Stanford, Prof. Mirhoseini spent several years in industry, working on frontier generative AI and deep reinforcement learning projects at Anthropic and Google Brain. She has led a diverse portfolio of AI and Systems projects, with publications in Nature, ICML, ICLR, NeurIPS, UAI, ASPLOS, SIGMETRICS, DAC, DATE, and ICCAD. She has received a number of awards, including the MIT Technology Review 35 Under 35, the Best Ph.D. Thesis at Rice University's ECE Department, and a Gold Medal in the National Math Olympiad in Iran. Her work has been covered in various media outlets, including MIT Technology Review, IEEE Spectrum, The Verge, Times of London, ZDNet, VentureBeat, and WIRED.