Data movement bottlenecks to large-scale model training: Scaling past 1e28 FLOP

FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI

Notable AI Models

Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data

Will we run out of ML data? Evidence from projecting dataset size trends (2022)