89 papers
Scaling Laws
0/89Compute/data/parameter scaling, Chinchilla, and compute-optimal training.
Progress0 of 89
2026
5- AprTrain-to-Test (T2) Scaling Laws: Integrating Pretraining and Test-Time Computescaling-laws2604.01411University of Wisconsin-MadisonApr 1, 2026score 9~122 min
- MarBeyond Language Modeling: An Exploration of Multimodal Pretrainingmultimodal2603.03276AI at MetaMar 3, 2026score 9~119 min
- MardaVinci-LLM:Towards the Science of Pretrainingpretraining2603.27164SII-GAIRMar 28, 2026score 9~110 min
- MarHow Far Can Unsupervised RLVR Scale LLM Training?training-methods2603.08660Tsinghua UniversityMar 9, 2026~126 min
- FebPrincipled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendationllm-systems2602.07298AI at MetaFeb 7, 2026score 4~120 min
2025
25- DecTowards a Science of Scaling Agent Systemsagents2512.08296Dec 9, 2025~118 min
- DecDynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Spacearchitecture2512.24617ByteDance SeedDec 31, 2025score 9~113 min
- NovVirtual Width Networksarchitecture2511.11238ByteDance SeedNov 14, 2025score 9~112 min
- OctEvery Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundationmoe2510.22115inclusionAIOct 25, 2025score 10~127 min
- OctScaling Language-Centric Omnimodal Representation Learningmultimodal2510.11693DAMO AcademyOct 13, 2025score 8~128 min
- OctThe Art of Scaling Reinforcement Learning Compute for LLMsrl-training2510.13786Oct 15, 2025~139 min
- OctSimPoly: Simulation of Polymers with Machine Learning Force Fields Derived from First Principlesscaling-laws2510.13696Microsoft ResearchOct 15, 2025~127 min
- OctScaling Laws Meet Model Architecture: Toward Inference-Efficient LLMsscaling-laws2510.18245Oct 21, 2025~103 min
- SepPre-training under infinite computepretraining2509.14786Sep 18, 2025~106 min
- SepDA$^{2}$: Depth Anything in Any Directionscaling-laws2509.26618Tencent HunyuanSep 30, 2025score 9~100 min
- SepFantastic Pretraining Optimizers and Where to Find Themtraining-methods2509.02046Sep 2, 2025~121 min
- JulDecoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationarchitecture2507.06607Microsoft ResearchJul 9, 2025score 9~109 min
- JulFalcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancearchitecture2507.22448Jul 30, 2025score 10~132 min
- JulFast and Simplex: 2-Simplicial Attention in Tritonscaling-laws2507.02754Jul 3, 2025score 10~112 min
- JunLanguage Modeling by Language Modelsarchitecture2506.20249Jun 25, 2025~92 min
- JunTransformers Meet In-Context Learning: A Universal Approximation Theoryinference-optimization2506.05200Jun 5, 2025~121 min
- MayInsights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturesmoe2505.09343NVIDIAMay 14, 2025score 10~111 min
- MayParallel Scaling Law for Language Modelsscaling-laws2505.10475May 15, 2025~87 min
- MayCritical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingtraining-methods2505.23971May 29, 2025~111 min
- AprDataDecide: How to Predict Best Pretraining Data with Small Experimentsdata2504.11393Apr 15, 2025~133 min
- AprON LINEAR REPRESENTATIONS AND PRETRAINING DATA FREQUENCY IN LANGUAGE MODELSuncategorized2504.12459Apr 16, 2025~95 min
- MarHolistically Evaluating the Environmental Impact of Creating Language Modelsevaluation2503.05804Mar 3, 2025~121 min
- FebCan 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalinginference-optimization2502.06703Feb 10, 2025score 9~110 min
- FebScaling Pre-training to One Hundred Billion Data for Vision Language Modelsmultimodal2502.07617DeepMindFeb 11, 2025score 9~135 min
- JanAn Empirical Study of Autoregressive Pre-training from Videospretraining2501.05453Jan 9, 2025~102 min
2024
26- DecByte Latent Transformer: Patches Scale Better Than Tokensarchitecture2412.09871Dec 13, 2024score 9~136 min
- DecApollo: An Exploration of Video Understanding in Large Multimodal Modelsmultimodal2412.10360Dec 13, 2024~111 min
- DecDensing Law of LLMsscaling-laws2412.04315OpenBMBDec 5, 2024score 8~113 min
- NovSparsing Law: Towards Large Language Models with Greater Activation Sparsityagents2411.02335Nov 4, 2024score 9~108 min
- NovHunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentmoe2411.02265Tencent HunyuanNov 4, 2024score 9~110 min
- OctPre-training Distillation for Large Language Models: A Design Space Explorationpretraining2410.16215Oct 21, 2024score 10~124 min
- OctInference Scaling for Long-Context Retrieval Augmented Generationreasoning2410.04343Oct 6, 2024~124 min
- OctStuffed Mamba: Oversized States Lead to the Inability to Forgettraining-methods2410.07145Oct 9, 2024score 9~127 min
- SepQwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutionmultimodal2409.12191Qwen / Alibaba CloudSep 18, 2024score 9~112 min
- SepScaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationpretraining2409.12903AppleSep 19, 2024score 9~102 min
- AugScaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parametersinference-optimization2408.03314Aug 6, 2024score 10~122 min
- AugTransformers are Universal In-context Learnerspretraining2408.01367Aug 2, 2024~122 min
- JulScaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariespretraining2407.13623Jul 18, 2024score 9~122 min
- JulScaling Retrieval-Based Language Models with a Trillion-Token Datastoreretrieval2407.12854Jul 9, 2024~112 min
- JulLarge Language Monkeys: Scaling Inference Compute with Repeated Samplingscaling-laws2407.21787Jul 31, 2024~130 min
- JulGENERALIZATION V.S. MEMORIZATION: TRACING LANGUAGE MODELS’ CAPABILITIES BACK TO PRETRAINING DATAtraining-methods2407.14985Jul 20, 2024~131 min
- JunHow Do Large Language Models Acquire Factual Knowledge During Pretraining?pretraining2406.11813Jun 17, 2024score 9~107 min
- MayBiMix: A Bivariate Data Mixing Law for Language Model Pretrainingpretraining2405.14908May 23, 2024score 9~105 min
- AprPhi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonepretraining2404.14219Microsoft ResearchApr 22, 2024score 9~114 min
- AprMiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiestraining-methods2404.06395OpenBMBApr 9, 2024score 9~139 min
- MarYi: Open Foundation Models by 01.AIscaling-laws2403.0465201-AIMar 7, 2024score 9~73 min
- FebThe Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitspretraining2402.17764DropboxFeb 27, 2024~118 min
- FebNeural Networks Learn Statistics of Increasing Complexityscaling-laws2402.04362EleutherAIFeb 6, 2024~108 min
- JanInfini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokensdata2401.17377Jan 30, 2024score 10~114 min
- JanDeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelsmoe2401.06066DeepSeekJan 11, 2024score 10~107 min
- JanDeepSeek LLM: Scaling Open-Source Language Models with Longtermismscaling-laws2401.02954Jan 5, 2024~129 min
2023
12- DecBeyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawsscaling-laws2401.00448Dec 31, 2023score 9~99 min
- OctConvNets Match Vision Transformers at Scalevision2310.16764Oct 25, 2023score 9~119 min
- SepAn Empirical Study of Scaling Instruct-Tuned Large Multimodal Modelsmultimodal2309.09958Sep 18, 2023score 9~101 min
- SepWhen Less is More: Investigating Data Pruning for Pretraining LLMs at Scalepretraining2309.04564Sep 8, 2023score 9~110 min
- SepScaling Laws for Sparsely-Connected Foundation Modelsscaling-laws2309.08520Sep 15, 2023score 9~115 min
- SepEffective Long-Context Scaling of Foundation Modelstraining-methods2309.16039Sep 27, 2023score 9~104 min
- AugScaling Relationship on Learning Mathematical Reasoning with Large Language Modelsscaling-laws2308.01825Aug 3, 2023score 9~111 min
- AugAccurate Computation of Quantum Excited States with Neural Networksscaling-laws2308.16848DeepMind5 citesAug 31, 2023~132 min
- AugStudying Large Language Model Generalization with Influence Functionstraining-methods2308.03296Aug 7, 2023score 8~102 min
- JulRead Moretraining-methods2307.05695EleutherAIJul 11, 2023score 6~112 min
- MarGPT-4 Technical Reportpretraining2303.08774Mar 15, 2023~118 min
- FebLLaMA: Open and Efficient Foundation Language Modelspretraining2302.13971Stability AIFeb 27, 2023~120 min
2022
6- JunBIG-bench: Beyond the Imitation Game Benchmarkscaling-laws2206.04615MistralJun 9, 2022~99 min
- JunEmergent Abilities of Large Language Modelsscaling-laws2206.07682Jun 15, 2022~128 min
- MayOPT: Open Pre-trained Transformer Language Modelspretraining2205.01068Meta AI / FAIRMay 2, 2022~125 min
- AprOpen Source and In-House: How Uber Optimizes LLM Trainingpretraining2204.02311UberApr 5, 2022~108 min
- MarTraining Compute-Optimal Large Language Modelsscaling-laws2203.15556Mar 29, 2022~111 min
- JanGrokking: Generalization Beyond Overfitting on Small Algorithmic Datasetstraining-methods2201.02177Jan 6, 2022~125 min
2021
5- DecGLaM: Efficient Scaling of Language Models with Mixture-of-Expertsmoe2112.06905Dec 13, 2021~114 min
- DecGopherscaling-laws2112.11446Dec 8, 2021~142 min
- NovCombined Scaling for Zero-shot Transfer Learningscaling-laws2111.10050Nov 19, 2021~124 min
- NovScaling Law for Recommendation Models: Towards General-purpose User Representationsscaling-laws2111.11294Nov 15, 2021~105 min
- JanSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitymoe2101.03961361 citesJan 11, 2021~109 min
2020
3- OctScaling Laws for Autoregressive Generative Modelingscaling-laws2010.14701Oct 28, 2020~128 min
- MayLanguage Models are Few-Shot Learnerspretraining2005.14165Meta AI / FAIRMay 28, 2020~119 min
- Jan2028: Two scenarios for global AI leadershipscaling-laws2001.08361Anthropic1,503 citesJan 23, 2020~129 min
2019
22014
12000
4- —Nonlinear Dimensionality Reduction by Locally Linear Embeddingscaling-laws~104 min
- —Local structures of gravity-free space and time and their phenomenascaling-laws~75 min
- —Spanner: Google’s Globally-Distributed Databasescaling-laws~125 min
- —Translating Embeddings for Modeling Multi-relational Datascaling-laws~92 min