322 papers
Pretraining
0/319Base model pretraining recipes, scaling laws, and corpora.
Progress0 of 319
2026
33- MaySlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingarchitecture2605.08738May 9, 2026~106 min
- MayEMO: Pretraining Mixture of Experts for Emergent Modularitymoe2605.06663Allen Institute for AIMay 7, 2026~100 min
- MayLet ViT Speak: Generative Language-Image Pre-trainingvision2605.00809ByteDanceMay 1, 2026~95 min
- AprEXAONE 4.5 Technical Reportmultimodal2604.08644LG EXAONEApr 9, 2026score 9~90 min
- AprAudio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musicmultimodal2604.10905NVIDIAApr 13, 2026score 9~106 min
- AprImage Generators are Generalist Vision Learnersmultimodal2604.20329DeepMindApr 22, 2026~135 min
- AprTrain-to-Test (T2) Scaling Laws: Integrating Pretraining and Test-Time Computescaling-laws2604.01411University of Wisconsin-MadisonApr 1, 2026score 9~122 min
- AprPre-train Space Reinforcement Learning: From P(y|x) to P(y)training-methods2604.14142Chinese Academic of Science Institute of AutomationApr 15, 2026score 9~101 min
- MarSommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelsdata2603.25750KAIST AIMar 20, 2026score 6~111 min
- MarTimer-S1: A Billion-Scale Time Series Foundation Model with Serial Scalingllm-systems2603.04791ByteDanceMar 5, 2026score 3~108 min
- MarBeyond Language Modeling: An Exploration of Multimodal Pretrainingmultimodal2603.03276AI at MetaMar 3, 2026score 9~119 min
- MarUnderstanding by Reconstruction: Reversing the Software Development Process for LLM Pretrainingpretraining2603.11103ByteDance SeedMar 11, 2026score 9~111 min
- MarV-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningpretraining2603.14482Meta LlamaMar 15, 2026score 7~123 min
- MardaVinci-LLM:Towards the Science of Pretrainingpretraining2603.27164SII-GAIRMar 28, 2026score 9~110 min
- MarSAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editingvision2603.19228BAIDUMar 19, 2026score 3~105 min
- FebUniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Modelarchitecture2602.14178ByteDanceFeb 15, 2026score 9~131 min
- FebOPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationdata2602.05400QwenFeb 5, 2026score 7~121 min
- FebData Science and Technology Towards AGI Part I: Tiered Data Managementdata2602.09003OpenBMBFeb 9, 2026score 8~115 min
- FebSLA2: Sparse-Linear Attention with Learnable Routing and QATdiffusion2602.12675Feb 13, 2026~115 min
- FebPrincipled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendationllm-systems2602.07298AI at MetaFeb 7, 2026score 4~120 min
- FebArcee Trinity Large Technical Reportmoe2602.17004Feb 19, 2026~114 min
- FebKimi K2.5: Visual Agentic Intelligencemultimodal2602.02276Moonshot / KimiFeb 2, 2026score 8~145 min
- FebSPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningpretraining2602.02472ByteDance SeedFeb 2, 2026score 2~112 min
- FebVideoWorld 2: Learning Transferable Knowledge from Real-world Videospretraining2602.10102ByteDance SeedFeb 10, 2026score 2~107 min
- FebMolHIT: Advancing Molecular-Graph Generation with Hierarchical Discrete Diffusion Modelspretraining2602.17602KAIST AIFeb 19, 2026score 2~123 min
- FebLate-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettertraining-methods2602.05393ByteDance SeedFeb 5, 2026score 8~112 min
- FebABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learningtraining-methods2602.11236Alibaba AMAP CV LabFeb 11, 2026score 2~111 min
- Febjina-embeddings-v5-text: Task-Targeted Embedding Distillationtraining-methods2602.15547Feb 17, 2026~106 min
- FebDreamDojo: A Generalist Robot World Model from Large-Scale Human Videosuncategorized2602.06949NVIDIAFeb 6, 2026score 2~117 min
- JanShaping capabilities with token-level data filteringsafety2601.21571AnthropicJan 29, 2026score 3~110 min
- JanHow Do Large Language Models Learn Concepts During Continual Pre-Training?training-methods2601.03570Jan 7, 2026score 4~121 min
- JanQuartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationtraining-methods2601.22813NVIDIAJan 30, 2026score 9~112 min
- JanDecouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingtraining-methods2602.00747Jan 31, 2026~121 min
2025
70- DecTowards Scalable Pre-training of Visual Tokenizers for Generationdiffusion2512.13687MiniMaxDec 15, 2025~122 min
- DecFour Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalinginference-optimization2512.02010Dec 1, 2025~116 min
- DecBolmo: Byteifying the Next Generation of Language Modelsllm-systems2512.15586Allen Institute for AIDec 17, 2025~120 min
- DecOlmo 3pretraining2512.13961Allen Institute for AIDec 15, 2025~105 min
- DecOn the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelstraining-methods2512.07783CMU-LTIDec 8, 2025score 9~115 min
- DecDiversity or Precision? A Deep Dive into Next Token Predictiontraining-methods2512.22955Dec 28, 2025~107 min
- DecLongCat-Image Technical Reportvision2512.07584Meituan LongCatDec 8, 2025score 6~124 min
- NovReusing Pre-Training Data at Test Time is a Compute Multiplierpretraining2511.04234AppleNov 6, 2025~116 min
- NovOlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observationpretraining2511.13655Ai2Nov 17, 2025score 6~130 min
- NovDepth Anything 3: Recovering the Visual Space from Any Viewsvision2511.10647Nov 13, 2025~100 min
- OctGame-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentsagents2510.23691ByteDanceOct 27, 2025score 8~124 min
- OctLess is More: Recursive Reasoning with Tiny Networksarchitecture2510.04871Samsung AI Lab (SAIL) MontrealOct 6, 2025score 9~116 min
- OctScaling Latent Reasoning via Looped Language Modelsarchitecture2510.25741ByteDance SeedOct 29, 2025~96 min
- OctKimi Linear: An Expressive, Efficient Attention Architecturearchitecture2510.26692Moonshot / KimiOct 30, 2025~98 min
- OctVibe Checker: Aligning Code Evaluation with Human Preferenceevaluation2510.07315Oct 8, 2025~114 min
- OctEvery Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundationmoe2510.22115inclusionAIOct 25, 2025score 10~127 min
- OctHeptapod: Language Modeling on Visual Signalspretraining2510.06673ByteDance SeedOct 8, 2025score 9~128 min
- OctCarleman Estimates for Backward Anisotropic Stochastic Parabolic Equations with General Dynamic Boundary Conditions and Applicationstraining-methods2510.12345Oct 14, 2025~132 min
- OctMemory Retrieval and Consolidation in Large Language Models through Function Tokensuncategorized2510.08203ByteDance SeedOct 9, 2025score 7~115 min
- OctThe End of Manual Decoding: Towards Truly End-to-End Language Modelsuncategorized2510.26697Oct 30, 2025~126 min
- SepEmbeddingGemma: Powerful and Lightweight Text Representationsllm-systems2509.20354DeepMindSep 24, 2025~107 min
- SepHunyuanImage 3.0 Technical Reportmoe2509.23951Tencent HunyuanSep 28, 2025score 9~121 min
- SepLLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingmultimodal2509.23661Sep 28, 2025score 9~125 min
- SepApertus: Democratizing Open and Compliant LLMs for Global Language Environmentspretraining2509.14233Swiss AI InitiativeSep 17, 2025score 9~107 min
- SepPre-training under infinite computepretraining2509.14786Sep 18, 2025~106 min
- SepThinking Augmented Pre-trainingpretraining2509.20186Sep 24, 2025~103 min
- SepPretraining Large Language Models with NVFP4pretraining2509.25149Sep 29, 2025~126 min
- SepRLP: Reinforcement as a Pretraining Objectivepretraining2510.01265NVIDIASep 26, 2025score 9~100 min
- SepFantastic Pretraining Optimizers and Where to Find Themtraining-methods2509.02046Sep 2, 2025~121 min
- SepScaling Agents via Continual Pre-trainingtraining-methods2509.13310Sep 16, 2025~117 min
- SepMuon Outperforms Adam in Tail-End Associative Memory Learningtraining-methods2509.26030Sep 30, 2025~124 min
- SepFront-Loading Reasoning: The Synergy between Pretraining and Post-Training Datatraining-methods2510.03264NVIDIASep 26, 2025score 9~115 min
- SepAToken: A Unified Tokenizer for Visionvision2509.14476AppleSep 17, 2025~104 min
- AugIntern-S1: A Scientific Multimodal Foundation Modelmultimodal2508.15763InternLM / Shanghai AI LabAug 21, 2025~122 min
- AugDeep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMspretraining2508.06601Aug 8, 2025score 6~110 min
- AugDINOv3vision2508.10104Meta AI / FAIRAug 13, 2025~120 min
- JulFalcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancearchitecture2507.22448Jul 30, 2025score 10~132 min
- JulMeta CLIP 2: A Worldwide Scaling Recipemultimodal2507.22062Jul 29, 2025score 10~113 min
- JulToken Bottleneck: One Token to Remember Dynamicstraining-methods2507.06543NAVER AI LabJul 9, 2025score 5~116 min
- JulGR-3 Technical Reportuncategorized2507.15493ByteDance SeedJul 21, 2025score 5~104 min
- JunOpenThoughts: Data Recipes for Reasoning Modelsdata2506.04178DeepSeekJun 4, 2025score 9~121 min
- JunThe Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Textpretraining2506.05209Jun 5, 2025score 10~119 min
- JunReinforcement Pre-Trainingpretraining2506.08007Jun 9, 2025~106 min
- JunAccurate and scalable exchange-correlation with deep learningpretraining2506.14665MicrosoftJun 17, 2025~145 min
- JunScaling Self-Supervised Representation Learning for Symbolic Piano Performancepretraining2506.23869Jun 30, 2025~109 min
- MayEfficientLLM: Efficiency in Large Language Modelsevaluation2505.13840May 20, 2025score 10~104 min
- MayModel Merging in Pre-training of Large Language Modelspretraining2505.12082May 17, 2025~98 min
- MayQuartet: Native FP4 Training Can Be Optimal for Large Language Modelstraining-methods2505.14669NVIDIAMay 20, 2025score 9~124 min
- AprDataDecide: How to Predict Best Pretraining Data with Small Experimentsdata2504.11393Apr 15, 2025~133 min
- AprBitNet b1.58 2B4T Technical Reportllm-systems2504.12285Microsoft ResearchApr 16, 2025score 9~106 min
- AprOpen-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resourcesmultimodal2504.00595Apr 1, 2025score 9~95 min
- AprInternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelsmultimodal2504.10479Apr 14, 2025score 10~101 min
- AprKimi-Audio Technical Reportmultimodal2504.18425Moonshot AIApr 25, 2025score 9~104 min
- AprNemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingpretraining2504.13161Apr 17, 2025score 10~127 min
- AprOLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokenstraining-methods2504.07096Apr 9, 2025score 9~92 min
- AprON LINEAR REPRESENTATIONS AND PRETRAINING DATA FREQUENCY IN LANGUAGE MODELSuncategorized2504.12459Apr 16, 2025~95 min
- MarRWKV-7 "Goose" with Expressive Dynamic State Evolutionarchitecture2503.14456Mar 18, 2025score 9~123 min
- MarPhi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAsmultimodal2503.01743Microsoft ResearchMar 3, 2025score 10~111 min
- MarPolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runspretraining2503.09543EleutherAIMar 12, 2025~146 min
- FebNative Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentionarchitecture2502.11089DeepSeekFeb 16, 2025score 9~127 min
- FebReformulation for Pretraining Data Augmentationdata2502.04235ByteDance SeedFeb 6, 2025score 9~95 min
- FebScaling Pre-training to One Hundred Billion Data for Vision Language Modelsmultimodal2502.07617DeepMindFeb 11, 2025score 9~135 min
- FebSigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featuresmultimodal2502.14786Google ResearchFeb 20, 2025score 8~116 min
- FebSmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelpretraining2502.02737Hugging Face Smol ModelsFeb 4, 2025score 10~133 min
- JanSigma: Differential Rescaling of Query, Key and Value for Efficient Language Modelsarchitecture2501.13629Jan 23, 2025score 9~111 min
- JanQwen2.5-1M Technical Reportcontext-optimization2501.15383Qwen / Alibaba CloudJan 26, 2025~114 min
- Jan2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretrainingdata2501.00958Jan 1, 2025score 10~113 min
- JanCosmos World Foundation Model Platform for Physical AIpretraining2501.03575NVIDIAJan 6, 2025score 5~137 min
- JanAn Empirical Study of Autoregressive Pre-training from Videospretraining2501.05453Jan 9, 2025~102 min
- JanOptimizing Large Language Model Training Using FP4 Quantizationtraining-methods2501.17116Jan 28, 2025score 9~107 min
2024
81- DecByte Latent Transformer: Patches Scale Better Than Tokensarchitecture2412.09871Dec 13, 2024score 9~136 min
- DecFourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationarchitecture2412.17739Dec 23, 2024score 9~109 min
- DecPaliGemma 2: A Family of Versatile VLMs for Transfermultimodal2412.03555Google ResearchDec 4, 2024score 9~114 min
- DecNext Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveymultimodal2412.18619Dec 16, 2024~124 min
- DecEXAONE 3.5: Series of Large Language Models for Real-world Use Casespretraining2412.04862LG EXAONEDec 6, 2024~97 min
- DecQwen2.5 Technical Reportpretraining2412.15115Qwen / Alibaba CloudDec 19, 2024~109 min
- Dec2 OLMo 2 Furiouspretraining2501.00656Allen Institute for AIDec 31, 2024~111 min
- DecPhi-4 Technical Reporttraining-methods2412.08905Microsoft ResearchDec 12, 2024~112 min
- DecNo More Adam: Learning Rate Scaling at Initialization is All You Needtraining-methods2412.11768Dec 16, 2024score 9~100 min
- NovKV Shifting Attention Enhances Language Modelingarchitecture2411.19574Nov 29, 2024score 9~124 min
- NovOPENCODER: THE OPEN COOKBOOK FOR TOP-TIER CODE LARGE LANGUAGE MODELScode2411.04905Nov 7, 2024~110 min
- NovREAD MOREdata2411.12372Together AINov 19, 2024~127 min
- NovHunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentmoe2411.02265Tencent HunyuanNov 4, 2024score 9~110 min
- NovBalancing Pipeline Parallelism with Vocabulary Parallelismtraining-methods2411.05288Nov 8, 2024score 9~118 min
- OctA Survey of Small Language Modelsllm-systems2410.20011Oct 25, 2024~119 min
- OctPre-training Distillation for Large Language Models: A Design Space Explorationpretraining2410.16215Oct 21, 2024score 10~124 min
- OctRATIONALYST: Mining Implicit Rationales for Process Supervision of Reasoningreasoning2410.01044Oct 1, 2024score 9~110 min
- OctDocument Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractionvision2410.21169Oct 28, 2024score 10~114 min
- SepInstruction Following without Instruction Tuningalignment2409.14254Sep 21, 2024score 9~115 min
- SepQwen2.5-Coder Technical Reportcode2409.12186Qwen / Alibaba CloudSep 18, 2024~110 min
- SepEuroLLM: Multilingual Language Models for Europellm-systems2409.16235Sep 24, 2024score 9~103 min
- SepOLMoE: Open Mixture-of-Experts Language Modelsmoe2409.02060Allen Institute for AISep 3, 2024~96 min
- SepEmu3: Next-Token Prediction is All You Needmultimodal2409.18869Sep 27, 2024~122 min
- SepScaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationpretraining2409.12903AppleSep 19, 2024score 9~102 min
- AugTransfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelmultimodal2408.11039Aug 20, 2024score 9~106 min
- AugTo Code, or Not To Code? Exploring Impact of Code in Pre-trainingpretraining2408.10914Aug 20, 2024~120 min
- AugBaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baselinepretraining2408.15079Aug 27, 2024score 9~122 min
- AugMemory-Efficient LLM Training with Online Subspace Descenttraining-methods2408.12857Aug 23, 2024score 7~126 min
- AugSAM 2: Segment Anything in Images and Videosvision2408.00714Meta AI / FAIRAug 1, 2024score 9~135 min
- JulSpectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalepretraining2407.12327Jul 17, 2024score 9~98 min
- JulScaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariespretraining2407.13623Jul 18, 2024score 9~122 min
- JulCompact Language Models via Pruning and Knowledge Distillationpretraining2407.14679Jul 19, 2024~140 min
- JulGemma 2: Improving Open Language Models at a Practical Sizepretraining2408.00118Google ResearchJul 31, 2024~106 min
- JulScaling Retrieval-Based Language Models with a Trillion-Token Datastoreretrieval2407.12854Jul 9, 2024~112 min
- JulScaling Granite Code Models to 128K Contexttraining-methods2407.13739IBM ResearchJul 18, 2024score 8~113 min
- JulGENERALIZATION V.S. MEMORIZATION: TRACING LANGUAGE MODELS’ CAPABILITIES BACK TO PRETRAINING DATAtraining-methods2407.14985Jul 20, 2024~131 min
- JunDataComp-LM: In Search of the Next Generation of Training Sets for Language Modelsdata2406.11794AppleJun 17, 2024score 10~93 min
- JunThe FineWeb Datasets: Decanting the Web for the Finest Text Data at Scaledata2406.17557FineDataJun 25, 2024score 10~123 min
- JunScaling Synthetic Data Creation with 1,000,000,000 Personasdata2406.20094Jun 28, 2024~88 min
- JunChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolsllm-systems2406.12793Zhipu / GLMJun 18, 2024score 9~108 min
- JunHow Do Large Language Models Acquire Factual Knowledge During Pretraining?pretraining2406.11813Jun 17, 2024score 9~107 min
- JunInstruction Pre-Training: Language Models are Supervised Multitask Learnerspretraining2406.14491Jun 20, 2024score 10~97 min
- JunDeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencepretraining2406.11931DeepSeekJun 17, 2024score 10~11 min
- MayBiMix: A Bivariate Data Mixing Law for Language Model Pretrainingpretraining2405.14908May 23, 2024score 9~105 min
- AprEagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencearchitecture2404.05892Apr 8, 2024score 9~133 min
- AprRecurrentGemma: Moving Past Transformers for Efficient Open Language Modelsllm-systems2404.07839Apr 11, 2024score 10~108 min
- AprJetMoE: Reaching Llama2 Performance with 0.1M Dollarsmoe2404.07413Apr 11, 2024score 9~79 min
- AprRho-1: Not All Tokens Are What You Needpretraining2404.07965Microsoft ResearchApr 11, 2024score 9~96 min
- AprReFT: Representation Finetuning for Language Modelstraining-methods2404.03592Apr 4, 2024~102 min
- AprStream of Search (SoS): Learning to Search in Languagetraining-methods2404.03683Apr 1, 2024score 9~121 min
- MarJamba: A Hybrid Transformer-Mamba Language Modelarchitecture2403.19887AI21Mar 28, 2024~130 min
- MarGemma: Open Models Based on Gemini Research and Technologyllm-systems2403.08295Google ResearchMar 13, 2024score 9~105 min
- MarDeepSeek-VL: Towards Real-World Vision-Language Understandingmultimodal2403.05525DeepSeekMar 8, 2024score 9~88 min
- MarSimple and Scalable Strategies to Continually Pre-train Large Language Modelspretraining2403.08763Mar 13, 2024score 9~127 min
- MarThe Unreasonable Ineffectiveness of the Deeper Layerspretraining2403.17887Mar 26, 2024score 9~100 min
- MarQuiet-STaR: Language Models Can Teach Themselves to Think Before Speakingreasoning2403.09629Mar 14, 2024score 9~107 min
- MarYi: Open Foundation Models by 01.AIscaling-laws2403.0465201-AIMar 7, 2024score 9~73 min
- MarGaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectiontraining-methods2403.03507Mar 6, 2024score 9~103 min
- MarRAFT: Adapting Language Model to Domain Specific RAGtraining-methods2403.10131Mar 15, 2024~103 min
- MarInternLM2 Technical Reporttraining-methods2403.17297InternLM / Shanghai AI LabMar 26, 2024score 9~92 min
- FebOLMo: Accelerating the Science of Language Modelsagents2402.00838Allen Institute for AIFeb 1, 2024score 10~113 min
- FebStarCoder 2 and The Stack v2: The Next Generationcode2402.19173RobloxFeb 29, 2024score 9~111 min
- FebMultilingual E5 Text Embeddings: A Technical Reportcontext-optimization2402.05672Feb 8, 2024~118 min
- FebData Engineering for Scaling Language Models to 128K Contextdata2402.10171Feb 15, 2024score 9~112 min
- FebThe Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicryllm-systems2402.04347Feb 6, 2024score 9~113 min
- FebFractal Patterns May Illuminate the Success of Next-Token Predictionpretraining2402.01825DeepMindFeb 2, 2024~134 min
- FebOmniPred: Language Models as Universal Regressorspretraining2402.14547DeepMindFeb 22, 2024score 5~101 min
- FebGenie: Generative Interactive Environmentspretraining2402.15391DeepMindFeb 23, 2024score 9~111 min
- FebNemotron-4 15B Technical Reportpretraining2402.16819Feb 26, 2024~106 min
- FebThe Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitspretraining2402.17764DropboxFeb 27, 2024~118 min
- FebDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelsreasoning2402.03300DeepSeekFeb 5, 2024score 10~130 min
- FebMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUstraining-methods2402.15627Feb 23, 2024score 9~116 min
- FebInternLM-Math: Open Math Large Language Models Toward Verifiable Reasoninguncategorized2402.06332InternLM / Shanghai AI LabFeb 9, 2024score 9~97 min
- JanMambaByte: Token-free Selective State Space Modelarchitecture2401.13660Jan 24, 2024score 9~116 min
- JanDeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecode2401.14196DeepSeekJan 25, 2024score 10~96 min
- JanRephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingdata2401.16380AppleJan 29, 2024score 9~109 min
- JanInfini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokensdata2401.17377Jan 30, 2024score 10~114 min
- JanDolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchdata2402.00159Allen Institute for AIJan 31, 2024score 10~137 min
- JanMixtral of Expertsmoe2401.04088Jan 8, 2024~103 min
- JanLearning Universal Predictorspretraining2401.14953DeepMindJan 26, 2024score 7~105 min
- JanDeepSeek LLM: Scaling Open-Source Language Models with Longtermismscaling-laws2401.02954Jan 5, 2024~129 min
2023
41- DecCodestral Mambaarchitecture2312.00752MistralDec 1, 2023~109 min
- DecLLM360: Towards Fully Transparent Open-Source LLMspretraining2312.06550Dec 11, 2023score 9~104 min
- DecBeyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelstraining-methods2312.06585Dec 11, 2023score 10~116 min
- DecVILA: On Pre-training for Visual Language Modelsvision2312.07533Dec 12, 2023score 9~90 min
- NovMemory Augmented Language Models through Mixture of Word Expertsarchitecture2311.10768Nov 15, 2023score 9~93 min
- NovQwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelsmultimodal2311.07919Qwen / Alibaba CloudNov 14, 2023score 8~116 min
- OctMistral 7Bpretraining2310.06825MistralOct 10, 2023~97 min
- OctLlemma: An Open Language Model For Mathematicspretraining2310.10631Stability AIOct 16, 2023score 9~105 min
- OctIN-CONTEXT PRETRAINING: LANGUAGE MODELING BEYOND DOCUMENT BOUNDARIESpretraining2310.10638Oct 16, 2023~111 min
- OctBitNet: Scaling 1-bit Transformers for Large Language Modelspretraining2310.11453Oct 17, 2023score 9~106 min
- OctSum of the GL(3) Fourier Coefficients over Quadratics and Mixed Powersreasoning2310.11408Oct 17, 2023~132 min
- OctConvNets Match Vision Transformers at Scalevision2310.16764Oct 25, 2023score 9~119 min
- OctPockEngine: Sparse and Efficient Fine-tuning in a Pocketno summary yetcs-lg2310.1775213 citesOct 26, 2023score 7
- SepScaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningpretraining2309.02591Sep 5, 2023score 10~113 min
- SepWhen Less is More: Investigating Data Pruning for Pretraining LLMs at Scalepretraining2309.04564Sep 8, 2023score 9~110 min
- SepTextbooks Are All You Need II: phi-1.5 technical reportpretraining2309.05463Microsoft ResearchSep 11, 2023~92 min
- SepScaling Laws for Sparsely-Connected Foundation Modelsscaling-laws2309.08520Sep 15, 2023score 9~115 min
- SepEffective Long-Context Scaling of Foundation Modelstraining-methods2309.16039Sep 27, 2023score 9~104 min
- AugYARN: EFFICIENT CONTEXT WINDOW EXTENSION OF LARGE LANGUAGE MODELStraining-methods2309.00071Qwen / Alibaba CloudAug 31, 2023~112 min
- JulLlama 2: Open Foundation and Fine-Tuned Chat Modelsalignment2307.09288NVIDIAJul 18, 2023~120 min
- JulIn-context Autoencoder for Context Compression in a Large Language Modelcontext-optimization2307.06945Jul 13, 2023score 9~107 min
- JulScaling TransNormer to 175 Billion Parametersllm-systems2307.14995Jul 27, 2023score 9~114 min
- JulRead Moretraining-methods2307.05695EleutherAIJul 11, 2023score 6~112 min
- JunExtending Context Window of Large Language Models via Positional Interpolationcontext-optimization2306.15595Zhipu / GLMJun 27, 2023score 9~122 min
- JunThe RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlydata2306.01116Jun 1, 2023score 9~120 min
- JunTextbooks Are All You Needdata2306.11644Microsoft ResearchJun 20, 2023~112 min
- JunRead Morepretraining2306.02254EleutherAIJun 4, 2023score 5~108 min
- JunQuantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingpretraining2306.12929Jun 22, 2023score 8~107 min
- JunEstimating the Causal Effect of Early ArXiving on Paper Acceptancesafety2306.13891Jun 24, 2023~116 min
- MayStarCoder: may the source be with you!code2305.06161Stability AIMay 9, 2023score 8~124 min
- MayDoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingdata2305.10429May 17, 2023score 9~98 min
- MayPaLM 2 Technical Reportpretraining2305.10403May 17, 2023~108 min
- MayScaling Speech Technology to 1,000+ Languagespretraining2305.13516Meta AI / FAIRMay 22, 2023~162 min
- MayDrag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldno summary yetcs-cv2305.1097311 citesMay 18, 2023score 2
- AprDINOv2: Learning Robust Visual Features without Supervisionpretraining2304.07193Meta AI / FAIRApr 14, 2023~104 min
- AprSegment Anythingvision2304.02643Meta AI / FAIRApr 5, 2023~122 min
- MarGPT-4 Technical Reportpretraining2303.08774Mar 15, 2023~118 min
- FebLLaMA: Open and Efficient Foundation Language Modelspretraining2302.13971Stability AIFeb 27, 2023~120 min
- FebThe geometry of hidden representations of large transformer modelsscaling-laws2302.00294Feb 1, 2023~127 min
- FebToolformer: Language Models Can Teach Themselves to Use Toolstraining-methods2302.04761Feb 9, 2023~129 min
- JanBLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelsmultimodal2301.12597SalesforceJan 30, 2023~119 min
2022
14- DecRobust Speech Recognition via Large-Scale Weak Supervisionpretraining2212.04356OpenAIDec 6, 2022~119 min
- NovRead Moretraining-methods2211.05100EleutherAINov 9, 2022~119 min
- SepAudioLM: a Language Modeling Approach to Audio Generationpretraining2209.03143Sep 7, 2022~118 min
- SepIn-context Learning and Induction Headsuncategorized2209.11895Sep 24, 2022~120 min
- AugAtlas: Few-shot Learning with Retrieval Augmented Language Modelsretrieval2208.03299Aug 5, 2022~124 min
- MayOPT: Open Pre-trained Transformer Language Modelspretraining2205.01068Meta AI / FAIRMay 2, 2022~125 min
- MayUL2: Unifying Language Learning Paradigmspretraining2205.05131Together AIMay 10, 2022~122 min
- AprGPT-NeoX-20B: An Open-Source Autoregressive Language Modelllm-systems2204.06745Stability AIApr 14, 2022~111 min
- AprFlamingo: a Visual Language Model for Few-Shot Learningmultimodal2204.14198Apr 29, 2022~97 min
- AprOpen Source and In-House: How Uber Optimizes LLM Trainingpretraining2204.02311UberApr 5, 2022~108 min
- MarTraining Compute-Optimal Large Language Modelsscaling-laws2203.15556Mar 29, 2022~111 min
- FebRethinking the Role of Demonstrations: What Makes In-Context Learning Work?prompting2202.12837Feb 25, 2022~119 min
- JanBLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationmultimodal2201.12086SalesforceJan 28, 2022~104 min
- JanGrokking: Generalization Beyond Overfitting on Small Algorithmic Datasetstraining-methods2201.02177Jan 6, 2022~125 min
2021
14- DecGLaM: Efficient Scaling of Language Models with Mixture-of-Expertsmoe2112.06905Dec 13, 2021~114 min
- DecGeneral Facial Representation Learning in a Visual-Linguistic Mannerpretraining2112.03109Dec 6, 2021~120 min
- DecContriever: Unsupervised Dense Information Retrieval with Contrastive Learningretrieval2112.09118Meta AI / FAIRDec 16, 2021~103 min
- DecGopherscaling-laws2112.11446Dec 8, 2021~142 min
- NovDeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingpretraining2111.09543Microsoft ResearchNov 18, 2021~106 min
- NovScaling Law for Recommendation Models: Towards General-purpose User Representationsscaling-laws2111.11294Nov 15, 2021~105 min
- NovFlorence: A New Foundation Model for Computer Visionvision2111.11432Nov 22, 2021~118 min
- AugSimVLM: Simple Visual Language Model Pretraining with Weak Supervisionmultimodal2108.10904Meta AI / FAIRAug 24, 2021~111 min
- JulTAPEX: Table Pre-training via Learning a Neural SQL Executorpretraining2107.07653Microsoft ResearchJul 16, 2021~88 min
- AprEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMdistributed-training2104.04473Apr 9, 2021~101 min
- FebALIGN: Large-Scale Image-Text Representation Learning with Noisy Text Supervisionmultimodal2102.05918Kakao Brain1,196 citesFeb 11, 2021~111 min
- FebA Fast Transformer Decoding System with Better Memory Bandwidth Utilizationmultimodal2103.00020Apple5,296 citesFeb 26, 2021~130 min
- FebConceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptspretraining2102.0898137 citesFeb 17, 2021~111 min
- FebMeasuring and Improving Consistency in Pretrained Language Modelsuncategorized2102.0101732 citesFeb 1, 2021~101 min
2020
18- DecThe Pile: An 800GB Dataset of Diverse Text for Language Modelingdata2101.00027Stability AIDec 31, 2020~117 min
- DecMemorizing Transformerstraining-methods2012.0036342 citesDec 1, 2020~111 min
- OctAutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptsprompting2010.15980Oct 29, 2020~100 min
- OctAn Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.vision2010.11929Apple21,567 citesOct 22, 2020~101 min
- JulFusion-in-Decoder: Open-Domain Question Answering with Retrieval-Augmented Generationllm-systems2007.01282Jul 2, 2020~90 min
- JunGShard: Scaling Giant Models with Conditional Computation and Automatic Shardingllm-systems2006.16668Jun 30, 2020~130 min
- JunBootstrap Your Own Latent: A New Approach to Self-Supervised Learningpretraining2006.077333,445 citesJun 13, 2020~122 min
- Junwav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationspretraining2006.11477Meta AI / FAIRJun 20, 2020~131 min
- MaySynthesizer: Rethinking Self-Attention in Transformer Modelsarchitecture2005.00743198 citesMay 2, 2020~101 min
- MayLanguage Models are Few-Shot Learnerspretraining2005.14165Meta AI / FAIRMay 28, 2020~119 min
- MayRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasksretrieval2005.11401Meta AI / FAIRMay 22, 2020~98 min
- AprLongformer: The Long-Document Transformerarchitecture2004.05150Allen Institute for AI2,199 citesApr 10, 2020~103 min
- AprETC: Encoding Long and Structured Inputs in Transformersarchitecture2004.0848328 citesApr 17, 2020~130 min
- AprOSCAR: Object-Semantics Aligned Pre-training for Vision-Language Tasksmultimodal2004.06165135 citesApr 13, 2020~128 min
- AprDense Passage Retrieval for Open-Domain Question Answeringretrieval2004.04906Meta AI / FAIRApr 10, 2020~113 min
- MarRouting Transformerarchitecture2003.0599749 citesMar 12, 2020~111 min
- FebREALM: Retrieval-Augmented Language Model Pre-Trainingpretraining2002.08909Google Research515 citesFeb 10, 2020~113 min
- Jan2028: Two scenarios for global AI leadershipscaling-laws2001.08361Anthropic1,503 citesJan 23, 2020~129 min
2019
20- DecPEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationpretraining1912.08777Google ResearchDec 18, 2019~84 min
- NovCompressive Transformerarchitecture1911.05507Nov 13, 2019~112 min
- NovUnsupervised Cross-lingual Representation Learning at Scalepretraining1911.02116Microsoft Research539 citesNov 5, 2019~92 min
- OctExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformertraining-methods1910.10683Allen Institute for AIOct 23, 2019~132 min
- OctBART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensiontraining-methods1910.13461Meta AI / FAIROct 29, 2019~102 min
- SepALBERT: A Lite BERT for Self-supervised Learning of Language Representationsarchitecture1909.11942984 citesSep 26, 2019~104 min
- SepMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismdistributed-training1909.08053Sep 17, 2019~105 min
- SepLanguage Models as Knowledge Bases?evaluation1909.01066Sep 3, 2019~114 min
- SepUNITER: Learning Universal Image-Text Representationsmultimodal1909.11740184 citesSep 25, 2019~113 min
- AugViLBERT: Pretraining Task-Agnostic Visiolinguistic Representationsmultimodal1908.02265Aug 6, 2019~109 min
- AugVisualBERT: A Simple and Performant Baseline for Vision and Languagemultimodal1908.03557Aug 9, 2019~119 min
- AugLXMERT: Learning Cross-Modality Encoder Representations from Transformersmultimodal1908.07490Aug 20, 2019~111 min
- JulSpanBERT: Improving Pre-training by Representing and Predicting Spanspretraining1907.10529179 citesJul 24, 2019~119 min
- JulRoBERTa: A Robustly Optimized BERT Pretraining Approachpretraining1907.11692Meta AI / FAIRJul 26, 2019~113 min
- JunXLNet: Generalized Autoregressive Pretraining for Language Understandingpretraining1906.08237Stitch FixJun 19, 2019~118 min
- JunLatent Retrieval for Weakly Supervised Open Domain Question Answeringretrieval1906.0030067 citesJun 1, 2019~102 min
- MayEfficientNet: Rethinking Model Scaling for Convolutional Neural Networksarchitecture1905.11946Google Research5,013 citesMay 28, 2019~109 min
- AprERNIE: Enhanced Representation through Knowledge Integrationpretraining1904.09223770 citesApr 19, 2019~100 min
- FebParameter-Efficient Transfer Learning for NLPtraining-methods1902.00751144 citesFeb 2, 2019~115 min
- JanTransformer-XL: Attentive Language Models Beyond a Fixed-Length Contextarchitecture1901.02860Jan 9, 2019~110 min
2018
6- OctHow Powerful Are Graph Neural Networks?architecture1810.00826Oct 1, 2018~119 min
- OctBERT: Pre-training of Deep Bidirectional Transformers for Language Understandingpretraining1810.04805DeepMind45,647 citesOct 11, 2018~131 min
- JulRepresentation Learning with Contrastive Predictive Codingpretraining1807.037483,768 citesJul 10, 2018~112 min
- MarUnsupervised Representation Learning by Predicting Image Rotationsvision1803.07728Mar 21, 2018~126 min
- FebDeep contextualized word representationspretraining1802.053651,791 citesFeb 15, 2018~112 min
- JanImproving Language Understanding by Generative Pre-Trainingtraining-methods1801.06146Jan 18, 2018~117 min
2017
12016
22015
22014
4- SepSequence to Sequence Learning with Neural Networkstraining-methods1409.321513,358 citesSep 10, 2014~112 min
- SepImageNet Large Scale Visual Recognition Challengevision1409.0575Sep 1, 2014~104 min
- MayDistributed Representations of Sentences and Documentsarchitecture1405.40535,128 citesMay 16, 2014~120 min
- JanNeural Word Embedding as Implicit Matrix Factorizationno summary yetpretrainingext:word-embed-mf-2014Jan 1, 2014
2013
4- DecExact solutions to the nonlinear dynamics of learning in deep linear neural networkstraining-methods1312.6120Dec 20, 2013~119 min
- NovRich feature hierarchies for accurate object detection and semantic segmentationvision1311.2524Nov 11, 2013~118 min
- OctDistributed Representations of Words and Phrases and their Compositionalitypretraining1310.454618,086 citesOct 16, 2013~114 min
- JanEfficient Estimation of Word Representations in Vector Spacepretraining1301.378118,139 citesJan 16, 2013~122 min
2000
12- —A Fast Learning Algorithm for Deep Belief Netspretraining~133 min
- —Distributed Representations of Sentences and Documentspretraining~113 min
- —Deep contextualized word representationspretraining~127 min
- —GloVe: Global Vectors for Word Representationpretraining~124 min
- —Improving Language Understanding by Generative Pre-Trainingpretraining~93 min
- —Language Models are Unsupervised Multitask Learnerspretraining~119 min
- —Hierarchical Dirichlet Processespretraining~112 min
- —Magnetic Lorentz Force Drives Giant Second-Harmonic Generation in Split-Ring Resonator Metamaterialspretraining~118 min
- —Reducing the Dimensionality of Data with Neural Networkspretraining~103 min
- —Kernel Methods in Machine Learningpretraining~131 min
- —Latent Dirichlet Allocationpretraining~118 min
- —Random Forestspretraining~147 min