Pretraining recipes
90 papers in this thread, across 21 domains.
Progress0 of 85
- 2026MolmoAct2: Action Reasoning Models for Real-world DeploymentAgents2605.02881Allen Institute for AIMay 4, 2026~118 minAgents
- 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language ModelsAlignment2605.11887Qwen / Alibaba CloudMay 12, 2026~138 minAlignment
- 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingArchitecture2605.08738May 9, 2026~106 minArchitecture
- 2026Qwen-Image-2.0 Technical ReportArchitecture2605.10730QwenMay 11, 2026~115 minArchitecture
- 2026Qwen-Image-VAE-2.0 Technical ReportVision2605.13565QwenMay 13, 2026~110 minVision
- 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open WebLLM Systems2604.08516Apr 9, 2026~100 minLLM Systems
- 2026Qwen3.5-Omni Technical ReportMultimodal2604.15804Apr 17, 2026~123 minMultimodal
- 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal InteractionMultimodal2604.27393OpenBMBApr 30, 2026~120 minMultimodal
- 2026Polyglot-Lion: Efficient Multilingual ASR for Singapore via Balanced Fine-Tuning of Qwen3-ASRArchitecture2603.16184Knovel EngineeringMar 17, 2026score 3~116 minArchitecture
- 2026MolmoB0T: Large-Scale Simulation Enables Zero-Shot ManipulationUncategorized2603.16861Ai2Mar 17, 2026score 9~120 minUncategorized
- 2026MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context ModelingArchitecture2602.11761Feb 12, 2026~104 minArchitecture
- 2026Qwen3-Coder-Next Technical ReportTraining Methods2603.00729QwenFeb 28, 2026score 9~123 minTraining Methods
- 2026Qwen3-TTS Technical ReportArchitecture2601.15621Qwen / Alibaba CloudJan 22, 2026score 2~120 minArchitecture
- 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and RankingMultimodal2601.04720Qwen / Alibaba CloudJan 8, 2026~114 minMultimodal
- 2026A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5Safety2601.10527Jan 15, 2026score 4~118 minSafety
- 2026TranslateGemma Technical ReportTraining Methods2601.09012Google ResearchJan 13, 2026~107 minTraining Methods
- 2026Qwen3-ASR Technical ReportTraining Methods2601.21337Qwen / Alibaba CloudJan 29, 2026score 2~125 minTraining Methods
- 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and GroundingVision2601.10611Jan 15, 2026score 9~97 minVision
- 2025Qwen-Image-Layered: Towards Inherent Editability via Layer DecompositionDiffusion2512.15603Qwen / Alibaba CloudDec 17, 2025score 6~110 minDiffusion
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsLLM Systems2512.02556DeepSeekDec 2, 2025~127 minLLM Systems
- 2025Bolmo: Byteifying the Next Generation of Language ModelsLLM Systems2512.15586Allen Institute for AIDec 17, 2025~120 minLLM Systems
- 2025Olmo 3Pretraining2512.13961Allen Institute for AIDec 15, 2025~105 minPretraining
- 2025QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory ManagementTraining Methods2512.12967Dec 15, 2025~110 minTraining Methods
- 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual TasksLLM Systems2511.07025NVIDIANov 10, 2025score 9~99 minLLM Systems
- 2025Qwen3-VL Technical ReportMultimodal2511.21631Nov 26, 2025~107 minMultimodal
- 2025OlmoEarth: Stable Latent Image Modeling for Multimodal Earth ObservationPretraining2511.13655Ai2Nov 17, 2025score 6~130 minPretraining
- 2025olmOCR 2: Unit Test Rewards for Document OCRMultimodal2510.19817Ai2Oct 22, 2025score 9~115 minMultimodal
- 2025Qwen3Guard Technical ReportSafety2510.14276Qwen / Alibaba CloudOct 16, 2025score 8~136 minSafety
- 2025EmbeddingGemma: Powerful and Lightweight Text RepresentationsLLM Systems2509.20354DeepMindSep 24, 2025~107 minLLM Systems
- 2025Qwen3-Omni Technical ReportMultimodal2509.17765Sep 22, 2025~129 minMultimodal
- 2025Qwen-Image Technical ReportMultimodal2508.02324Qwen / Alibaba CloudAug 4, 2025~137 minMultimodal
- 2025MolmoAct: Action Reasoning Models that can Reason in SpaceUncategorized2508.07917Aug 11, 2025~111 minUncategorized
- 2025FlexOlmo: Open Language Models for Flexible Data UseMixture of Experts2507.07024Jul 9, 2025~105 minMixture of Experts
- 2025MiniCPM4: Ultra-Efficient LLMs on End DevicesLLM Systems2506.07900OpenBMBJun 9, 2025score 9~122 minLLM Systems
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation ModelsPretraining2506.05176Qwen / Alibaba CloudJun 5, 2025score 10~82 minPretraining
- 2025MagistralReasoning2506.10910MistralJun 12, 2025~105 minReasoning
- 2025QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context OptimizationContext Optimization2505.18092May 23, 2025score 9~115 minContext Optimization
- 2025Qwen3 Technical ReportLLM Systems2505.09388Qwen / Alibaba CloudMay 14, 2025~128 minLLM Systems
- 2025An Empirical Study of Qwen3 QuantizationLow Precision2505.02214May 4, 2025score 9~89 minLow Precision
- 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI ArchitecturesMixture of Experts2505.09343NVIDIAMay 14, 2025score 10~111 minMixture of Experts
- 2025Llama-Nemotron: Efficient Reasoning ModelsReasoning2505.00949DeepSeekMay 2, 2025~96 minReasoning
- 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement LearningRL Training2505.17667May 23, 2025score 9~118 minRL Training
- 2025Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic ResourcesMultimodal2504.00595Apr 1, 2025score 9~95 minMultimodal
- 2025Phi-4-reasoning Technical ReportReasoning2504.21318Microsoft ResearchApr 30, 2025~123 minReasoning
- 2025OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training TokensTraining Methods2504.07096Apr 9, 2025score 9~92 minTraining Methods
- 2025Gemma 3 Technical ReportArchitecture2503.19786Google ResearchMar 25, 2025~112 minArchitecture
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAsMultimodal2503.01743Microsoft ResearchMar 3, 2025score 10~111 minMultimodal
- 2025Qwen2.5-Omni Technical ReportMultimodal2503.20215Qwen / Alibaba CloudMar 26, 2025~106 minMultimodal
- 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training RunsPretraining2503.09543EleutherAIMar 12, 2025~146 minPretraining
- 2025Qwen2.5-VL Technical ReportMultimodal2502.13923Qwen / Alibaba CloudFeb 19, 2025~122 minMultimodal
- 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language ModelsMultimodal2502.18443Feb 25, 2025~135 minMultimodal
- 2025Qwen2.5-1M Technical ReportContext Optimization2501.15383Qwen / Alibaba CloudJan 26, 2025~114 minContext Optimization
- 2024PaliGemma 2: A Family of Versatile VLMs for TransferMultimodal2412.03555Google ResearchDec 4, 2024score 9~114 minMultimodal
- 2024Qwen2.5 Technical ReportPretraining2412.15115Qwen / Alibaba CloudDec 19, 2024~109 minPretraining
- 20242 OLMo 2 FuriousPretraining2501.00656Allen Institute for AIDec 31, 2024~111 minPretraining
- 2024Phi-4 Technical ReportTraining Methods2412.08905Microsoft ResearchDec 12, 2024~112 minTraining Methods
- 2024Qwen2.5-Coder Technical ReportCode2409.12186Qwen / Alibaba CloudSep 18, 2024~110 minCode
- 2024OLMoE: Open Mixture-of-Experts Language ModelsMixture of Experts2409.02060Allen Institute for AISep 3, 2024~96 minMixture of Experts
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any ResolutionMultimodal2409.12191Qwen / Alibaba CloudSep 18, 2024score 9~112 minMultimodal
- 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal ModelsMultimodal2409.17146Allen Institute for AISep 25, 2024score 9~121 minMultimodal
- 2024A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Rationo summary yetcs cl2409.06624Alibaba0 citesSep 10, 2024cs cl
- 2024Qwen2 Technical ReportLLM Systems2407.10671Qwen / Alibaba CloudJul 15, 2024~114 minLLM Systems
- 2024Qwen2-Audio Technical ReportMultimodal2407.10759Qwen / Alibaba CloudJul 15, 2024score 6~104 minMultimodal
- 2024Gemma 2: Improving Open Language Models at a Practical SizePretraining2408.00118Google ResearchJul 31, 2024~106 minPretraining
- 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" CycleSafety2407.13833Microsoft ResearchJul 18, 2024score 9~110 minSafety
- 2024The Llama 3 Herd of ModelsTraining Methods2407.21783Jul 31, 2024~129 minTraining Methods
- 2024KAN: Kolmogorov-Arnold NetworksArchitecture2404.19756Apr 30, 2024score 6~117 minArchitecture
- 2024Extending Llama-3's Context Ten-Fold OvernightContext Optimization2404.19553Apr 30, 2024score 9~103 minContext Optimization
- 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language ModelsLLM Systems2404.07839Apr 11, 2024score 10~108 minLLM Systems
- 2024JetMoE: Reaching Llama2 Performance with 0.1M DollarsMixture of Experts2404.07413Apr 11, 2024score 9~79 minMixture of Experts
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your PhonePretraining2404.14219Microsoft ResearchApr 22, 2024score 9~114 minPretraining
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training StrategiesTraining Methods2404.06395OpenBMBApr 9, 2024score 9~139 minTraining Methods
- 2024Gemma: Open Models Based on Gemini Research and TechnologyLLM Systems2403.08295Google ResearchMar 13, 2024score 9~105 minLLM Systems
- 2024DeepSeek-VL: Towards Real-World Vision-Language UnderstandingMultimodal2403.05525DeepSeekMar 8, 2024score 9~88 minMultimodal
- 2024OLMo: Accelerating the Science of Language ModelsAgents2402.00838Allen Institute for AIFeb 1, 2024score 10~113 minAgents
- 2024Mixtral of ExpertsMixture of Experts2401.04088Jan 8, 2024~103 minMixture of Experts
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language ModelsMultimodal2311.07919Qwen / Alibaba CloudNov 14, 2023score 8~116 minMultimodal
- 2023Mistral 7BPretraining2310.06825MistralOct 10, 2023~97 minPretraining
- 2023Qwen Technical ReportAlignment2309.16609Stability AISep 28, 2023score 9~119 minAlignment
- 2023Textbooks Are All You Need II: phi-1.5 technical reportPretraining2309.05463Microsoft ResearchSep 11, 2023~92 minPretraining
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and BeyondMultimodal2308.12966Qwen / Alibaba CloudAug 24, 2023~97 minMultimodal
- 2023Llama 2: Open Foundation and Fine-Tuned Chat ModelsAlignment2307.09288NVIDIAJul 18, 2023~120 minAlignment
- 2023PaLM 2 Technical ReportPretraining2305.10403May 17, 2023~108 minPretraining
- 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init AttentionTraining Methods2303.16199Mar 28, 2023~100 minTraining Methods
- 2023LLaMA: Open and Efficient Foundation Language ModelsPretraining2302.13971Stability AIFeb 27, 2023~120 minPretraining
- 2022Open Source and In-House: How Uber Optimizes LLM TrainingPretraining2204.02311UberApr 5, 2022~108 minPretraining
- 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetsno summary yetcs cl2106.10328OpenAI74 citesJun 18, 2021cs cl
- 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningno summary yetcs lg2102.12560Google Research5 citesFeb 24, 2021cs lg
- 2019Pythia: AI-assisted Code Completion Systemno summary yetcs se1912.00742Microsoft Research114 citesNov 29, 2019cs se
- 2017On the Gaussianity of Kolmogorov Complexity of Mixing Sequencesno summary yetcs it1702.01317Microsoft Research0 citesFeb 4, 2017cs it