84 papers
Distributed Training
0/83Multi-GPU and multi-node training systems.
Progress0 of 83
2026
7- MayLongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationarchitecture2605.18739NVIDIAMay 18, 2026~131 min
- FebUntied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunkingllm-systems2602.21196TogetherFeb 24, 2026score 9~105 min
- FebSPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningpretraining2602.02472ByteDance SeedFeb 2, 2026score 2~112 min
- FebCanzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizerstraining-methods2602.06079QwenFeb 4, 2026score 5~119 min
- JanHybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextsarchitecture2601.22156OpenBMBJan 29, 2026score 9~127 min
- JanMoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUsdistributed-training2601.05296Jan 8, 2026~113 min
- JanMegaFlow: Large-Scale Distributed Orchestration System for the Agentic Eraserving2601.07526QwenJan 12, 2026score 6~109 min
2025
17- DecFast-FoundationStereo: Real-Time Zero-Shot Stereo Matchingarchitecture2512.11130NVIDIADec 11, 2025score 4~108 min
- NovParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernelsdistributed-training2511.13940Together AINov 17, 2025~135 min
- NovFP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Errortraining-methods2511.02302Nov 4, 2025~122 min
- NovNemotron Elastic: Towards Efficient Many-in-One Reasoning LLMstraining-methods2511.16664NVIDIANov 20, 2025score 9~127 min
- OctLongCat-Flash-Omni Technical Reportmultimodal2511.00279Meituan LongCatOct 31, 2025score 9~117 min
- OctRDMA Point-to-Point Communication for LLM Systemsserving2510.27656Oct 31, 2025~117 min
- OctDCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelismtraining-methods2510.10620Oct 12, 2025~99 min
- OctEfficient Long-context Language Model Training by Core Attention Disaggregationtraining-methods2510.18121Oct 20, 2025score 10~105 min
- SepSharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharingtraining-methods2509.08721DeepSeekSep 10, 2025score 10~108 min
- AugTPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferenceserving2508.15881DeepSeekAug 21, 2025score 9~117 min
- JulZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attentiondistributed-training2507.01004Jul 1, 2025score 9~111 min
- JulFlexOlmo: Open Language Models for Flexible Data Usemoe2507.07024Jul 9, 2025~105 min
- JulHelix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decodingserving2507.07120Jul 7, 2025~95 min
- JulAXLearn: Modular Large Model Training on Heterogeneous Infrastructuretraining-methods2507.05411AppleJul 7, 2025score 8~94 min
- MayInsights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturesmoe2505.09343NVIDIAMay 14, 2025score 10~111 min
- AprMSCCL++: Rethinking GPU Communication Abstractions for AI Inferenceserving2504.09014Apr 11, 2025~111 min
- JanStreaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchdistributed-training2501.18512Jan 30, 2025score 10~109 min
2024
16- NovContext Parallelism for Scalable Million-Token Inferenceinference-optimization2411.01783Nov 4, 2024~130 min
- NovStar Attention: Efficient LLM Inference over Long Sequencesinference-optimization2411.17116Nov 26, 2024score 9~124 min
- NovBalancing Pipeline Parallelism with Vocabulary Parallelismtraining-methods2411.05288Nov 8, 2024score 9~118 min
- NovDeMo: Decoupled Momentum Optimizationtraining-methods2411.19870Nov 29, 2024score 7~101 min
- OctRevisiting Reliability in Large-Scale Machine Learning Research Clustersdistributed-training2410.21680Oct 29, 2024~112 min
- AugThe Mamba in the Llama: Distilling and Accelerating Hybrid Modelsarchitecture2408.15237Aug 27, 2024~116 min
- JulInference Performance Optimization for Large Language Models on CPUsinference-optimization2407.07304Jul 10, 2024score 9~98 min
- JunAdam-mini: Use Fewer Learning Rates To Gain Moretraining-methods2406.16793Jun 24, 2024score 9~100 min
- MayOpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Frameworkalignment2405.11143May 20, 2024score 9~99 min
- MayDistributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inferenceinference-optimization2405.14105May 23, 2024score 9~89 min
- MarDiPaCo: Distributed Path Compositiondistributed-training2403.10616DeepMindMar 15, 2024score 9~133 min
- MarBranch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMmoe2403.07816Mar 12, 2024score 9~107 min
- MarLoHan: Low-Cost High-Performance Framework to Fine-Tune 100B Model on a Consumer GPUtraining-methods2403.06504Mar 11, 2024score 9~120 min
- FebGriffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelsarchitecture2402.19427Google ResearchFeb 29, 2024score 9~123 min
- FebMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUstraining-methods2402.15627Feb 23, 2024score 9~116 min
- JanAsynchronous Local-SGD Training for Language Modelingtraining-methods2401.09135DeepMindJan 17, 2024score 7~125 min
2023
13- NovUltra-Long Sequence Distributed Transformerdistributed-training2311.02382Nov 4, 2023score 9~130 min
- NovZero Bubble Pipeline Parallelismdistributed-training2401.10241Nov 30, 2023score 9~105 min
- NovAMSP: Super-Scaling LLM Training via Advanced Model States Partitioningtraining-methods2311.00257Nov 1, 2023score 9~117 min
- NovDiLoCo: Distributed Low-Communication Training of Language Modelstraining-methods2311.08105DeepMindNov 14, 2023score 9~128 min
- OctRing Attention with Blockwise Transformers for Near-Infinite Contextllm-systems2310.01889Oct 3, 2023~101 min
- OctFP8-LM: Training FP8 Large Language Modelstraining-methods2310.18313Oct 27, 2023~108 min
- SepDeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelsdistributed-training2309.14509Sep 25, 2023score 8~106 min
- JulLongNet: Scaling Transformers to 1,000,000,000 Tokensarchitecture2307.02486Jul 5, 2023~123 min
- JunFull Parameter Fine-tuning for Large Language Models with Limited Resourcestraining-methods2306.09782Jun 16, 2023score 9~100 min
- MayFast Distributed Inference Serving for Large Language Modelsserving2305.05920May 10, 2023~112 min
- AprPyTorch FSDP: Experiences on Scaling Fully Sharded Data Paralleldistributed-training2304.11277Apr 21, 2023~121 min
- AprPyTorch FSDP: Experiences on Scaling Fully Sharded Data Paralleltraining-methods2304.11277Apr 21, 2023~121 min
- FebAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingserving2302.11665Feb 22, 2023~109 min
2022
4- JunTutel: Adaptive Mixture-of-Experts at Scalemoe2206.03382Jun 7, 2022~114 min
- AprOpen Source and In-House: How Uber Optimizes LLM Trainingpretraining2204.02311UberApr 5, 2022~108 min
- MarPathways: Asynchronous Distributed Dataflow for MLdistributed-training2203.12533Mar 23, 2022~108 min
- JanAlpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learningdistributed-training2201.12023Jan 28, 2022~97 min
2021
6- OctColossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingdistributed-training2110.14883Oct 28, 2021~114 min
- MayGSPMD: General and Scalable Parallelization for ML Computation Graphsdistributed-training2105.04663May 10, 2021~138 min
- AprEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMdistributed-training2104.04473Apr 9, 2021~101 min
- AprZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningdistributed-training2104.07857Apr 16, 2021~116 min
- FebTeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelsdistributed-training2102.07988Feb 16, 2021~109 min
- JanZeRO-Offload: Democratizing Billion-Scale Model Trainingtraining-methods2101.06840Jan 18, 2021~102 min
2020
5- AugPollux: Co-Adaptive Cluster Scheduling for Goodput-Optimized Deep Learningdistributed-training2008.12260Aug 27, 2020~116 min
- JunPyTorch Distributed: Experiences on Accelerating Data Parallel Trainingdistributed-training2006.15704Jun 28, 2020~109 min
- JunGShard: Scaling Giant Models with Conditional Computation and Automatic Shardingllm-systems2006.16668Jun 30, 2020~130 min
- JunDeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameterstraining-methods2006.05525Jun 9, 2020~126 min
- FebMiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformersdistributed-training2002.10957Microsoft ResearchFeb 25, 2020~97 min
2019
22018
6- NovGPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelismdistributed-training1811.06965236 citesNov 16, 2018~120 min
- NovMesh-TensorFlow: Deep Learning for Supercomputerstraining-methods1811.0208452 citesNov 5, 2018~105 min
- JulBeyond Data and Model Parallelism for Deep Neural Networksdistributed-training1807.05358Jul 14, 2018~114 min
- JunPipeDream: Fast and Efficient Pipeline Parallel DNN Trainingdistributed-training1806.03377Jun 8, 2018~103 min
- FebHorovod: Fast and Easy Distributed Deep Learning in TensorFlowdistributed-training1802.05799Feb 15, 2018~105 min
- FebIMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturesrl-training1802.01561612 citesFeb 5, 2018~119 min
2017
12016
3- MayTensorFlow: A system for large-scale machine learningllm-systems1605.086958,823 citesMay 27, 2016~115 min
- MarTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemsdistributed-training1603.04467Mar 14, 2016~107 min
- FebCommunication-Efficient Learning of Deep Networks from Decentralized Datallm-systems1602.056295,178 citesFeb 17, 2016~109 min