101 papers
Low Precision
0/100FP8, FP4, quantization, and mixed-precision training.
Progress0 of 100
2026
12- MayLongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationarchitecture2605.18739NVIDIAMay 18, 2026~131 min
- MayOSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationinference-optimization2605.17757TogetherMay 18, 2026~114 min
- MayMeasuring Maximum Activations in Open Large Language Modelsserving2605.15572BAIDUMay 15, 2026~113 min
- AprNemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningarchitecture2604.12374NVIDIAApr 14, 2026score 10~121 min
- AprNemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencemultimodal2604.24954NVIDIAApr 27, 2026~116 min
- AprFP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scalingtraining-methods2604.06916NVIDIAApr 8, 2026~116 min
- MarSparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsitylow-precision2603.05168Microsoft ResearchMar 5, 2026score 9~79 min
- MarMASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelsmultimodal2603.04800alibaba-incMar 5, 2026score 9~125 min
- FebSLA2: Sparse-Linear Attention with Learnable Routing and QATdiffusion2602.12675Feb 13, 2026~115 min
- JanFP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learninglow-precision2601.18150NVIDIAJan 26, 2026score 9~102 min
- JanJet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowrl-training2601.14243NVIDIAJan 20, 2026score 9~104 min
- JanQuartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimationtraining-methods2601.22813NVIDIAJan 30, 2026score 9~112 min
2025
32- DecStronger Normalization-Free Transformersarchitecture2512.10938Dec 11, 2025~114 min
- DecNVIDIA Nemotron 3: Efficient and Open Intelligencearchitecture2512.20856NVIDIADec 24, 2025~74 min
- DecFour Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scalinginference-optimization2512.02010Dec 1, 2025~116 min
- DecTurboDiffusion: Accelerating Video Diffusion Models by 100-200 Timesinference-optimization2512.16093University of California, BerkeleyDec 18, 2025score 10~99 min
- DecCUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learningtraining-methods2512.02551NVIDIADec 2, 2025~96 min
- NovWUSH: Near-Optimal Adaptive Transforms for LLM Quantizationalignment2512.00956 IST Austria Distributed Algorithms and Systems LabNov 30, 2025score 9~117 min
- NovNVIDIA Nemotron Nano V2 VLmultimodal2511.03929NVIDIANov 6, 2025score 9~122 min
- NovFP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Errortraining-methods2511.02302Nov 4, 2025~122 min
- OctEvery Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoningarchitecture2510.19338Ant GroupOct 22, 2025score 10~141 min
- OctINT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatsinference-optimization2510.25602NVIDIAOct 29, 2025score 10~113 min
- OctQeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMsrl-training2510.11696NVIDIAOct 13, 2025score 10~124 min
- OctWhy Low-Precision Transformer Training Fails: An Analysis on Flash Attentiontraining-methods2510.04212Tsinghua UniversityOct 5, 2025score 9~125 min
- OctBitNet Distillationtraining-methods2510.13998Microsoft ResearchOct 15, 2025score 9~127 min
- OctDefeating the Training-Inference Mismatch via FP16training-methods2510.26788Sea AI LabOct 30, 2025score 9~127 min
- SepSANA-Video: Efficient Video Generation with Block Linear Diffusion Transformerarchitecture2509.24695NVIDIASep 29, 2025score 8~105 min
- SepDC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Spacediffusion2509.25180NVIDIASep 29, 2025score 8~99 min
- SepBridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationinference-optimization2509.23202NVIDIASep 27, 2025score 9~99 min
- SepLongLive: Real-time Interactive Long Video Generationllm-systems2509.22622NVIDIASep 26, 2025score 6~111 min
- SepPretraining Large Language Models with NVFP4pretraining2509.25149Sep 29, 2025~126 min
- JulThe Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithmllm-systems2507.18553 IST Austria Distributed Algorithms and Systems LabJul 24, 2025score 9~149 min
- MaySageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Traininginference-optimization2505.11594May 16, 2025score 10~110 min
- MaySageAttention2++: A More Efficient Implementation of SageAttention2inference-optimization2505.21136May 27, 2025score 9~93 min
- MayAn Empirical Study of Qwen3 Quantizationlow-precision2505.02214May 4, 2025score 9~89 min
- MayInsights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturesmoe2505.09343NVIDIAMay 14, 2025score 10~111 min
- MayQuartet: Native FP4 Training Can Be Optimal for Large Language Modelstraining-methods2505.14669NVIDIAMay 20, 2025score 9~124 min
- AprBitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMsarchitecture2504.18415Apr 25, 2025score 9~90 min
- AprBitNet b1.58 2B4T Technical Reportllm-systems2504.12285Microsoft ResearchApr 16, 2025score 9~106 min
- MarQuantization for OpenAI's Whisper Models: A Comparative Analysisinference-optimization2503.09905OpenAIMar 12, 2025score 3~112 min
- FebMaximal Brain Damage Without Data or Optimization: Disrupting Neural Networks via Sign-Bit Flipssafety2502.07408NVIDIAFeb 11, 2025~127 min
- FebMatryoshka Quantizationtraining-methods2502.06786Feb 10, 2025score 9~112 min
- JanStreaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchdistributed-training2501.18512Jan 30, 2025score 10~109 min
- JanOptimizing Large Language Model Training Using FP4 Quantizationtraining-methods2501.17116Jan 28, 2025score 9~107 min
2024
21- DecAPOLLO: SGD-like Memory, AdamW-level Performancetraining-methods2412.05270Dec 6, 2024score 9~116 min
- NovBitNet a4.8: 4-bit Activations for 1-bit LLMsarchitecture2411.04965Nov 7, 2024score 9~92 min
- NovWhen Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingarchitecture2411.13476Nov 20, 2024score 8~114 min
- Nov"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationinference-optimization2411.02355DropboxNov 4, 2024score 10~118 min
- NovSageAttention2 Technical Report: Accurate 4 Bit Attention for Plug-and-play Inference Accelerationinference-optimization2411.10958Nov 17, 2024score 9~109 min
- NovMH-MoE: Multi-Head Mixture-of-Expertsmoe2411.16205Nov 25, 2024score 9~107 min
- OctAddition is All You Need for Energy-efficient Language Modelsinference-optimization2410.00907Oct 1, 2024score 9~109 min
- OctSageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationinference-optimization2410.02367Oct 3, 2024score 9~112 min
- OctPrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationinference-optimization2410.05265Oct 7, 2024score 9~117 min
- OctA Survey of Small Language Modelsllm-systems2410.20011Oct 25, 2024~119 min
- OctEoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximationllm-systems2410.21271NVIDIAOct 28, 2024score 6~113 min
- SepMaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelsllm-systems2409.17481Sep 26, 2024score 9~118 min
- SepVPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelslow-precision2409.17066Sep 25, 2024score 9~107 min
- AugJamba-1.5: Hybrid Transformer-Mamba Models at Scalearchitecture2408.12570Aug 22, 2024~102 min
- JulFlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioninference-optimization2407.08608DropboxJul 11, 2024~111 min
- JulSpectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalepretraining2407.12327Jul 17, 2024score 9~98 min
- JunT-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgeinference-optimization2407.00088Jun 25, 2024score 8~124 min
- MarQuaRot: Outlier-Free 4-Bit Inference in Rotated LLMsinference-optimization2404.00456Mar 30, 2024~106 min
- MarGaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectiontraining-methods2403.03507Mar 6, 2024score 9~103 min
- FebThe Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitspretraining2402.17764DropboxFeb 27, 2024~118 min
- JanFP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Designserving2401.14112Jan 25, 2024score 9~96 min
2023
18- DecZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskslow-precision2312.08583Dec 14, 2023score 9~123 min
- OctLLM-FP4: 4-Bit Floating-Point Quantized Transformersinference-optimization2310.16836Oct 25, 2023score 9~104 min
- OctMicroscaling Data Formats for Deep Learninglow-precision2310.10537Oct 16, 2023score 9~124 min
- OctQMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelsmoe2310.16795Oct 25, 2023score 9~111 min
- OctBitNet: Scaling 1-bit Transformers for Large Language Modelspretraining2310.11453Oct 17, 2023score 9~106 min
- OctLoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelstraining-methods2310.08659Oct 12, 2023score 9~110 min
- OctFP8-LM: Training FP8 Large Language Modelstraining-methods2310.18313Oct 27, 2023~108 min
- SepEfficient Post-training Quantization with FP8 Formatsinference-optimization2309.14592Sep 26, 2023score 9~126 min
- AugFLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Searchinference-optimization2308.03290Aug 7, 2023score 8~119 min
- AugOmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelsinference-optimization2308.13137Aug 25, 2023score 9~100 min
- JunAWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Accelerationinference-optimization2306.00978Microsoft ResearchJun 1, 2023~110 min
- JunSqueezeLLM: Dense-and-Sparse Quantizationinference-optimization2306.07629Jun 13, 2023~109 min
- JunBinary and Ternary Natural Language Generationlow-precision2306.01841Jun 2, 2023score 9~115 min
- JunQuantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingpretraining2306.12929Jun 22, 2023score 8~107 min
- JunTraining Transformers with 4-bit Integerstraining-methods2306.11987Jun 21, 2023score 9~99 min
- MayQLoRA: Efficient Finetuning of Quantized LLMstraining-methods2305.14314AI21May 23, 2023score 9~103 min
- MaySingle-Shot Implicit Morphable Faces with Consistent Texture Parameterizationno summary yetcs-cv2305.0304314 citesMay 4, 2023score 1
- MarREAD MOREinference-optimization2303.06865Together AIMar 13, 2023~115 min
2022
4- NovSmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelsinference-optimization2211.10438Nov 18, 2022~96 min
- NovEFFICIENTLY SCALING TRANSFORMER INFERENCEserving2211.05102Nov 9, 2022~147 min
- AugLLM.int8(): 8-bit Matrix Multiplication for Transformers at Scaleinference-optimization2208.07339Aug 15, 2022~109 min
- JunZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformersinference-optimization2206.01861Jun 4, 2022~92 min
2021
12020
12019
32018
12017
22016
4- SepGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationarchitecture1609.08144Sep 26, 2016~114 min
- MarXNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkslow-precision1603.05279Mar 16, 2016~118 min
- FebEIE: Efficient Inference Engine on Compressed Deep Neural Networkarchitecture1602.01528Feb 4, 2016~119 min
- FebBinarized Neural Networkslow-precision1602.02830Feb 9, 2016~111 min