172 papers
Serving
0/172vLLM, SGLang, TensorRT-LLM, and inference servers.
Progress0 of 172
2026
26- MayUniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsificationinference-optimization2605.06221TencentMay 7, 2026~123 min
- MayFull Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Stepsinference-optimization2605.16928RTP-LLMMay 16, 2026~132 min
- MayOSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationinference-optimization2605.17757TogetherMay 18, 2026~114 min
- MayBEAM: Binary Expert Activation Masking for Dynamic Routing in MoEmoe2605.14438alibaba-incMay 14, 2026~112 min
- MayMeasuring Maximum Activations in Open Large Language Modelsserving2605.15572BAIDUMay 15, 2026~113 min
- AprAccelerating Speculative Decoding with Block Diffusion Draft Treesinference-optimization2604.12989Apr 14, 2026~107 min
- AprStochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharinginference-optimization2604.22782AppleApr 3, 2026~101 min
- AprAccelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingrl-training2604.26779NVIDIAApr 29, 2026~133 min
- AprTstars-Tryon 1.0: Robust and Realistic Virtual Try-On for Diverse Fashion Itemsvision2604.19748alibaba-incApr 21, 2026~107 min
- MarPolyglot-Lion: Efficient Multilingual ASR for Singapore via Balanced Fine-Tuning of Qwen3-ASRarchitecture2603.16184Knovel EngineeringMar 17, 2026score 3~116 min
- MarKernel-Smith: A Unified Recipe for Evolutionary Kernel Optimizationcode2603.28342NVIDIAMar 30, 2026~139 min
- MarIndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuseinference-optimization2603.12201DeepSeekMar 12, 2026~129 min
- MarUnderstand and Accelerate Memory Processing Pipeline for Large Language Model Inferenceinference-optimization2603.29002Mar 30, 2026score 9~128 min
- FebParallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronizationarchitecture2602.07306AppleFeb 7, 2026~107 min
- FebFASA: Frequency-aware Sparse Attentioninference-optimization2602.03152Feb 3, 2026~117 min
- FebLycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decodinginference-optimization2602.04541Feb 4, 2026~104 min
- FebDFlash: Block Diffusion for Flash Speculative Decodinginference-optimization2602.06036Feb 5, 2026~116 min
- FebSPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodinginference-optimization2604.09557NVIDIAFeb 10, 2026score 9~89 min
- FebSemantic Search At LinkedInllm-systems2602.07309LinkedInFeb 7, 2026~139 min
- FebScaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgmentsscaling-laws2602.23234AppleFeb 26, 2026~104 min
- FebDualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferenceserving2602.21548DeepSeekFeb 25, 2026score 9~109 min
- JanFlashLabs Chroma 1.0: A Real-Time End-to-End Spoken Dialogue Model with Personalized Voice Cloningarchitecture2601.11141FlashLabsJan 16, 2026score 2~112 min
- JanQwen3-TTS Technical Reportarchitecture2601.15621Qwen / Alibaba CloudJan 22, 2026score 2~120 min
- JanScaling Embeddings Outperforms Scaling Experts in Language Modelsarchitecture2601.21204Meituan LongCatJan 29, 2026score 8~115 min
- JanMegaFlow: Large-Scale Distributed Orchestration System for the Agentic Eraserving2601.07526QwenJan 12, 2026score 6~109 min
- JanQwen3-ASR Technical Reporttraining-methods2601.21337Qwen / Alibaba CloudJan 29, 2026score 2~125 min
2025
44- DecMAI-UI Technical Report: Real-World Centric Foundation GUI Agentsagents2512.22047TongyiLabDec 26, 2025score 4~124 min
- DecKernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Metaagents2512.23236Dec 29, 2025~128 min
- DecFast and Accurate Causal Parallel Decoding using Jacobi Forcinginference-optimization2512.14681Dec 16, 2025score 9~104 min
- DecDEER: Draft with Diffusion, Verify with Autoregressive Modelsinference-optimization2512.15176Dec 17, 2025score 10~125 min
- DecCUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learningtraining-methods2512.02551NVIDIADec 2, 2025~96 min
- NovMixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interactionarchitecture2512.07846Nov 25, 2025~117 min
- NovInferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulationinference-optimization2511.20714DAMO AcademyNov 25, 2025score 9~100 min
- OctParallel Loop Transformer for Efficient Test-Time Computation Scalingarchitecture2510.24824ByteDance SeedOct 28, 2025score 9~123 min
- OctKimi Linear: An Expressive, Efficient Attention Architecturearchitecture2510.26692Moonshot / KimiOct 30, 2025~98 min
- OctPaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelmultimodal2510.14528Baidu ResearchOct 16, 2025~118 min
- OctQwen3Guard Technical Reportsafety2510.14276Qwen / Alibaba CloudOct 16, 2025score 8~136 min
- OctScaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Searchserving2510.22101LinkedInOct 25, 2025~111 min
- OctRDMA Point-to-Point Communication for LLM Systemsserving2510.27656Oct 31, 2025~117 min
- SepAstra: A Multi-Agent System for GPU Kernel Performance Optimizationagents2509.07506Sep 9, 2025~102 min
- SepSet Block Decoding is a Language Model Inference Acceleratorinference-optimization2509.04185Sep 4, 2025score 9~102 min
- SepBridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationinference-optimization2509.23202NVIDIASep 27, 2025score 9~99 min
- SepEmbeddingGemma: Powerful and Lightweight Text Representationsllm-systems2509.20354DeepMindSep 24, 2025~107 min
- SepQwen3-Omni Technical Reportmultimodal2509.17765Sep 22, 2025~129 min
- AugLess Is More: Training-Free Sparse Attention with Global Locality for Efficient Reasoninginference-optimization2508.07101Aug 9, 2025score 9~111 min
- AugTPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferenceserving2508.15881DeepSeekAug 21, 2025score 9~117 min
- AugTaming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inferenceserving2508.19559Aug 27, 2025score 9~115 min
- JulHelix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decodingserving2507.07120Jul 7, 2025~95 min
- JunInfini-gram mini: Exact n-gram Search at the Internet Scale with FM-Indexdata2506.12229Jun 13, 2025score 9~123 min
- JunRectified Sparse Attentioninference-optimization2506.04108Jun 4, 2025score 9~102 min
- JunBeyond the Buzz: A Pragmatic Take on Inference Disaggregationserving2506.05508Jun 5, 2025~100 min
- JunPerformance Prediction for Large Systems via Text-to-Text Regressionserving2506.21718DeepMindJun 26, 2025score 3~109 min
- MayCASS: Nvidia to AMD Transpilation with Data, Models, and Benchmarkcode2505.16968NVIDIAMay 22, 2025score 6~112 min
- MaySageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Traininginference-optimization2505.11594May 16, 2025score 10~110 min
- MaySageAttention2++: A More Efficient Implementation of SageAttention2inference-optimization2505.21136May 27, 2025score 9~93 min
- MayFast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodinginference-optimization2505.22618May 28, 2025score 10~107 min
- MayPrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationsserving2505.07203May 12, 2025~105 min
- AprHogwild! Inference: Parallel LLM Generation via Concurrent Attentioninference-optimization2504.06261Apr 8, 2025score 9~116 min
- AprSleep-time Compute: Beyond Inference Scaling at Test-timeinference-optimization2504.13171Apr 17, 2025~127 min
- AprMegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelismserving2504.02263Apr 3, 2025~102 min
- AprMSCCL++: Rethinking GPU Communication Abstractions for AI Inferenceserving2504.09014Apr 11, 2025~111 min
- MarEAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testinference-optimization2503.01840Mar 3, 2025score 10~123 min
- MarQuantization for OpenAI's Whisper Models: A Comparative Analysisinference-optimization2503.09905OpenAIMar 12, 2025score 3~112 min
- MarQwen2.5-Omni Technical Reportmultimodal2503.20215Qwen / Alibaba CloudMar 26, 2025~106 min
- FebTransMLA: Multi-Head Latent Attention Is All You Needinference-optimization2502.07864DeepSeekFeb 11, 2025score 10~101 min
- FebInfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPUinference-optimization2502.08910Feb 13, 2025score 10~130 min
- FebLServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioninference-optimization2502.14866Feb 20, 2025score 9~120 min
- JanQwen2.5-1M Technical Reportcontext-optimization2501.15383Qwen / Alibaba CloudJan 26, 2025~114 min
- JanReward-Guided Speculative Decoding for Efficient LLM Reasoningreasoning2501.19324SalesforceJan 31, 2025score 9~129 min
- JanFlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingserving2501.01005Jan 2, 2025~136 min
2024
54- DecFLEX ATTENTION: A PROGRAMMING MODEL FOR GENERATING OPTIMIZED ATTENTION KERNELSarchitecture2412.05496Dec 7, 2024~116 min
- DecEfficiently Serving LLM Reasoning Programs with Certaindexserving2412.20993Dec 30, 2024score 9~106 min
- NovContext Parallelism for Scalable Million-Token Inferenceinference-optimization2411.01783Nov 4, 2024~130 min
- NovSageAttention2 Technical Report: Accurate 4 Bit Attention for Plug-and-play Inference Accelerationinference-optimization2411.10958Nov 17, 2024score 9~109 min
- NovStar Attention: Efficient LLM Inference over Long Sequencesinference-optimization2411.17116Nov 26, 2024score 9~124 min
- NovMARCONI: PREFIX CACHING FOR THE ERA OF HYBRID LLMSserving2411.19379Nov 28, 2024~99 min
- OctSageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationinference-optimization2410.02367Oct 3, 2024score 9~112 min
- OctPrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationinference-optimization2410.05265Oct 7, 2024score 9~117 min
- OctDuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headsinference-optimization2410.10819Oct 14, 2024score 9~125 min
- OctSwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationserving2410.03960SnowflakeOct 4, 2024score 9~128 min
- OctOrca: A Distributed Serving System for Transformer-Based Generative Modelsserving2410.17840Oct 23, 2024~108 min
- OctREAD MOREserving2410.20399Together AIOct 27, 2024~108 min
- OctShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferenceserving2410.21465ByteDance SeedOct 28, 2024score 9~114 min
- SepRetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalinference-optimization2409.10516Sep 16, 2024score 8~105 min
- SepVPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelslow-precision2409.17066Sep 25, 2024score 9~107 min
- AugDolphin: Long Context as a New Modality for Energy-Efficient On-Device Language Modelsagents2408.15518Aug 28, 2024score 10~87 min
- AugJamba-1.5: Hybrid Transformer-Mamba Models at Scalearchitecture2408.12570Aug 22, 2024~102 min
- AugNanoFlow: Towards Optimal Large Language Model Serving Throughputserving2408.12757Aug 22, 2024~108 min
- AugLLM Pruning and Distillation in Practice: The Minitron Approachtraining-methods2408.11796Aug 21, 2024~110 min
- JulMInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioninference-optimization2407.02490Qwen / Alibaba CloudJul 2, 2024score 9~128 min
- JulInference Performance Optimization for Large Language Models on CPUsinference-optimization2407.07304Jul 10, 2024score 9~98 min
- JulFlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioninference-optimization2407.08608DropboxJul 11, 2024~111 min
- JulAn Efficient Inference Framework for Early-exit Large Language Modelsinference-optimization2407.20272Jul 25, 2024~103 min
- JulLiNR: Model Based Neural Retrieval on GPUs at LinkedInserving2407.13218Jul 18, 2024~132 min
- JunBlock Transformer: Global-to-Local Language Modeling for Fast Inferencearchitecture2406.02657Jun 4, 2024score 9~100 min
- JunA Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compressioninference-optimization2406.11430Jun 17, 2024score 8~93 min
- JunInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementinference-optimization2406.19707Jun 28, 2024~107 min
- JunT-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgeinference-optimization2407.00088Jun 25, 2024score 8~124 min
- JunLlumnix: Dynamic Scheduling for Large Language Model Servingserving2406.03243Jun 5, 2024~113 min
- JunPowerInfer-2: Fast Large Language Model Inference on a Smartphoneserving2406.06282Jun 10, 2024score 8~108 min
- JunMooncake: A KVCache-centric Disaggregated Architecture for LLM Servingserving2407.00079Jun 24, 2024~127 min
- MayLayer-Condensed KV Cache for Efficient Inference of Large Language Modelsarchitecture2405.10637May 17, 2024score 9~96 min
- MayReducing Transformer Key-Value Cache Size with Cross-Layer Attentionarchitecture2405.12981May 21, 2024score 9~103 min
- MayClover: Regressive Lightweight Speculative Decoding with Sequential Knowledgeinference-optimization2405.00263May 1, 2024score 9~105 min
- MayDistributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inferenceinference-optimization2405.14105May 23, 2024score 9~89 min
- MayNearest Neighbor Speculative Decoding for LLM Generation and Attributionllm-systems2405.19325May 29, 2024score 9~103 min
- MayParrot: Efficient Serving of LLM-based Applications with Semantic Variableserving2405.19888May 30, 2024score 9~98 min
- AprLayerSkip: Enabling Early Exit Inference and Self-Speculative Decodinginference-optimization2404.16710Apr 25, 2024score 9~107 min
- AprKangaroo: Lossless Self-Speculative Decoding via Double Early Exitinginference-optimization2404.18911HUAWEI Noah's Ark LabApr 29, 2024score 9~106 min
- MarRecurrent Drafter for Fast Speculative Decoding in Large Language Modelsllm-systems2403.09919AppleMar 14, 2024~125 min
- MarTaming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serveserving2403.02310Mar 4, 2024~86 min
- FebMobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casesarchitecture2402.14905Meta AI / FAIRFeb 22, 2024score 10~111 min
- FebSimple linear attention language models balance the recall-throughput tradeoffarchitecture2402.18668Together AIFeb 28, 2024score 9~112 min
- FebBreak the Sequential Dependency of LLM Inference Using LOOKAHEAD DECODINGinference-optimization2402.02057Feb 3, 2024~122 min
- FebOuroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodinginference-optimization2402.13720Feb 21, 2024score 9~105 min
- FebLiRank: Industrial Large Scale Ranking Models at LinkedInllm-systems2402.06859Feb 10, 2024score 3~133 min
- FebHydragen: High-Throughput LLM Inference with Shared Prefixesserving2402.05099Feb 7, 2024score 9~105 min
- FebChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitionserving2402.15220Feb 23, 2024score 10~108 min
- JanSliceGPT: Compress Large Language Models by Deleting Rows and Columnsarchitecture2401.15024Jan 26, 2024score 8~122 min
- JanMedusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headsinference-optimization2401.10774Jan 19, 2024score 10~108 min
- JanEAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintyinference-optimization2401.15077Jan 26, 2024~133 min
- JanInference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadsllm-systems2401.11181Jan 20, 2024~115 min
- JanDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingserving2401.09670Jan 18, 2024~102 min
- JanFP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Designserving2401.14112Jan 25, 2024score 9~96 min
2023
28- DecSGLang: Efficient Execution of Structured Language Model Programsagents2312.07104Dec 12, 2023~101 min
- DecSparQ Attention: Bandwidth-Efficient LLM Inferenceinference-optimization2312.04985Dec 8, 2023score 9~132 min
- DecCascade Speculative Drafting for Even Faster LLM Inferenceinference-optimization2312.11462Dec 18, 2023score 9~101 min
- DecLLM in a Flash: Efficient Large Language Model Inference with Limited Memoryinference-optimization2312.11514AppleDec 12, 2023score 10~117 min
- NovFlashDecoding++: Faster Large Language Model Inference on GPUsinference-optimization2311.01282Nov 2, 2023score 9~129 min
- NovPrompt Cache: Modular Attention Reuse for Low-Latency Inferenceinference-optimization2311.04934Nov 7, 2023score 9~118 min
- NovFast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Fasterinference-optimization2311.08263Nov 14, 2023score 9~102 min
- NovRelax: Composable Abstractions for End-to-End Dynamic Machine Learningserving2311.02103Nov 1, 2023~107 min
- NovS-LORA: SERVING THOUSANDS OF CONCURRENT LORA ADAPTERSserving2311.03285Nov 6, 2023~106 min
- OctREAD MOREinference-optimization2310.17157Together AIOct 26, 2023score 9~117 min
- OctQMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelsmoe2310.16795Oct 25, 2023score 9~111 min
- OctPUNICA: MULTI-TENANT LORA SERVINGserving2310.18547Oct 28, 2023~100 min
- SepEFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKSinference-optimization2309.17453Sep 29, 2023~109 min
- SepEfficient Memory Management for Large Language Model Serving with PagedAttentionserving2309.06180Sep 12, 2023score 10~122 min
- AugAccelerating LLM Inference with Staged Speculative Decodinginference-optimization2308.04623Aug 8, 2023~104 min
- AugSARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillsllm-systems2308.16369Aug 31, 2023~105 min
- JulSkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferenceinference-optimization2307.02628Jul 5, 2023score 9~96 min
- JunFaster Causal Attention Over Large Sequences Through Sparse Flash Attentionarchitecture2306.01160Jun 1, 2023score 9~101 min
- JunAWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Accelerationinference-optimization2306.00978Microsoft ResearchJun 1, 2023~110 min
- JunSqueezeLLM: Dense-and-Sparse Quantizationinference-optimization2306.07629Jun 13, 2023~109 min
- JunH_2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelsserving2306.14048Jun 24, 2023score 9~87 min
- MayUnified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systemsarchitecture2305.12102May 20, 2023~102 min
- MayScaling Catalog Attribute Extraction with Multi-modal LLMsllm-systems2305.05176InstacartMay 9, 2023score 8~97 min
- MayCheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIsserving2305.02440May 3, 2023score 9~111 min
- MayFast Distributed Inference Serving for Large Language Modelsserving2305.05920May 10, 2023~112 min
- MaySpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verificationserving2305.09781May 16, 2023~113 min
- MarREAD MOREinference-optimization2303.06865Together AIMar 13, 2023~115 min
- FebAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingserving2302.11665Feb 22, 2023~109 min
2022
7- NovSmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelsinference-optimization2211.10438Nov 18, 2022~96 min
- NovFast Inference from Transformers via Speculative Decodinginference-optimization2211.17192Nov 30, 2022~104 min
- NovEFFICIENTLY SCALING TRANSFORMER INFERENCEserving2211.05102Nov 9, 2022~147 min
- AugLLM.int8(): 8-bit Matrix Multiplication for Transformers at Scaleinference-optimization2208.07339Aug 15, 2022~109 min
- JunZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformersinference-optimization2206.01861Jun 4, 2022~92 min
- JunDeepSpeed-Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scaleserving2207.00032Jun 30, 2022~127 min
- MarPathways: Asynchronous Distributed Dataflow for MLdistributed-training2203.12533Mar 23, 2022~108 min
2020
22019
12018
12017
12016
4- DecClipper: A Low-Latency Online Prediction Serving Systemserving1612.03079Dec 9, 2016~118 min
- MayTensorFlow: A system for large-scale machine learningllm-systems1605.086958,823 citesMay 27, 2016~115 min
- MarTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemsdistributed-training1603.04467Mar 14, 2016~107 min
- FebEIE: Efficient Inference Engine on Compressed Deep Neural Networkarchitecture1602.01528Feb 4, 2016~119 min