124 papers
Context Optimization
0/123KV cache compression, long context, and efficient attention.
Progress0 of 123
2026
20- MayGated DeltaNet-2: Decoupling Erase and Write in Linear Attentionarchitecture2605.22791NVIDIAMay 21, 2026~109 min
- MayContext Training with Active Information Seekingcontext-optimization2605.13050DeepmindMay 13, 2026~123 min
- MayFull Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Stepsinference-optimization2605.16928RTP-LLMMay 16, 2026~132 min
- MayTraining Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Contextmultimodal2605.13831ByteDance SeedMay 13, 2026~98 min
- MayMemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelsmultimodal2605.14906NVIDIAMay 14, 2026~125 min
- MayEndPrompt: Efficient Long-Context Extension via Terminal Anchoringtraining-methods2605.14589BAIDUMay 14, 2026~127 min
- MayWorldKV: Efficient World Memory with World Retrieval and Compressionno summary yetcs-cv2605.22718KAIST AIMay 21, 2026
- AprAttention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcontext-optimization2604.10098LongCatApr 11, 2026score 9~127 min
- AprTriAttention: Efficient Long Reasoning with Trigonometric KV Compressioninference-optimization2604.04921NVIDIAApr 6, 2026~116 min
- AprStochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharinginference-optimization2604.22782AppleApr 3, 2026~101 min
- MarCan Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streamsevaluation2603.07392KAIST AIMar 8, 2026score 9~117 min
- MarScaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspacestraining-methods2603.06713Microsoft ResearchMar 5, 2026score 9~112 min
- FebFASA: Frequency-aware Sparse Attentioninference-optimization2602.03152Feb 3, 2026~117 min
- FebLycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decodinginference-optimization2602.04541Feb 4, 2026~104 min
- FebWhen to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoningllm-systems2602.10560ByteDance SeedFeb 11, 2026score 8~103 min
- FebUntied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunkingllm-systems2602.21196TogetherFeb 24, 2026score 9~105 min
- JanConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocationarchitecture2601.21420ByteDance SeedJan 29, 2026score 3~103 min
- JanHybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextsarchitecture2601.22156OpenBMBJan 29, 2026score 9~127 min
- JanSWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscode2601.16746ByteDanceJan 23, 2026score 8~126 min
- JanFast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioninference-optimization2601.17668NAVER AI LabJan 25, 2026score 9~116 min
2025
38- DecQwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Managementtraining-methods2512.12967Dec 15, 2025~110 min
- DecEnd-to-End Test-Time Training for Long Contexttraining-methods2512.23675Dec 29, 2025~120 min
- DecStoryMem: Multi-shot Long Video Storytelling with Memoryvision2512.19539ByteDanceDec 22, 2025score 3~115 min
- NovSSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Spacetraining-methods2511.20102Nov 25, 2025score 9~129 min
- OctScaling Long-Horizon LLM Agent via Context-Foldingagents2510.11967ByteDance SeedOct 13, 2025score 8~108 min
- OctAgentFold: Long-Horizon Web Agents with Proactive Context Managementagents2510.24699TongyiLabOct 28, 2025score 9~101 min
- OctArtificial Hippocampus Networks for Efficient Long-Context Modelingarchitecture2510.07318ByteDance SeedOct 8, 2025score 9~123 min
- OctHigher-order Linear Attentionarchitecture2510.27258Oct 31, 2025~112 min
- OctLongCodeZip: Compress Long Context for Code Language Modelscode2510.00446Oct 1, 2025~119 min
- OctQuantum simulation of electronic structure via quantum fast multipole methodcontext-optimization2510.07380Google ResearchOct 8, 2025~117 min
- OctAgentic Context Engineering: Evolving Contexts for Self-Improving Language Modelsllm-systems2510.04618Oct 6, 2025score 9~107 min
- OctLoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextsreasoning2510.19363Microsoft ResearchOct 22, 2025score 9~114 min
- OctScaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Searchserving2510.22101LinkedInOct 25, 2025~111 min
- OctScalable In-context Ranking with Generative Modelstraining-methods2510.05396DeepmindOct 6, 2025score 8~107 min
- OctDCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelismtraining-methods2510.10620Oct 12, 2025~99 min
- OctEfficient Long-context Language Model Training by Core Attention Disaggregationtraining-methods2510.18121Oct 20, 2025score 10~105 min
- OctParallelMuse: Agentic Parallel Thinking for Deep Information Seekingtraining-methods2510.24698TongyiLabOct 28, 2025score 9~120 min
- OctDeepSeek-OCR: Contexts Optical Compressionvision2510.18234DeepSeekOct 21, 2025score 9~115 min
- SepMetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactionmultimodal2509.18095Sep 22, 2025score 9~117 min
- JulA Systematic Analysis of Hybrid Linear Attentionarchitecture2507.06457Jul 8, 2025score 9~92 min
- JulZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attentiondistributed-training2507.01004Jul 1, 2025score 9~111 min
- JulGemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiesllm-systems2507.06261Jul 7, 2025score 10~104 min
- JulA Survey of Context Engineering for Large Language Modelsprompting2507.13334Jul 17, 2025~88 min
- JunInference-Time Hyper-Scaling with KV Cache Compressioninference-optimization2506.05345NVIDIAJun 5, 2025score 9~102 min
- JunLongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMsinference-optimization2506.14429Jun 17, 2025score 9~112 min
- MayQwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimizationcontext-optimization2505.18092May 23, 2025score 9~115 min
- MayQwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningrl-training2505.17667May 23, 2025score 9~118 min
- AprMulti-Token Attentionarchitecture2504.00927Apr 1, 2025~100 min
- AprEagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelsmultimodal2504.15271NVIDIAApr 21, 2025score 9~103 min
- MarForgetting Transformer: Softmax Attention with a Forget Gatearchitecture2503.02130Mar 3, 2025score 9~118 min
- MarGemma 3 Technical Reportarchitecture2503.19786Google ResearchMar 25, 2025~112 min
- MarA Comprehensive Survey on Long Context Language Modelingllm-systems2503.17407Mar 20, 2025~121 min
- FebLM2: Large Memory Modelsarchitecture2502.06049Feb 9, 2025~93 min
- FebInfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPUinference-optimization2502.08910Feb 13, 2025score 10~130 min
- FebLServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioninference-optimization2502.14866Feb 20, 2025score 9~120 min
- FebLongRoPE2: Near-Lossless LLM Context Window Scalingscaling-laws2502.20082Feb 27, 2025score 10~92 min
- JanMiniMax-01: Scaling Foundation Models with Lightning Attentionarchitecture2501.08313MiniMaxJan 14, 2025~123 min
- JanQwen2.5-1M Technical Reportcontext-optimization2501.15383Qwen / Alibaba CloudJan 26, 2025~114 min
2024
36- DecFourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationarchitecture2412.17739Dec 23, 2024score 9~109 min
- NovWhen Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingarchitecture2411.13476Nov 20, 2024score 8~114 min
- NovMARCONI: PREFIX CACHING FOR THE ERA OF HYBRID LLMSserving2411.19379Nov 28, 2024~99 min
- OctSelective Attention Improves Transformerarchitecture2410.02703Oct 3, 2024score 9~123 min
- OctDIFFERENTIAL TRANSFORMERarchitecture2410.05258Oct 7, 2024~99 min
- OctDuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headsinference-optimization2410.10819Oct 14, 2024score 9~125 min
- OctInference Scaling for Long-Context Retrieval Augmented Generationreasoning2410.04343Oct 6, 2024~124 min
- OctShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferenceserving2410.21465ByteDance SeedOct 28, 2024score 9~114 min
- OctStuffed Mamba: Oversized States Lead to the Inability to Forgettraining-methods2410.07145Oct 9, 2024score 9~127 min
- SepA Controlled Study on Long Context Extension and Generalization in LLMstraining-methods2409.12181Sep 18, 2024~146 min
- AugDolphin: Long Context as a New Modality for Energy-Efficient On-Device Language Modelsagents2408.15518Aug 28, 2024score 10~87 min
- AugRecent Surge in Public Interest in Transportation: Sentiment Analysis of Baidu Apollo Go Using Weibo Datacontext-optimization2408.10088Aug 19, 2024score 1~117 min
- AugFocusLLM: Precise Understanding of Long Context by Dynamic Condensingcontext-optimization2408.11745Aug 21, 2024score 9~135 min
- AugTransformers are Universal In-context Learnerspretraining2408.01367Aug 2, 2024~122 min
- AugMulti-Layer Transformers Gradient Can be Approximated in Almost Linear Timetraining-methods2408.13233Aug 23, 2024score 9~100 min
- JulLearning to (Learn at Test Time): RNNs with Expressive Hidden Statesarchitecture2407.04620Jul 5, 2024~108 min
- JulGoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compressionarchitecture2407.12077Jul 16, 2024score 9~124 min
- JulHuman-inspired Episodic Memory for Infinite Context LLMsinference-optimization2407.09450Jul 12, 2024score 9~127 min
- JulLazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferenceinference-optimization2407.14057AppleJul 19, 2024score 9~100 min
- JulScaling Granite Code Models to 128K Contexttraining-methods2407.13739IBM ResearchJul 18, 2024score 8~113 min
- JunSamba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingarchitecture2406.07522Jun 11, 2024score 9~115 min
- AprLeave No Context Behind: Efficient Infinite Context Transformers with Infini-attentionarchitecture2404.07143Apr 10, 2024score 9~113 min
- AprExtending Llama-3's Context Ten-Fold Overnightcontext-optimization2404.19553Apr 30, 2024score 9~103 min
- AprRULER: What's the Real Context Size of Your Long-Context Language Models?evaluation2404.06654Qwen / Alibaba CloudApr 9, 2024score 10~109 min
- MarGemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcontext-optimization2403.05530Mar 8, 2024score 10~133 min
- MarLLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncontext-optimization2403.12968Microsoft ResearchMar 19, 2024score 8~113 min
- MarYi: Open Foundation Models by 01.AIscaling-laws2403.0465201-AIMar 7, 2024score 9~73 min
- MarInternLM2 Technical Reporttraining-methods2403.17297InternLM / Shanghai AI LabMar 26, 2024score 9~92 min
- FebA Human-Inspired Reading Agent with Gist Memory of Very Long Contextsagents2402.09727DeepMindFeb 15, 2024score 9~103 min
- FebLongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscontext-optimization2402.13753Feb 21, 2024~93 min
- FebData Engineering for Scaling Language Models to 128K Contextdata2402.10171Feb 15, 2024score 9~112 min
- FebWorld Model on Million-Length Video And Language With Blockwise RingAttentionllm-systems2402.08268Feb 13, 2024score 9~106 min
- FebResonance RoPE: Improving Context Length Generalization of Large Language Modelsreasoning2403.00071Feb 29, 2024score 9~100 min
- FebTraining-Free Long-Context Scaling of Large Language Modelstraining-methods2402.17463Qwen / Alibaba CloudFeb 27, 2024score 9~121 min
- JanInfini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokensdata2401.17377Jan 30, 2024score 10~114 min
- JanSoaring from 4K to 400K: Extending LLM's Context with Activation Beaconllm-systems2401.03462Jan 7, 2024score 9~109 min
2023
18- NovPrompt Cache: Modular Attention Reuse for Low-Latency Inferenceinference-optimization2311.04934Nov 7, 2023score 9~118 min
- OctRing Attention with Blockwise Transformers for Near-Infinite Contextllm-systems2310.01889Oct 3, 2023~101 min
- OctIN-CONTEXT PRETRAINING: LANGUAGE MODELING BEYOND DOCUMENT BOUNDARIESpretraining2310.10638Oct 16, 2023~111 min
- SepDeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelsdistributed-training2309.14509Sep 25, 2023score 8~106 min
- SepEFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKSinference-optimization2309.17453Sep 29, 2023~109 min
- SepLongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelstraining-methods2309.12307Sep 21, 2023score 9~93 min
- SepEffective Long-Context Scaling of Foundation Modelstraining-methods2309.16039Sep 27, 2023score 9~104 min
- AugLM-Infinite: Simple On-the-Fly Length Generalization for Large Language Modelsinference-optimization2308.16137Aug 30, 2023score 9~105 min
- AugYARN: EFFICIENT CONTEXT WINDOW EXTENSION OF LARGE LANGUAGE MODELStraining-methods2309.00071Qwen / Alibaba CloudAug 31, 2023~112 min
- JulLongNet: Scaling Transformers to 1,000,000,000 Tokensarchitecture2307.02486Jul 5, 2023~123 min
- JulLost in the Middle: How Language Models Use Long Contextscontext-optimization2307.03172Together AIJul 6, 2023score 9~101 min
- JulIn-context Autoencoder for Context Compression in a Large Language Modelcontext-optimization2307.06945Jul 13, 2023score 9~107 min
- JulFocused Transformer: Contrastive Training for Context Scalingllm-systems2307.03170Jul 6, 2023score 9~116 min
- JunExtending Context Window of Large Language Models via Positional Interpolationcontext-optimization2306.15595Zhipu / GLMJun 27, 2023score 9~122 min
- JunAugmenting Language Models with Long-Term Memoryllm-systems2306.07174Jun 12, 2023score 9~107 min
- MayMEGABYTE: Predicting Million-byte Sequences with Multiscale Transformersarchitecture2305.07185May 12, 2023score 9~116 min
- MayBlockwise Parallel Transformer for Long Context Large Modelstraining-methods2305.19370May 30, 2023score 8~112 min
- AprRethinking the Role of Token Retrieval in Multi-Vector Retrievalretrieval2304.01982Apr 4, 2023~101 min
2021
22020
5- SepDeepETA: How Uber Predicts Arrival Times Using Deep Learningarchitecture2009.14794Uber122 citesSep 30, 2020~124 min
- JulBig Bird: Transformers for Longer Sequencesarchitecture2007.14062Google Research273 citesJul 28, 2020~124 min
- JunDeepETA: How Uber Predicts Arrival Times Using Deep Learningarchitecture2006.04768UberJun 8, 2020~115 min
- AprLongformer: The Long-Document Transformerarchitecture2004.05150Allen Institute for AI2,199 citesApr 10, 2020~103 min
- MarRouting Transformerarchitecture2003.0599749 citesMar 12, 2020~111 min
2019
4- NovCompressive Transformerarchitecture1911.05507Nov 13, 2019~112 min
- MayAdaptive Attention Span in Transformerstraining-methods1905.07799May 19, 2019~109 min
- AprGenerating Long Sequences with Sparse Transformersarchitecture1904.10509Apr 23, 2019~125 min
- JanTransformer-XL: Attentive Language Models Beyond a Fixed-Length Contextarchitecture1901.02860Jan 9, 2019~110 min