337 papers
Architecture
0/336Transformers, attention variants, and model design.
Progress0 of 336
2026
49- MaySlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingarchitecture2605.08738May 9, 2026~106 min
- MayQwen-Image-2.0 Technical Reportarchitecture2605.10730QwenMay 11, 2026~115 min
- MayLongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationarchitecture2605.18739NVIDIAMay 18, 2026~131 min
- Mayoptimize_anything: A Universal API for Optimizing any Text Parameterarchitecture2605.19633May 19, 2026~111 min
- MayGated DeltaNet-2: Decoupling Erase and Write in Linear Attentionarchitecture2605.22791NVIDIAMay 21, 2026~109 min
- MayEMO: Pretraining Mixture of Experts for Emergent Modularitymoe2605.06663Allen Institute for AIMay 7, 2026~100 min
- MayDECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devicesmoe2605.10933Tsinghua NLP GroupMay 11, 2026~105 min
- MayRetrieval from Within: An Intrinsic Capability of Attention-Based Modelsretrieval2605.05806NVIDIAMay 7, 2026~119 min
- MayEnd-to-End Autoregressive Image Generation with 1D Semantic Tokenizertraining-methods2605.00503ByteDance SeedMay 1, 2026~115 min
- MayQwen-Image-VAE-2.0 Technical Reportvision2605.13565QwenMay 13, 2026~110 min
- AprUniversal YOCO for Efficient Depth Scalingarchitecture2604.01220Microsoft ResearchApr 1, 2026score 9~105 min
- AprNemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningarchitecture2604.12374NVIDIAApr 14, 2026score 10~121 min
- AprAttention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcontext-optimization2604.10098LongCatApr 11, 2026score 9~127 min
- AprTuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationmultimodal2604.24763Meta AIApr 27, 2026~119 min
- AprMiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactionmultimodal2604.27393OpenBMBApr 30, 2026~120 min
- AprELT: Elastic Looped Transformers for Visual Generationvision2604.09168DeepmindApr 10, 2026score 9~111 min
- MarAttention Residualsarchitecture2603.15031Moonshot AIMar 16, 2026score 9~110 min
- MarMamba-3: Improved Sequence Modeling using State Space Principlesarchitecture2603.15569Together AIMar 16, 2026~121 min
- MarMixture-of-Depths Attentionarchitecture2603.15619ByteDance SeedMar 16, 2026score 9~111 min
- MarPolyglot-Lion: Efficient Multilingual ASR for Singapore via Balanced Fine-Tuning of Qwen3-ASRarchitecture2603.16184Knovel EngineeringMar 17, 2026score 3~116 min
- MarREAD MOREinference-optimization2603.05451Together AIMar 5, 2026~113 min
- MarUniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representationsmultimodal2603.10702Tencent HunyuanMar 11, 2026score 6~110 min
- MarLongCat-Next: Lexicalizing Modalities as Discrete Tokensmultimodal2603.27538Meituan LongCatMar 29, 2026score 9~113 min
- MarHY-WU (Part I): An Extensible Functional Neural Memory Framework and An Instantiation in Text-Guided Image Editingtraining-methods2603.07236Tencent HunyuanMar 7, 2026score 6~135 min
- MarLoGeR: Long-Context Geometric Reconstruction with Hybrid Memoryvision2603.03269DeepmindMar 3, 2026score 9~132 min
- FebFSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Spacearchitecture2602.02092ByteDanceFeb 2, 2026score 9~118 min
- FebHySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingarchitecture2602.03560Xiaomi MiMoFeb 3, 2026score 9~105 min
- FebParallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronizationarchitecture2602.07306AppleFeb 7, 2026~107 min
- FebRecurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoningarchitecture2602.07845Ai2Feb 8, 2026score 6~107 min
- FebMiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modelingarchitecture2602.11761Feb 12, 2026~104 min
- FebUniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Modelarchitecture2602.14178ByteDanceFeb 15, 2026score 9~131 min
- FebTest-Time Training with KV Binding Is Secretly Linear Attentionllm-systems2602.21204NVIDIAFeb 24, 2026score 9~131 min
- FebOmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scalemoe2602.05711Feb 5, 2026~115 min
- FebArcee Trinity Large Technical Reportmoe2602.17004Feb 19, 2026~114 min
- FebLate-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettertraining-methods2602.05393ByteDance SeedFeb 5, 2026score 8~112 min
- FebBitDance: Scaling Autoregressive Generative Models with Binary Tokensuncategorized2602.14041ByteDanceFeb 15, 2026score 9~104 min
- FebVGG-T$^3$: Offline Feed-Forward 3D Reconstruction at Scalevision2602.23361NVIDIAFeb 26, 2026score 2~97 min
- JanDeep Delta Learningarchitecture2601.00417Jan 1, 2026~93 min
- JanRigMo: Unifying Rig and Motion Learning for Generative Animationarchitecture2601.06378SnapJan 10, 2026score 1~115 min
- JanMHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Headarchitecture2601.07832DAGroup-PKUJan 12, 2026score 9~105 min
- JanFlashLabs Chroma 1.0: A Real-Time End-to-End Spoken Dialogue Model with Personalized Voice Cloningarchitecture2601.11141FlashLabsJan 16, 2026score 2~112 min
- JanImplicit Neural Representation Facilitates Unified Universal Vision Encodingarchitecture2601.14256TikTokJan 20, 2026score 2~109 min
- JanQwen3-TTS Technical Reportarchitecture2601.15621Qwen / Alibaba CloudJan 22, 2026score 2~120 min
- JanSAMTok: Representing Any Mask with Two Wordsarchitecture2601.16093ByteDanceJan 22, 2026score 9~122 min
- JanPost-LayerNorm Is Back: Stable, ExpressivE, and Deeparchitecture2601.19895ByteDance SeedJan 27, 2026score 5~108 min
- JanDeepSeek-OCR 2: Visual Causal Flowarchitecture2601.20552DeepSeekJan 28, 2026score 8~118 min
- JanScaling Embeddings Outperforms Scaling Experts in Language Modelsarchitecture2601.21204Meituan LongCatJan 29, 2026score 8~115 min
- JanConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocationarchitecture2601.21420ByteDance SeedJan 29, 2026score 3~103 min
- JanHybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextsarchitecture2601.22156OpenBMBJan 29, 2026score 9~127 min
2025
75- DecBlurDM: A Blur Diffusion Model for Image Deblurringarchitecture2512.03979NVIDIADec 3, 2025score 1~106 min
- DecStronger Normalization-Free Transformersarchitecture2512.10938Dec 11, 2025~114 min
- DecFast-FoundationStereo: Real-Time Zero-Shot Stereo Matchingarchitecture2512.11130NVIDIADec 11, 2025score 4~108 min
- DecNemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningarchitecture2512.20848NVIDIADec 23, 2025~114 min
- DecNVIDIA Nemotron 3: Efficient and Open Intelligencearchitecture2512.20856NVIDIADec 24, 2025~74 min
- DecDynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Spacearchitecture2512.24617ByteDance SeedDec 31, 2025score 9~113 min
- DecmHC: Manifold-Constrained Hyper-Connectionsarchitecture2512.24880DeepSeekDec 31, 2025score 5~125 min
- DecQwen-Image-Layered: Towards Inherent Editability via Layer Decompositiondiffusion2512.15603Qwen / Alibaba CloudDec 17, 2025score 6~110 min
- DecBolmo: Byteifying the Next Generation of Language Modelsllm-systems2512.15586Allen Institute for AIDec 17, 2025~120 min
- DecUnderstanding and Harnessing Sparsity in Unified Multimodal Modelsmoe2512.02351ByteDance SeedDec 2, 2025score 9~121 min
- DecEnd-to-End Test-Time Training for Long Contexttraining-methods2512.23675Dec 29, 2025~120 min
- NovTiDAR: Think in Diffusion, Talk in Autoregressionarchitecture2511.08923NVIDIANov 12, 2025score 9~120 min
- NovVirtual Width Networksarchitecture2511.11238ByteDance SeedNov 14, 2025score 9~112 min
- NovNemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelsarchitecture2511.18890NVIDIANov 24, 2025score 9~106 min
- NovNemotron Elastic: Towards Efficient Many-in-One Reasoning LLMstraining-methods2511.16664NVIDIANov 20, 2025score 9~127 min
- NovHunyuanVideo 1.5 Technical Reportvision2511.18870Tencent HunyuanNov 24, 2025score 5~112 min
- NovPixelDiT: Pixel Diffusion Transformers for Image Generationvision2511.20645NVIDIANov 25, 2025score 9~106 min
- OctLess is More: Recursive Reasoning with Tiny Networksarchitecture2510.04871Samsung AI Lab (SAIL) MontrealOct 6, 2025score 9~116 min
- OctArtificial Hippocampus Networks for Efficient Long-Context Modelingarchitecture2510.07318ByteDance SeedOct 8, 2025score 9~123 min
- OctVLA-0: Building State-of-the-Art VLAs with Zero Modificationarchitecture2510.13054NVIDIAOct 15, 2025score 6~100 min
- OctMoGA: Mixture-of-Groups Attention for End-to-End Long Video Generationarchitecture2510.18692ByteDanceOct 21, 2025score 9~116 min
- OctEvery Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoningarchitecture2510.19338Ant GroupOct 22, 2025score 10~141 min
- OctParallel Loop Transformer for Efficient Test-Time Computation Scalingarchitecture2510.24824ByteDance SeedOct 28, 2025score 9~123 min
- OctScaling Latent Reasoning via Looped Language Modelsarchitecture2510.25741ByteDance SeedOct 29, 2025~96 min
- OctKimi Linear: An Expressive, Efficient Attention Architecturearchitecture2510.26692Moonshot / KimiOct 30, 2025~98 min
- OctHigher-order Linear Attentionarchitecture2510.27258Oct 31, 2025~112 min
- OctOmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMmultimodal2510.15870NVIDIAOct 17, 2025score 10~108 min
- OctRevisiting Multimodal Positional Encoding in Vision-Language Modelsmultimodal2510.23095QwenOct 27, 2025score 10~105 min
- OctHeptapod: Language Modeling on Visual Signalspretraining2510.06673ByteDance SeedOct 8, 2025score 9~128 min
- OctScaling Laws Meet Model Architecture: Toward Inference-Efficient LLMsscaling-laws2510.18245Oct 21, 2025~103 min
- OctThe End of Manual Decoding: Towards Truly End-to-End Language Modelsuncategorized2510.26697Oct 30, 2025~126 min
- SepHunyuan3D-Omni: A Unified Framework for Controllable Generation of 3D Assetsarchitecture2509.21245Tencent HunyuanSep 25, 2025score 3~93 min
- SepSLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attentionarchitecture2509.24006Tsinghua UniversitySep 28, 2025score 10~110 min
- SepInfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationarchitecture2509.24663OpenBMBSep 29, 2025score 9~116 min
- SepSANA-Video: Efficient Video Generation with Block Linear Diffusion Transformerarchitecture2509.24695NVIDIASep 29, 2025score 8~105 min
- AugVibeVoice Technical Reportinference-optimization2508.19205Microsoft ResearchAug 26, 2025~119 min
- AugNVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelreasoning2508.14444NVIDIAAug 20, 2025score 10~119 min
- JulA Systematic Analysis of Hybrid Linear Attentionarchitecture2507.06457Jul 8, 2025score 9~92 min
- JulDecoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationarchitecture2507.06607Microsoft ResearchJul 9, 2025score 9~109 min
- JulStreaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Orderingarchitecture2507.18446NVIDIAJul 24, 2025~117 min
- JulFalcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancearchitecture2507.22448Jul 30, 2025score 10~132 min
- JulLocality-aware Parallel Decoding for Efficient Autoregressive Image Generationinference-optimization2507.01957Jul 2, 2025score 9~111 min
- JulFast and Simplex: 2-Simplicial Attention in Tritonscaling-laws2507.02754Jul 3, 2025score 10~112 min
- JulAXLearn: Modular Large Model Training on Heterogeneous Infrastructuretraining-methods2507.05411AppleJul 7, 2025score 8~94 min
- JunLog-Linear Attentionarchitecture2506.04761Jun 5, 2025~122 min
- JunMultiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generationarchitecture2506.09991Jun 11, 2025score 9~114 min
- JunLanguage Modeling by Language Modelsarchitecture2506.20249Jun 25, 2025~92 min
- JunTransformers Meet In-Context Learning: A Universal Approximation Theoryinference-optimization2506.05200Jun 5, 2025~121 min
- JunMiniCPM4: Ultra-Efficient LLMs on End Devicesllm-systems2506.07900OpenBMBJun 9, 2025score 9~122 min
- JunAccurate and scalable exchange-correlation with deep learningpretraining2506.14665MicrosoftJun 17, 2025~145 min
- MayHunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generationarchitecture2505.04512Tencent HunyuanMay 7, 2025score 3~117 min
- MayMiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoderarchitecture2505.07916May 12, 2025score 2~112 min
- MayCASS: Nvidia to AMD Transpilation with Data, Models, and Benchmarkcode2505.16968NVIDIAMay 22, 2025score 6~112 min
- MayQwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimizationcontext-optimization2505.18092May 23, 2025score 9~115 min
- MayLlama-Nemotron: Efficient Reasoning Modelsreasoning2505.00949DeepSeekMay 2, 2025~96 min
- MayParallel Scaling Law for Language Modelsscaling-laws2505.10475May 15, 2025~87 min
- MayRADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scaletraining-methods2505.03005May 5, 2025score 9~133 min
- AprMulti-Token Attentionarchitecture2504.00927Apr 1, 2025~100 min
- AprBitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMsarchitecture2504.18415Apr 25, 2025score 9~90 min
- AprSmolVLM: Redefining small and efficient multimodal modelsmultimodal2504.05299Hugging Face Smol ModelsApr 7, 2025~116 min
- AprThe Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformermultimodal2504.10462ByteDanceApr 14, 2025score 8~103 min
- AprPixel-SAIL: Single Transformer For Pixel-Grounded Understandingmultimodal2504.10465ByteDanceApr 14, 2025score 9~120 min
- AprMinitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruningtraining-methods2504.11409Apr 15, 2025score 9~124 min
- MarForgetting Transformer: Softmax Attention with a Forget Gatearchitecture2503.02130Mar 3, 2025score 9~118 min
- MarTransformers without Normalizationarchitecture2503.10622Mar 13, 2025~114 min
- MarFlowTok: Flowing Seamlessly Across Text and Image Tokensarchitecture2503.10772ByteDance SeedMar 13, 2025score 9~117 min
- MarRWKV-7 "Goose" with Expressive Dynamic State Evolutionarchitecture2503.14456Mar 18, 2025score 9~123 min
- MarGemma 3 Technical Reportarchitecture2503.19786Google ResearchMar 25, 2025~112 min
- MarQwen2.5-Omni Technical Reportmultimodal2503.20215Qwen / Alibaba CloudMar 26, 2025~106 min
- FebLM2: Large Memory Modelsarchitecture2502.06049Feb 9, 2025~93 min
- FebPoly-Autoregressive Prediction for Modeling Interactionsarchitecture2502.08646DeepMindFeb 12, 2025~104 min
- FebMoBA: Mixture of Block Attention for Long-Context LLMsarchitecture2502.13189Moonshot AIFeb 18, 2025score 9~112 min
- FebTransMLA: Multi-Head Latent Attention Is All You Needinference-optimization2502.07864DeepSeekFeb 11, 2025score 10~101 min
- JanTensor Product Attention Is All You Needarchitecture2501.06425math-aiJan 11, 2025score 9~113 min
- JanSigma: Differential Rescaling of Query, Key and Value for Efficient Language Modelsarchitecture2501.13629Jan 23, 2025score 9~111 min
2024
59- DecFLEX ATTENTION: A PROGRAMMING MODEL FOR GENERATING OPTIMIZED ATTENTION KERNELSarchitecture2412.05496Dec 7, 2024~116 min
- DecMultimodal Latent Language Modeling with Next-Token Diffusionarchitecture2412.08635Microsoft ResearchDec 11, 2024score 6~129 min
- DecByte Latent Transformer: Patches Scale Better Than Tokensarchitecture2412.09871Dec 13, 2024score 9~136 min
- DecFourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationarchitecture2412.17739Dec 23, 2024score 9~109 min
- DecTraining Large Language Models to Reason in a Continuous Latent Spacereasoning2412.06769Dec 9, 2024~126 min
- DecMix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNtraining-methods2412.13795Dec 18, 2024score 8~111 min
- NovSparsing Law: Towards Large Language Models with Greater Activation Sparsityagents2411.02335Nov 4, 2024score 9~108 min
- NovBitNet a4.8: 4-bit Activations for 1-bit LLMsarchitecture2411.04965Nov 7, 2024score 9~92 min
- NovMixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelsarchitecture2411.04996Nov 7, 2024score 10~118 min
- NovWhen Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingarchitecture2411.13476Nov 20, 2024score 8~114 min
- NovHymba: A Hybrid-head Architecture for Small Language Modelsarchitecture2411.13676NVIDIANov 20, 2024score 9~102 min
- NovKV Shifting Attention Enhances Language Modelingarchitecture2411.19574Nov 29, 2024score 9~124 min
- OctSelective Attention Improves Transformerarchitecture2410.02703Oct 3, 2024score 9~123 min
- OctDIFFERENTIAL TRANSFORMERarchitecture2410.05258Oct 7, 2024~99 min
- OctJanus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationmultimodal2410.13848DeepSeekOct 17, 2024score 8~79 min
- OctStuffed Mamba: Oversized States Lead to the Inability to Forgettraining-methods2410.07145Oct 9, 2024score 9~127 min
- SepConfigurable Foundation Models: Building LLMs from a Modular Perspectivearchitecture2409.02877OpenBMBSep 4, 2024score 9~129 min
- SepNVLM: Open Frontier-Class Multimodal LLMsmultimodal2409.11402Sep 17, 2024~107 min
- SepOryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutionmultimodal2409.12961Sep 19, 2024score 9~102 min
- SepAttention Heads of Large Language Models: A Surveyreasoning2409.03752Sep 5, 2024~127 min
- AugDolphin: Long Context as a New Modality for Energy-Efficient On-Device Language Modelsagents2408.15518Aug 28, 2024score 10~87 min
- AugAchieving Human Level Competitive Robot Table Tennisarchitecture2408.03906DeepMindAug 7, 2024score 2~120 min
- AugJamba-1.5: Hybrid Transformer-Mamba Models at Scalearchitecture2408.12570Aug 22, 2024~102 min
- AugThe Mamba in the Llama: Distilling and Accelerating Hybrid Modelsarchitecture2408.15237Aug 27, 2024~116 min
- AugLayerwise Recurrent Router for Mixture-of-Expertsmoe2408.06793Aug 13, 2024score 8~110 min
- AugTransformers are Universal In-context Learnerspretraining2408.01367Aug 2, 2024~122 min
- JulLearning to (Learn at Test Time): RNNs with Expressive Hidden Statesarchitecture2407.04620Jul 5, 2024~108 min
- JulGoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compressionarchitecture2407.12077Jul 16, 2024score 9~124 min
- JulGemma 2: Improving Open Language Models at a Practical Sizepretraining2408.00118Google ResearchJul 31, 2024~106 min
- JunBlock Transformer: Global-to-Local Language Modeling for Fast Inferencearchitecture2406.02657Jun 4, 2024score 9~100 min
- JunSamba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingarchitecture2406.07522Jun 11, 2024score 9~115 min
- MayLayer-Condensed KV Cache for Efficient Inference of Large Language Modelsarchitecture2405.10637May 17, 2024score 9~96 min
- MayReducing Transformer Key-Value Cache Size with Cross-Layer Attentionarchitecture2405.12981May 21, 2024score 9~103 min
- MayTransformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualityarchitecture2405.21060Amazon ScienceMay 31, 2024score 10~123 min
- AprMixture-of-Depths: Dynamically allocating compute in transformer-based language modelsarchitecture2404.02258Apr 2, 2024score 9~101 min
- AprEagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencearchitecture2404.05892Apr 8, 2024score 9~133 min
- AprLeave No Context Behind: Efficient Infinite Context Transformers with Infini-attentionarchitecture2404.07143Apr 10, 2024score 9~113 min
- AprImproving Dictionary Learning with Gated Sparse Autoencodersarchitecture2404.16014DeepMindApr 24, 2024~108 min
- AprKAN: Kolmogorov-Arnold Networksarchitecture2404.19756Apr 30, 2024score 6~117 min
- AprRecurrentGemma: Moving Past Transformers for Efficient Open Language Modelsllm-systems2404.07839Apr 11, 2024score 10~108 min
- MarJamba: A Hybrid Transformer-Mamba Language Modelarchitecture2403.19887AI21Mar 28, 2024~130 min
- MarDiPaCo: Distributed Path Compositiondistributed-training2403.10616DeepMindMar 15, 2024score 9~133 min
- MarThe Unreasonable Ineffectiveness of the Deeper Layerspretraining2403.17887Mar 26, 2024score 9~100 min
- FebMobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casesarchitecture2402.14905Meta AI / FAIRFeb 22, 2024score 10~111 min
- FebSimple linear attention language models balance the recall-throughput tradeoffarchitecture2402.18668Together AIFeb 28, 2024score 9~112 min
- FebGriffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelsarchitecture2402.19427Google ResearchFeb 29, 2024score 9~123 min
- FebThe Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicryllm-systems2402.04347Feb 6, 2024score 9~113 min
- FebLiRank: Industrial Large Scale Ranking Models at LinkedInllm-systems2402.06859Feb 10, 2024score 3~133 min
- FebTraining-Free Long-Context Scaling of Large Language Modelstraining-methods2402.17463Qwen / Alibaba CloudFeb 27, 2024score 9~121 min
- JanGATS: Gather-Attend-Scatterarchitecture2401.08525DeepMindJan 16, 2024~108 min
- JanMambaByte: Token-free Selective State Space Modelarchitecture2401.13660Jan 24, 2024score 9~116 min
- JanSliceGPT: Compress Large Language Models by Deleting Rows and Columnsarchitecture2401.15024Jan 26, 2024score 8~122 min
- JanTransformers are Multi-State RNNsinference-optimization2401.06104Jan 11, 2024score 9~93 min
- JanMoE-Mamba: Efficient Selective State Space Models with Mixture of Expertsmoe2401.04081Jan 8, 2024score 9~97 min
- JanMixtral of Expertsmoe2401.04088Jan 8, 2024~103 min
- JanDeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelsmoe2401.06066DeepSeekJan 11, 2024score 10~107 min
- JanInternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelmultimodal2401.16420InternLM / Shanghai AI LabJan 29, 2024score 9~91 min
- JanLearning Universal Predictorspretraining2401.14953DeepMindJan 26, 2024score 7~105 min
- JanLightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelstraining-methods2401.04658Jan 9, 2024score 9~118 min
2023
37- DecCodestral Mambaarchitecture2312.00752MistralDec 1, 2023~109 min
- DecSwitchHead: Accelerating Transformers with Mixture-of-Experts Attentionmoe2312.07987Dec 13, 2023score 9~109 min
- NovRethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformersarchitecture2311.10642Nov 17, 2023score 9~106 min
- NovMemory Augmented Language Models through Mixture of Word Expertsarchitecture2311.10768Nov 15, 2023score 9~93 min
- NovUltra-Long Sequence Distributed Transformerdistributed-training2311.02382Nov 4, 2023score 9~130 min
- NovFlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Coresinference-optimization2311.05908Together AINov 10, 2023score 9~128 min
- NovMirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalitiesmultimodal2311.05698DeepMindNov 9, 2023score 6~120 min
- OctMonarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecturearchitecture2310.12109Together AIOct 18, 2023~94 min
- OctScalable Neural Network Kernelsarchitecture2310.13225DeepMindOct 20, 2023~110 min
- OctRing Attention with Blockwise Transformers for Near-Infinite Contextllm-systems2310.01889Oct 3, 2023~101 min
- OctBitNet: Scaling 1-bit Transformers for Large Language Modelspretraining2310.11453Oct 17, 2023score 9~106 min
- OctVeRA: Vector-based Random Matrix Adaptationtraining-methods2310.11454Oct 17, 2023score 9~107 min
- SepGated recurrent neural networks discover attentionrl-training2309.01775Sep 4, 2023score 9~157 min
- SepReplacing softmax with ReLU in Vision Transformersvision2309.08586Sep 15, 2023~96 min
- AugFrom Sparse to Soft Mixture of Expertsarchitecture2308.00951DeepMindAug 2, 2023score 9~109 min
- JulLongNet: Scaling Transformers to 1,000,000,000 Tokensarchitecture2307.02486Jul 5, 2023~123 min
- JulRetentive Network: A Successor to Transformer for Large Language Modelsarchitecture2307.08621Jul 17, 2023score 9~130 min
- JulFlashAttention-2: Faster Attention with Better Parallelism and Work Partitioninginference-optimization2307.08691Jul 17, 2023~110 min
- JulFocused Transformer: Contrastive Training for Context Scalingllm-systems2307.03170Jul 6, 2023score 9~116 min
- JulScaling TransNormer to 175 Billion Parametersllm-systems2307.14995Jul 27, 2023score 9~114 min
- JunFaster Causal Attention Over Large Sequences Through Sparse Flash Attentionarchitecture2306.01160Jun 1, 2023score 9~101 min
- JunBlock-State Transformerarchitecture2306.09539Jun 15, 2023score 9~108 min
- JunAugmenting Language Models with Long-Term Memoryllm-systems2306.07174Jun 12, 2023score 9~107 min
- JunQuantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingpretraining2306.12929Jun 22, 2023score 8~107 min
- MayFast Conformer with Linearly Scalable Attention for Efficient Speech Recognitionarchitecture2305.05084NVIDIAMay 8, 2023~102 min
- MayMEGABYTE: Predicting Million-byte Sequences with Multiscale Transformersarchitecture2305.07185May 12, 2023score 9~116 min
- MayRead Morearchitecture2305.13048EleutherAIMay 22, 2023score 9~106 min
- MayGQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointsarchitecture2305.13245May 22, 2023~87 min
- MayRecasting Self-Attention with Holographic Reduced Representationsarchitecture2305.19534EleutherAIMay 31, 2023~118 min
- MayBrainformers: Trading Simplicity for Efficiencyarchitecture2306.00008May 29, 2023score 9~88 min
- MayBlockwise Parallel Transformer for Long Context Large Modelstraining-methods2305.19370May 30, 2023score 8~112 min
- AprEfficient Sequence Transduction by Jointly Predicting Tokens and Durationsarchitecture2304.06795NVIDIAApr 13, 2023~99 min
- AprDynamic Chunk Convolution for Unified Streaming and Non-Streaming Conformer ASRarchitecture2304.09325Apr 18, 2023~96 min
- FebLearning a Fourier Transform for Linear Relative Positional Encodings in Transformersarchitecture2302.01925DeepMindFeb 3, 2023~123 min
- FebHyena Hierarchy: Towards Larger Convolutional Language Modelsarchitecture2302.10866Together AIFeb 21, 2023~120 min
- FebThe geometry of hidden representations of large transformer modelsscaling-laws2302.00294Feb 1, 2023~127 min
- JanTracr: Compiled Transformers as a Laboratory for Interpretabilityarchitecture2301.05062DeepMindJan 12, 2023~123 min
2022
6- DecHungry Hungry Hippos: Towards Language Modeling with State Space Modelsarchitecture2212.14052Together AIDec 28, 2022~118 min
- DecScalable Diffusion Models with Transformersvision2212.09748Dec 19, 2022~103 min
- SepIn-context Learning and Induction Headsuncategorized2209.11895Sep 24, 2022~120 min
- MayREAD MOREinference-optimization2205.14135Together AIMay 27, 2022~127 min
- MarPathways: Asynchronous Distributed Dataflow for MLdistributed-training2203.12533Mar 23, 2022~108 min
- JanConvNeXt: A ConvNet for the 2020svision2201.03545Meta AI / FAIRJan 10, 2022~111 min
2021
14- NovOpenPrompt: An Open-source Framework for Prompt-learningarchitecture2111.01998Nov 3, 2021~102 min
- OctSpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingarchitecture2110.07205Microsoft ResearchOct 14, 2021~105 min
- AugTrain Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationpretraining2108.12409BaichuanAug 27, 2021~104 min
- MayFNet: Mixing Tokens with Fourier Transformsarchitecture2105.03824Google ResearchMay 9, 2021~126 min
- MaySegFormer: Simple and Efficient Design for Semantic Segmentation with Transformersarchitecture2105.15203May 31, 2021~100 min
- MayGSPMD: General and Scalable Parallelization for ML Computation Graphsdistributed-training2105.04663May 10, 2021~138 min
- MayMLP-Mixer: An all-MLP Architecture for Visionvision2105.016012 citesMay 4, 2021~103 min
- AprRoFormer: Enhanced Transformer with Rotary Position Embeddingpretraining2104.09864SnowflakeApr 20, 2021~105 min
- MarCoordinate Attention for Efficient Mobile Network Designvision2103.02907Mar 4, 2021~97 min
- MarSwin Transformer: Hierarchical Vision Transformer Using Shifted Windowsvision2103.14030Microsoft ResearchMar 25, 2021~120 min
- FebNyströmformer: A Nyström-Based Algorithm for Approximating Self-Attentionarchitecture2102.03902Feb 7, 2021~109 min
- FebViLT: Vision-and-Language Transformer Without Convolution or Region Supervisionmultimodal2102.03334Feb 5, 2021~128 min
- FebPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionsvision2102.12122Feb 24, 2021~94 min
- JanTokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetarchitecture2101.11986Jan 28, 2021~113 min
2020
13- DecTraining data-efficient image transformers & distillation through attentionvision2012.12877Meta AI / FAIRDec 23, 2020~106 min
- OctLearning to Execute Programs with Instruction Pointer Attention Graph Neural Networksarchitecture2010.1262110 citesOct 23, 2020~118 min
- OctAn Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.vision2010.11929Apple21,567 citesOct 22, 2020~101 min
- SepDeepETA: How Uber Predicts Arrival Times Using Deep Learningarchitecture2009.14794Uber122 citesSep 30, 2020~124 min
- JulBig Bird: Transformers for Longer Sequencesarchitecture2007.14062Google Research273 citesJul 28, 2020~124 min
- JunDeepETA: How Uber Predicts Arrival Times Using Deep Learningarchitecture2006.04768UberJun 8, 2020~115 min
- MaySynthesizer: Rethinking Self-Attention in Transformer Modelsarchitecture2005.00743198 citesMay 2, 2020~101 min
- MayEnd-to-End Object Detection with Transformersvision2005.12872Meta AI / FAIR831 citesMay 26, 2020~116 min
- AprLongformer: The Long-Document Transformerarchitecture2004.05150Allen Institute for AI2,199 citesApr 10, 2020~103 min
- AprETC: Encoding Long and Structured Inputs in Transformersarchitecture2004.0848328 citesApr 17, 2020~130 min
- AprColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTretrieval2004.12832Apr 27, 2020~118 min
- MarRouting Transformerarchitecture2003.0599749 citesMar 12, 2020~111 min
- JanReformer: The Efficient Transformerarchitecture2001.04451323 citesJan 13, 2020~104 min
2019
11- DecPyTorch: An Imperative Style, High-Performance Deep Learning Libraryarchitecture1912.01703Dec 3, 2019~105 min
- NovCompressive Transformerarchitecture1911.05507Nov 13, 2019~112 min
- SepALBERT: A Lite BERT for Self-supervised Learning of Language Representationsarchitecture1909.11942984 citesSep 26, 2019~104 min
- SepMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismdistributed-training1909.08053Sep 17, 2019~105 min
- AugSentence-BERT: Sentence Embeddings using Siamese BERT-Networksarchitecture1908.10084MozillaAug 27, 2019~110 min
- JunXLNet: Generalized Autoregressive Pretraining for Language Understandingpretraining1906.08237Stitch FixJun 19, 2019~118 min
- MayEfficientNet: Rethinking Model Scaling for Convolutional Neural Networksarchitecture1905.11946Google Research5,013 citesMay 28, 2019~109 min
- MayAdaptive Attention Span in Transformerstraining-methods1905.07799May 19, 2019~109 min
- AprGenerating Long Sequences with Sparse Transformersarchitecture1904.10509Apr 23, 2019~125 min
- FebParameter-Efficient Transfer Learning for NLPtraining-methods1902.00751144 citesFeb 2, 2019~115 min
- JanTransformer-XL: Attentive Language Models Beyond a Fixed-Length Contextarchitecture1901.02860Jan 9, 2019~110 min
2018
9- DecA Style-Based Generator Architecture for Generative Adversarial Networksvision1812.04948Dec 12, 2018~106 min
- NovMesh-TensorFlow: Deep Learning for Supercomputerstraining-methods1811.0208452 citesNov 5, 2018~105 min
- OctHow Powerful Are Graph Neural Networks?architecture1810.00826Oct 1, 2018~119 min
- OctBERT: Pre-training of Deep Bidirectional Transformers for Language Understandingpretraining1810.04805DeepMind45,647 citesOct 11, 2018~131 min
- JunDARTS: Differentiable Architecture Searcharchitecture1806.090551,402 citesJun 24, 2018~133 min
- JunNeural Ordinary Differential Equationsdiffusion1806.07366Jun 19, 2018~115 min
- MarSelf-Attention with Relative Position Representationsarchitecture1803.02155272 citesMar 6, 2018~100 min
- FebSpectral Normalization for Generative Adversarial Networksarchitecture1802.05957Feb 16, 2018~122 min
- FebTVM: An Automated End-to-End Optimizing Compiler for Deep Learninginference-optimization1802.04799Feb 12, 2018~120 min
2017
6- NovNeural Discrete Representation Learningarchitecture1711.009371,969 citesNov 2, 2017~112 min
- OctFraud Detection: Using Relational Graph Learning to Detect Collusionarchitecture1710.10903UberOct 30, 2017~112 min
- SepSqueeze-and-Excitation Networksvision1709.01507Sep 5, 2017~97 min
- JunAttention Is All You Needarchitecture1706.03762NVIDIAJun 12, 2017~118 min
- AprMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationsarchitecture1704.04861Apr 17, 2017~124 min
- JanWasserstein GANtraining-methods1701.07875Jan 26, 2017~99 min
2016
12- DecPointNet: Deep Learning on Point Sets for 3D Classification and Segmentationarchitecture1612.00593Dec 2, 2016~106 min
- DecFeature Pyramid Networks for Object Detectionvision1612.03144Dec 9, 2016~121 min
- NovGenerative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Dataarchitecture1611.01578Uber909 citesNov 5, 2016~96 min
- NovAggregated Residual Transformations for Deep Neural Networksarchitecture1611.05431Nov 16, 2016~113 min
- SepWaveNet: A Generative Model for Raw Audioarchitecture1609.03499Sep 12, 2016~132 min
- AugDensely Connected Convolutional Networksarchitecture1608.06993Aug 25, 2016~106 min
- AugPruning Filters for Efficient ConvNetsarchitecture1608.08710Aug 31, 2016~105 min
- JulLayer Normalizationtraining-methods1607.06450Stability AIJul 21, 2016~98 min
- JulInstance Normalization: The Missing Ingredient for Fast Stylizationvision1607.08022Jul 27, 2016~91 min
- JunGaussian Error Linear Units (GELUs)architecture1606.08415Jun 27, 2016~107 min
- MayTensorFlow: A system for large-scale machine learningllm-systems1605.086958,823 citesMay 27, 2016~115 min
- FebSqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model sizevision1602.07360Feb 24, 2016~121 min
2015
19- DecMXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemsarchitecture1512.01274Dec 3, 2015~114 min
- DecDeep Residual Learning for Image Recognitionvision1512.03385AppleDec 10, 2015~106 min
- NovDueling Network Architectures for Deep Reinforcement Learningarchitecture1511.065811,816 citesNov 20, 2015~96 min
- NovUnsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkspretraining1511.06434Stitch FixNov 19, 2015~111 min
- SepCharacter-level Convolutional Networks for Text Classificationarchitecture1509.01626Sep 4, 2015~104 min
- AugEffective Approaches to Attention-based Neural Machine Translationarchitecture1508.04025Aug 17, 2015~128 min
- JunPointer Networksarchitecture1506.03134Stitch Fix4 citesJun 9, 2015~115 min
- JunFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networksvision1506.014976,265 citesJun 4, 2015~105 min
- JunSpatial Transformer Networksvision1506.0202563 citesJun 5, 2015~122 min
- JunYou Only Look Once: Unified, Real-Time Object Detectionvision1506.02640Jun 8, 2015~97 min
- MayHighway Networksarchitecture1505.00387May 3, 2015~98 min
- MayU-Net: Convolutional Networks for Biomedical Image Segmentationvision1505.04597May 18, 2015~115 min
- AprFast R-CNNvision1504.080831,769 citesApr 30, 2015~120 min
- FebBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftarchitecture1502.03167Dropbox24,373 citesFeb 11, 2015~113 min
- FebImproved Semantic Representations From Tree-Structured Long Short-Term Memory Networksarchitecture1503.00075Feb 28, 2015~128 min
- FebDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationtraining-methods1502.01852Feb 6, 2015~99 min
- FebDRAW: A Recurrent Neural Network For Image Generationvision1502.04623965 citesFeb 16, 2015~121 min
- —Deep learningarchitecture~18 min
- —PyTorch: An Imperative Style, High-Performance Deep Learning Libraryarchitecture~117 min
2014
14- DecEmpirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingarchitecture1412.3555Dec 11, 2014~108 min
- NovFully Convolutional Networks for Semantic Segmentationvision1411.4038Nov 14, 2014~127 min
- OctMemory Networksarchitecture1410.3916Oct 15, 2014~138 min
- OctNeural Turing Machinesarchitecture1410.5401110 citesOct 20, 2014~129 min
- SepNeural Machine Translation by Jointly Learning to Align and Translatealignment1409.0473Sep 1, 2014~101 min
- SepGoing Deeper with Convolutionsarchitecture1409.48421,390 citesSep 17, 2014~113 min
- SepOn the Properties of Neural Machine Translation: Encoder–Decoder Approachesscaling-laws1409.1259Sep 3, 2014~114 min
- SepSequence to Sequence Learning with Neural Networkstraining-methods1409.321513,358 citesSep 10, 2014~112 min
- SepVery Deep Convolutional Networks for Large-Scale Image Recognitionvision1409.1556Sep 4, 2014~129 min
- AugConvolutional Neural Networks for Sentence Classificationarchitecture1408.5882Aug 25, 2014~108 min
- JunLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translationarchitecture1406.1078Jun 3, 2014~119 min
- JunGenerative Adversarial Networksarchitecture1406.2661Google ResearchJun 10, 2014~117 min
- JunCaffe: Convolutional Architecture for Fast Feature Embeddingarchitecture1408.50934,309 citesJun 20, 2014~99 min
- JunSpatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitionvision1406.4729Jun 18, 2014~110 min
2013
4- DecLearning Factored Representations in a Deep Mixture of Expertsarchitecture1312.4314137 citesDec 16, 2013~116 min
- DecAuto-Encoding Variational Bayesarchitecture1312.6114Stitch FixDec 20, 2013~109 min
- DecOverFeat: Integrated Recognition, Localization and Detection using Convolutional Networksvision1312.6229Dec 21, 2013~129 min
- FebMaxout Networksarchitecture1302.4389Feb 18, 2013~131 min
2009
12000
8- —Bigtable: A Distributed Storage System for Structured Dataarchitecture~125 min
- —An Introduction to Conditional Random Fieldsarchitecture~124 min
- —Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterionarchitecture~143 min
- —Mining Frequent Patterns without Candidate Generationarchitecture~99 min
- —The Google File Systemarchitecture~118 min
- —Deep Sparse Rectifier Neural Networksarchitecture~119 min
- —Support Vector Clusteringarchitecture~89 min
- —TensorFlow: A System for Large-Scale Machine Learningarchitecture~115 min