Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

🎯 Pitch

Memory processing operations like retrieval and relevancy scoring consume up to 97% of LLM inference latency, yet they are almost entirely ignored by current GPU-bound optimization efforts. By offloading these irregular, memory-bound stages to an FPGA while keeping dense compute on the GPU, the authors unlock up to 2.2× faster inference and 4.7× energy savings.

1. Executive Summary

This paper empirically analyzes and accelerates the memory processing pipeline that underpins modern long-context LLM inference optimizations, unifying sparse attention, retrieval-augmented generation, and compressed contextual memory under a common four-stage framework—Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference—and profiling these stages across representative methods (DeepSeek Attention, SeerAttention-R, LServe, DRAGIN, MemAgent, and Memory as Context) on AMD MI210 and NVIDIA A100 GPUs. The authors demonstrate that memory processing accounts for 22%–97% of total inference latency, identify strong computational heterogeneity across stages (memory-bound, irregular retrieval versus compute-bound, regular preparation), and propose mapping these heterogeneous operations onto a GPU-FPGA heterogeneous system—offloading sparse, irregular, memory-bounded kernels to an Alveo U55C FPGA while retaining dense compute on the GPU—achieving up to 2.2× end-to-end speedup and 4.7× energy reduction over GPU-only baselines, with speedups for batch inference generally increasing with batch size for sparse attention and RAG methods but degrading for synthesized memory approaches, establishing that heterogeneous acceleration is effective for memory-processing-heavy LLM inference only when the offloaded operations constitute a sufficient fraction of total latency and the interconnect overhead remains small relative to computation time.

2. Context and Motivation

The Core Problem: LLM Inference Optimizations Lack a Unified Computational Understanding

The central gap this paper addresses is the absence of a systematic computational characterization of memory operations across diverse LLM inference optimizations. Modern LLMs increasingly rely on sophisticated memory management techniques—sparse attention (Beltagy et al., 2020; Liu et al., 2025; Yang et al., 2025b; Gao et al., 2025), retrieval-augmented generation (Lewis et al., 2020; Su et al., 2024; Trivedi et al., 2023), compressed contextual memory (Behrouz et al., 2025; Yu et al., 2025; He et al., 2025a), and test-time training (Sun et al., 2024; Zhang et al., 2025b)—to handle long-context tasks like multi-step reasoning (Yao et al., 2023), paper reading, deep reasoning, and creative writing. State-of-the-art models now process and generate 128K to 1M tokens per request (Comanici et al., 2025; Shen et al., 2025; Grattafiori et al., 2024), with KV cache storage for 1M tokens requiring up to 69 GB of GPU memory for large models (Agarwal et al., 2025).

The problem is that prior work treats these optimizations as isolated techniques. Each method—whether DeepSeek's sparse attention indexer (Liu et al., 2025), LServe's hierarchical paged KV cache (Yang et al., 2025b), DRAGIN's dynamic retrieval triggers (Su et al., 2024), or Titans' compressed memory embeddings (Behrouz et al., 2025)—was developed, profiled, and optimized independently. The field lacks a unifying framework for understanding what these methods share computationally, where their bottlenecks live, and whether a single acceleration strategy can benefit them collectively. This is not merely a taxonomic gap: without a common computational abstraction, each new memory optimization requires bespoke performance analysis and hardware mapping, and opportunities for cross-method acceleration remain invisible.

Why This Problem Is Important: Memory Processing Is the Dominant Bottleneck

The practical stakes are high because memory processing is not a minor overhead—it dominates end-to-end latency. The paper's profiling reveals that memory processing accounts for 22%–81% of decoding latency for sparse attention methods at 1M-token sequence lengths (Figure 3), 40%–61% for RAG when processing 20M documents (Figure 4), and up to 97% for synthesized memory methods like MemAgent (Figure 5, left). This grows with memory size: for sparse attention, the fraction rises from 1%–11% at 4K tokens to 22%–81% at 1M tokens (Figure 3). For two-stage RAG, reranker computation dominates memory processing latency, leading to a high percentage even at moderate document counts with slow growth thereafter (Figure 4). For parameterized memory (Titans/HMT, LaCT), memory processing is time-consuming even with short contexts because of the multiple linear projections required to map segment and memory embeddings into shared latent spaces (Figure 5, right). For synthesized memory (MemAgent), the model uses LLM decoding to generate textual memory, pushing memory processing to 97% of total latency (Figure 5, left).

This matters for three reasons, only partially covered in the executive summary:

Scaling trends amplify the bottleneck. As models move toward million-token contexts (Comanici et al., 2025), the memory processing fraction grows monotonically with sequence length (Figures 3, 4). This means that without targeted acceleration of memory operations, longer contexts—which are the direction of the field—will become increasingly bottlenecked on non-attention, memory-management computation.
Energy cost at scale. Production LLM serving incurs substantial energy costs. Since memory processing constitutes a large and growing fraction of latency, it also constitutes a large fraction of energy consumption. Accelerating memory processing thus directly reduces serving cost, which the paper quantifies as 1.11–4.66× geomean energy reduction per request across methods (Table 3).
Hardware underutilization. The computational characteristics of memory processing—irregular access patterns, data-dependent control flow, memory-bounded arithmetic—differ fundamentally from the dense matrix multiplications that dominate the rest of LLM inference (Table 2). GPUs, while excellent at the latter, systematically underutilize their computational resources and off-chip memory bandwidth on the former (Boutros et al., 2020; Song et al., 2022; Rajashekar et al., 2024; He et al., 2024). Thus, even when memory processing's absolute latency is not the majority, it represents wasted capability—GPU hardware that could be doing useful work is instead stalled on memory-bound, irregular operations.

Where Existing Approaches Fall Short

The paper identifies four categories of prior work, each with specific limitations that motivate the need for a unified approach.

Category 1: Method-specific implementations without cross-method analysis. Each LLM inference optimization was developed and profiled in isolation. DeepSeek Attention (Liu et al., 2025) introduces a lightweight indexer for multi-headed latent attention, but its profiling focuses only on DeepSeek-specific operations. SeerAttention-R (Gao et al., 2025) extends an auxiliary predictor to the decoding phase, but its analysis is confined to block-sparse attention mechanisms. LServe (Yang et al., 2025b) introduces hierarchical paged KV caches, but its evaluation considers only its own paging scheme. DRAGIN (Su et al., 2024), FLARE (Jiang et al., 2023), and Fixed-sentence RAG (Trivedi et al., 2023) each focus on their specific retrieval triggering strategies without comparing computational characteristics across RAG variants. MemAgent (Yu et al., 2025) and Titans (Behrouz et al., 2025) analyze their own compressed memory approaches.

The consequence: there is no principled way to answer questions like "Do DeepSeek Attention and DRAGIN share computational bottlenecks?" or "Would an acceleration designed for sparse attention also help RAG?" Each method's computational profile is known only to its developers, and no framework exists for reasoning about them collectively. The paper's four-stage pipeline (Figure 2) directly addresses this gap by providing such a framework.

Category 2: Surveys that classify memory types but not processing structure. Recent surveys (Wu et al., 2025; Zhang et al., 2025a) provide taxonomies of LLM memory—categorizing memory by origin (parametric vs. contextual), representation form (embeddings vs. text), and retention duration—but do not articulate the shared procedural structure by which memory is generated, accessed, and updated. Wu et al. propose a three-dimensional, eight-quadrant taxonomy based on memory origin, representation form, and retention duration, drawing parallels to human cognition. Zhang et al. evaluate memory effectiveness across four categories: parametric memory (model weights), contextual memory (KV caches), external memory (indexed vectors), and procedural memory (event stores). However, these surveys classify what memory is, not how memory is processed during inference. The paper argues that this omission "hinders a systematic understanding of memory processing in LLM inference and limits opportunities for optimizations" (Appendix A). The four-stage pipeline (Prepare Memory → Compute Relevancy → Retrieval → Apply to Inference) fills this gap by providing a processing-centric abstraction rather than a storage-centric taxonomy.

Category 3: FPGA-based LLM accelerators that target specific operations, not the memory processing pipeline. Prior work has demonstrated FPGA acceleration for LLM inference (Zeng et al., 2024; Yang et al., 2024a; He et al., 2025c; Zhang et al., 2026), and for sparse or irregular operations more broadly (He et al., 2024; Rajashekar et al., 2024). FlightLLM (Zeng et al., 2024) provides a complete mapping flow for LLM inference on FPGAs but targets the standard transformer forward pass—not the memory processing operations added by long-context optimizations. GLITCHES (Yang et al., 2024a) explores GPU-FPGA collaborative inference through prefill-decode disaggregation, but again for standard transformer operations, not memory management. LUT-LLM (He et al., 2025c) focuses on efficient FPGA-based LLM decoding with memory-based computations, but without considering the heterogeneous operations introduced by sparse attention, RAG, or compressed memory. FlexLLM (Zhang et al., 2026) provides composable HLS libraries for flexible hybrid LLM accelerators, but as a library infrastructure rather than an analysis of memory processing. These accelerators demonstrate that FPGAs can accelerate LLM inference but do not characterize which emerging long-context operations are most suitable for FPGA offload or provide a general mapping strategy.

Category 4: Computational heterogeneity in LLM inference recognized but not systematically analyzed for memory processing. Prior work (Chen et al., 2024) has analyzed arithmetic intensity and computation patterns in standard LLM inference, noting that attention and feedforward layers exhibit different computational properties. However, this analysis predates or does not cover the diverse memory processing operations introduced by sparse attention indexers, RAG retrieval heuristics (BM25 scoring, embedding search, reranking), compressed memory cross-attention, and test-time training loss computation. The paper extends this heterogeneity analysis into the memory pipeline specifically, quantifying arithmetic intensity (FLOPs/byte) across all four stages for each method type and distinguishing between memory-bound (1–10 FLOPs/byte for Compute Relevancy and Retrieval in sparse attention) and compute-bound (10–100 FLOPs/byte for Prepare Memory in sparse attention) operations (Table 2, Figure 13–14 in Appendix B).

How This Paper Positions Itself Relative to Existing Work

The paper makes three interrelated conceptual moves that position it as a bridge between algorithmic LLM research and hardware acceleration, with the executive summary providing the high-level framing and the detailed analysis providing the empirical foundation:

Move 1: Unify through processing structure, not memory type. Unlike prior taxonomies (Wu et al., 2025; Zhang et al., 2025a) that classify memory by what it stores or represents, this paper classifies by what computation is performed on it. The four-stage pipeline (Prepare Memory, Compute Relevancy, Retrieval, Apply to Inference) is shown to apply across sparse attention (DeepSeek Attention, SeerAttention-R, LServe), RAG (single-stage and two-stage), compressed contextual memory (Titans/HMT, MemAgent), and test-time training (TTT/LaCT), with Table 1 providing a method-by-stage computation mapping. This unification is the paper's primary analytical contribution: it provides a shared vocabulary and framework for discussing and comparing methods that previously appeared unrelated.

Critically, the pipeline abstraction is not merely descriptive—it directly informs hardware mapping decisions. Because the stages exhibit consistent computational properties across methods (Prepare Memory as regular, compute-bound; Compute Relevancy and Retrieval as irregular, memory-bound), a single hardware mapping strategy (GPU for the former, FPGA for the latter) applies across diverse algorithmic instantiations. This is the paper's key bridging insight: the pipeline abstraction reveals that what appears algorithmically diverse is computationally similar at the stage level.

Move 2: Profile holistically rather than method-by-method. The paper conducts systematic profiling across all stages for each method type, measuring both latency breakdown (Figures 3–5) and arithmetic intensity (Figures 13–14 in Appendix B). This profiling reveals that the bottleneck is not uniformly distributed: for sparse attention and RAG, Compute Relevancy and Retrieval dominate (Sections 3.2, Appendix B); for MemAgent, Prepare Memory (LLM decoding) dominates; for Memory as Context, Compute Relevancy and Retrieval are bottlenecks but grow more slowly than in sparse attention; for TTT/LaCT, Prepare Memory and Apply to Inference (forward and backward passes) dominate, leading the paper to exclude TTT from the heterogeneous deployment (Section 4). This profiling-based method selection is a strength: the paper does not blindly apply its heterogeneous mapping to all methods but uses computational profiling to determine where it will help and where it will not.

Move 3: Map to heterogeneous hardware using a principled cost-benefit analysis. The mapping decision is not based on "FPGAs are good for irregular computation" in the abstract, but on a specific two-criteria optimization (Section 5.2): (1) deploy operations to the device best suited to their computational characteristics, but (2) balance this against PCIe communication overhead, prioritizing the latter when it dominates. This leads to non-obvious decisions: for example, extracting KV cache entries for top-k tokens is memory-bound, but the paper does not fuse it with top-k selection on the FPGA because the PCIe overhead to transfer the full KV cache would outweigh the fusion benefit. Instead, only the retrieved indices are returned to the GPU (kilobytes, microseconds of transfer), and KV cache extraction happens locally on the GPU. This principled approach to offloading decisions—quantifying both kernel speedup and communication cost—distinguishes the paper from prior FPGA-based LLM accelerators that offloaded without this systematic cost modeling.

Implicit positioning: building a reusable infrastructure. The paper concludes by stating that the implemented kernels form a reusable library: "Existing methods use provided host binaries, while users can build new algorithms by recombining kernels and interfacing them through our GPU-FPGA communication API" (Section 5.4), with block-based RAG using BM25 and max-reduction kernels given as an example. This positions the work not just as a point demonstration but as a potential platform for future heterogeneous memory processing acceleration, with the pipeline abstraction serving as the interface specification. The paper acknowledges that "arbitrary methods may require custom kernels" and that automation is future work, but the library-based approach represents a practical step toward making heterogeneous acceleration accessible beyond the methods studied in the paper.

Where the Paper Draws Boundaries

The paper is explicit about its scope limitations, which help clarify its positioning:

Hardware scope: The prototype uses an AMD MI210 GPU and Alveo U55C FPGA connected via PCIe. The U55C is fabricated in a 16 nm process (vs. the MI210's 6 nm) and costs half as much (Section 6.1). The paper estimates results on NVIDIA A100 (Appendix H, Figures 25–26) to demonstrate generalizability but acknowledges physical access limitations. More modern FPGA platforms (AMD Versal V80) are mentioned as a path to further improvement (Section 6.1).
Method scope: The paper covers representative methods across sparse attention, RAG, compressed memory, and test-time training, but does not claim exhaustiveness. TTT/LaCT is profiled but not deployed on the heterogeneous system because computational profiling reveals insufficient heterogeneity (the bottleneck is compute-bound forward/backward passes, not memory-bound retrieval). This is a deliberate exclusion that demonstrates the framework's utility for deciding when not to offload.
Kernel coverage: The paper acknowledges that some FPGA kernel designs, particularly for MemAgent and Memory as Context, adopt design paradigms from prior FPGA-based LLM accelerators (FlightLLM, LUT-LLM, GLITCHES) rather than introducing entirely novel architectures. The novel FPGA contribution is primarily in the General Setup kernel (Figure 7) for fused Compute Relevancy + Retrieval in sparse attention and RAG.

The Underlying Thesis

The paper's core thesis—implicit in its structure but articulated through Claims 1–3 (Section 1)—is that the computational properties of LLM memory processing are sufficiently uniform across diverse algorithmic optimizations, and sufficiently heterogeneous from the rest of LLM inference, that a single general-purpose heterogeneous mapping strategy (GPU for dense compute-bound stages, FPGA for irregular memory-bound stages) yields practical end-to-end benefits across a wide range of methods, with the caveat that the offloaded operations must constitute enough of total latency and the interconnect overhead must remain small. This is a systems-level hypothesis about the structure of modern LLM inference, not just a point demonstration on a particular accelerator. The paper sets out to prove this hypothesis through systematic profiling, principled mapping, and cross-method evaluation.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This is a systems measurement and acceleration paper whose core idea is that the diverse memory-management optimizations used in modern LLM inference—sparse attention, retrieval-augmented generation, compressed contextual memory—can be unified into a common four-stage pipeline (Prepare Memory, Compute Relevancy, Retrieval, Apply to Inference), and that because the computational characteristics of these stages are fundamentally heterogeneous (some are compute-bound and regular, others are memory-bound and irregular), a GPU-FPGA heterogeneous system can accelerate them more efficiently than a GPU alone by mapping each stage to the device that best matches its computational profile. The system solves the problem of wasted GPU capability on memory-processing bottlenecks by offloading sparse, irregular, memory-bounded operations to an FPGA while retaining dense, compute-intensive operations on the GPU, with the critical design constraint that the PCIe communication overhead of data transfer between devices must remain small relative to the computation time saved.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five logical components connected by explicit data flows and a decision procedure for mapping:

Profiling and Classification Framework — A measurement infrastructure that decomposes any LLM inference optimization's memory operations into the four pipeline stages, measures per-stage latency fractions and arithmetic intensity (FLOPs/byte), and classifies each stage as compute-bound vs. memory-bound and regular vs. irregular in access pattern. This is used once offline to determine the hardware mapping, not at runtime.
GPU Execution Engine — Runs the dense, compute-bound, regular-access stages of the pipeline: Prepare Memory (linear projections, RoPE embeddings, pooling for compressed keys) and Apply to Inference (sparse attention computation, MLP forward passes, prefilling for synthesized memory). Also runs the non-memory "Rest Ops" of the LLM (feedforward layers, layer normalization, the standard transformer forward pass). Stores the full memory (original KV caches, documents, memory embeddings) in GPU HBM (64 GB, 1.6 TB/s bandwidth on MI210).
FPGA Execution Engine — Runs the fused Compute Relevancy and Retrieval stages (and, for specialized cases, Prepare Memory or Apply to Inference when those are memory-bound and dominate latency). The FPGA kernel (Figure 7) is a streaming dataflow design with three physical memory tiers (BRAM at 21.8 TB/s, URAM at 10.4 TB/s, HBM at 460 GB/s on U55C) and two functional modules: an Inner Product Engine that computes relevance scores between query vectors and key vectors, and a Top-K Retriever that maintains a running top-k list using a parallel reduction tree, outputting only the retrieved indices back to the GPU.
PCIe Communication Layer — A peer-to-peer (P2P) DMA channel between GPU HBM and FPGA HBM via CPU-hosted pinned memory buffers, bypassing system DRAM (Figure 15b). This transfers compressed memory indices from GPU to FPGA (KB-scale, microseconds) and retrieved indices back from FPGA to GPU (KB-scale, microseconds). For specialized mappings (MemAgent, Memory as Context), larger data transfers (embedded vectors, KV caches) are amortized over long computation times.
Mapping Decision Procedure (Section 5.2) — A two-criteria offline policy: Criterion 1 assigns each stage to the device best matching its computational characteristics (GPU for compute-bound/regular, FPGA for memory-bound/irregular); Criterion 2 overrides Criterion 1 when PCIe overhead would exceed kernel-level speedup (e.g., KV cache extraction remains on GPU despite being memory-bound because transferring full KV caches would be too expensive). The procedure also determines dynamic fallback: when sequence length exceeds 1M tokens for LServe or DeepSeek Attention, or when batch size exceeds 2 for MemAgent, the system falls back to GPU-only execution because the FPGA advantage diminishes or reverses.

Information flows as follows for the General Setup (sparse attention and RAG, Figure 6a): LLM forward pass executes on GPU → GPU runs Prepare Memory (generating compressed key vectors) → Compressed keys transferred via PCIe P2P to FPGA HBM → FPGA runs fused Compute Relevancy + Retrieval kernel (computing scores, selecting top-k) → Retrieved indices transferred back to GPU via PCIe → GPU runs Apply to Inference (using retrieved indices to select KV cache entries and compute attention) → GPU continues with Rest Ops of LLM.

For synthesized memory (MemAgent, Figure 6b): GPU runs LLM prefilling on the input segment → KV cache transferred to FPGA via PCIe → FPGA runs LLM decoding (Prepare Memory, generating textual memory) → Token IDs transferred back to GPU → GPU concatenates synthesized memory with next segment and runs Apply to Inference (prefilling).

For Memory as Context (Titans/HMT, Figure 6c): GPU runs model forward pass to generate segment embeddings → Segment embeddings and past memory embeddings streamed to FPGA → FPGA fuses query generation (linear projection on segment embeddings) with cross-attention (scoring against memory embeddings) and selection (top-k/weighted sum) → Retrieved memory embeddings transferred back to GPU → GPU uses retrieved embeddings in Apply to Inference for subsequent segment processing.

3.3 Roadmap for the Deep Dive

First, the memory processing pipeline formalization (Definition 3.1 and the four stages): what each stage computes, how the stages compose, and why this abstraction captures methods as diverse as DeepSeek Attention and MemAgent under a single framework.
Second, the profiling methodology and findings (Section 3.2 and Appendix B): how latency breakdowns and arithmetic intensity are measured, what the numbers concretely are for each method, and how the bottleneck stages are identified—since the profiling results determine which stages are offloaded to the FPGA.
Third, the computational heterogeneity analysis (Section 4, Table 2): the quantitative (arithmetic intensity) and qualitative (access patterns, data dependencies) dimensions that distinguish memory-bound/irregular stages from compute-bound/regular stages, and how this analysis informs hardware mapping decisions.
Fourth, the GPU-FPGA system architecture (Sections 5.1–5.2, Figure 6): the rationale for choosing FPGA over GPU for specific operations, the mapping criteria (Criterion 1: match device to computational profile; Criterion 2: PCIe overhead must be smaller than kernel speedup), and the three deployment configurations (General Setup, Synthesized Memory, Memory as Context).
Fifth, the FPGA kernel design (Section 5.3, Figure 7): the streaming dataflow architecture of the General Setup kernel, the three-level physical memory hierarchy, the Inner Product Engine and Top-K Retriever microarchitectures, and how the design achieves higher effective memory bandwidth than GPU SRAM for the memory-bound stages.
Sixth, the deployment library and reuse strategy (Section 5.4): how individual pipeline stage kernels are composed into reusable modules, how new methods can be constructed by recombining existing kernels, and the explicit boundary drawn between provided infrastructure and future automation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems analysis paper whose core idea is that LLM inference memory optimizations share a common computational pipeline whose heterogeneity can be exploited by a GPU-FPGA system. The paper does not propose a new algorithm; it proposes a new understanding (the four-stage pipeline) and a new execution strategy (heterogeneous mapping based on computational characteristics), validated through systematic profiling and end-to-end hardware measurements.

3.4.1 The Memory Processing Pipeline Formalization

The paper first defines what "memory" means in the context of LLM inference and what operations constitute "memory processing," establishing the abstraction that the rest of the paper depends on.

Definition 3.1 (paraphrased): A generative language model $L$ is decomposed into two functional components: a memory generator $g$ that produces a memory representation $M_{<t}$ from the past input sequence $\{x_i\}_{i<t}$ , and a memory processor $f$ that takes the memory $M_{<t}$ and the current input $x_t$ and produces an intermediate output $O_{<t} = f(M_{<t}, x_t)$ , which is then used by the rest of the model to generate the final output token $y_t$ . The memory generator $g$ runs once to initialize memory; the memory processor $f$ runs repeatedly during inference (once per token in sparse attention, once per RAG trigger event, once per segment in compressed memory).

What this definition accomplishes: It separates the computation that creates memory representations (projections to produce KV caches, embedding models for RAG, text encoding for synthesized memory) from the computation that uses them during inference (scoring relevance, selecting, attending). The paper's focus is exclusively on improving the efficiency of $f$ —the memory processor—because this is the component that all the studied LLM inference optimizations modify. For example, in DeepSeek Attention, $g$ is the Multi-headed Latent Attention (MLA) projections that produce latent KV vectors and indexing vectors, while $f$ is the sparse attention mechanism that computes scores on the indexing vectors, selects top-k, and uses the selected latent KV vectors in attention computation. In RAG, $g$ is the embedding model or tokenizer that produces document indices, while $f$ is the retrieval and context concatenation procedure.

The four-stage pipeline (Figure 2): The paper decomposes the memory processor $f$ into four sequential stages, with the output of each stage feeding into the next:

Stage 1: Prepare Memory — $\text{prep}(M_{<t}) = I_{<t}$ — Converts the raw memory into a memory index $I_{<t}$ that is more efficient to search and access during subsequent stages. This is a preprocessing step that may involve dimensionality reduction, format conversion, or structuring. For DeepSeek Attention, this stage projects the latent KV vectors (which live in a high-dimensional space) into lightweight indexing vectors through linear projections with Rotary Position Embedding (RoPE). For SeerAttention-R, this stage performs average pooling on query and key vectors to produce block-level representations. For LServe, this stage organizes blocks of tokens into logical pages and physical pages, where each logical page is represented by a min vector and a max vector per channel. For RAG with BM25, this stage tokenizes documents and builds term-frequency histograms. For two-stage RAG, this stage runs an embedding model (bge-large-en-v1.5) to produce dense vector representations of documents.

Why this stage exists: Raw memory representations (full KV caches, complete documents) are too large to search through efficiently at every inference step. By compressing or structuring them into a more compact index, the model trades a one-time preprocessing cost for per-step search efficiency. The key design choice across methods is what compression to apply: linear projection + RoPE (DeepSeek), pooling (SeerAttention-R), min/max per block (LServe), tokenization + frequency counts (BM25), or dense embeddings (two-stage RAG).
Stage 2: Compute Relevancy — $\text{comp}(I_{<t}, x_t) = S$ — Given the processed memory index $I_{<t}$ and the current input (query) $x_t$ , produces relevancy scores $S$ where each entry $S_i$ quantifies how relevant the $i$ -th memory entry is to the current query. Higher scores indicate higher relevance. For DeepSeek Attention, this computes multi-headed dot product scores between the query's indexing vectors (64 query heads) and all past key indexing vectors, then applies a weighted average based on query weights derived from the input token (Figure 7, Inner Product Engine). For SeerAttention-R, this computes dot products between projected query vectors and projected key vectors, producing one score per block of 64 tokens. For LServe, this computes the maximum dot product between a query vector and the two channel-extremal vectors (min and max) for each logical page. For RAG with BM25, this computes the BM25 score between the query and each document, using term-frequency and inverse-document-frequency statistics. For two-stage RAG, this stage has two substages: first-stage semantic similarity via inner product with embedding vectors, plus BM25 scores; second-stage reranking where a transformer model (bge-reranker-large) scores each candidate document against the query.

Why this stage exists: It converts the unstructured relationship between the query and the memory into a quantitative signal that can be used for selection. The key design choice is what scoring function to use: inner product (fast, differentiable, parallelizable), BM25 (lexical, interpretable, no training needed), or neural reranker (more accurate, computationally expensive). The scoring function determines the subsequent retrieval heuristics and, critically, the computational properties (arithmetic intensity, memory access pattern) of this stage.
Stage 3: Retrieval — $\text{ret}(M_{<t}, S) = M'_{<t}$ — Given the original memory $M_{<t}$ and the relevancy scores $S$ , produces a retrieved memory subset $M'_{<t}$ by selecting memory entries based on the scores and certain heuristics. This is typically a top-k selection (DeepSeek Attention: select the 2048 tokens with the highest scores; SeerAttention-R top-k mode: select blocks up to a token budget of 4096), threshold-based selection (SeerAttention-R threshold mode: select blocks where score exceeds 5e-4), page-level selection (LServe: select physical pages where the maximum logical page score is highest), or score-weighted combination (Titans/HMT: produce a weighted sum of memory embeddings based on cross-attention scores). For RAG, this stage selects the top-k documents (k=64 for first-stage, k=10 after reranking in two-stage RAG) to append to the query context.

Why this stage exists: It enforces sparsity—the model accesses only a subset of memory entries, not all of them. This is the core efficiency gain of all the studied methods: without retrieval, every inference step would need to process the entire memory (full attention, full document set), which becomes infeasible at million-token scales. The key design choice is what selection heuristic to use: hard top-k (guarantees budget, may miss relevant low-scoring entries), threshold-based (adapts selection count to score distribution, may exceed budget), or soft weighting (retains all information, no sparsity gain). The selection heuristic also determines whether the output is indices (requiring subsequent memory access on the GPU) or weighted embeddings (directly consumable).
Stage 4: Apply to Inference — $\text{apply}(M'_{<t}, x_t) = O_{<t}$ — Integrates the retrieved memory $M'_{<t}$ and the target input $x_t$ into intermediate outputs $O_{<t}$ that are used in subsequent model computations to generate the final token. For DeepSeek Attention, this stage performs the actual MLA computation using only the KV latent embeddings corresponding to the top-k indices. For SeerAttention-R and LServe, this performs block-sparse attention where only the selected blocks of tokens are attended to. For RAG, this stage concatenates the selected documents with the query text to form an augmented input sequence for the generator model. For MemAgent, this stage performs LLM prefilling to consume the synthesized memory and the current segment text. For Titans/HMT, this stage appends the retrieved memory embeddings to the current segment processing.

Why this stage exists: It is the actual productive computation—the reason memory was prepared, scored, and retrieved. The key design choice is how retrieved content is integrated: as attention input (sparse attention), as in-context text (RAG), as latent embeddings (Memory as Context), or as full model context (MemAgent). This determines whether the Apply stage is compute-bound (attention computation, prefilling) or essentially a data movement/concatenation operation (RAG context assembly).

How the pipeline handles missing stages (Section 3.1): Some methods skip stages. MemAgent skips Compute Relevancy (it always retrieves memory from the immediately preceding segment, so no scoring is needed). TTT/LaCT skips Retrieval (it incorporates parameterized memory through a direct forward pass without selection). When a stage is not required, "it introduces no overhead, as data can bypass the stage without additional computation or control complexity" (Section 3.1). This is important for the pipeline abstraction's generality: it does not force unnecessary operations on methods that naturally lack them.

Execution timing and frequency (Section 3.1): The stages are not invoked uniformly. "RAG typically prepares memory once and repeatedly performs retrieval, while sparse attention processes memory for every token." This difference in invocation frequency determines which stages dominate end-to-end latency for each method: in sparse attention, the per-token Compute Relevancy and Retrieval accumulate across all decoding steps, making them the bottleneck; in RAG, Prepare Memory is amortized over many queries, and retrieval is triggered only at specific events (sentence boundaries for Fixed-sentence RAG, confidence drops for FLARE, attention-uncertainty signals for DRAGIN), making retrieval frequency—and thus the relative latency contribution—dependent on the triggering heuristic.

The training-time vs. inference-time distinction (Table 1, "Test-time Training" row): TTT/LaCT uses backpropagation during inference to update model parameters, which constitutes the Compute Relevancy stage (computing the loss on the current input). However, the dominant operations in TTT are the forward and backward passes through the model—the same dense, compute-bound matrix multiplications that characterize standard LLM inference. This is why the paper later excludes TTT from the heterogeneous system: the computational heterogeneity is insufficient (Section 4, "TTT: The heterogeneity is insufficient").

What this abstraction enables that prior work lacked: Prior work (Wu et al., 2025; Zhang et al., 2025a) classified LLM memory by what it stores (parametric, contextual, external, procedural), which is a storage-centric taxonomy. This paper's processing-centric taxonomy classifies by what operations are performed on memory, which directly maps to hardware requirements. A storage classification tells you that DeepSeek Attention and Titans both use "contextual memory"; a processing classification tells you that despite this storage similarity, DeepSeek Attention's bottleneck is in the Compute Relevancy stage (inner products on compressed keys) while Titans' bottleneck is in a fused Compute Relevancy + Retrieval stage (cross-attention on segment and memory embeddings). These map to the same FPGA kernel architecture (the General Setup, with appropriate modifications) despite the algorithmic differences, because at the stage level they share the same computational pattern: memory-bound, irregular-access scoring and selection. The processing abstraction reveals hardware-relevant similarities that a storage abstraction would miss.

3.4.2 Profiling Methodology: Measuring Where Time Goes

The paper's profiling establishes the empirical basis for all subsequent design decisions. The methodology is carefully described across Sections 3.2, 4, and Appendices B-D.

What is measured: For each method in Table 1, the paper measures two things: (1) the fraction of end-to-end latency attributable to each pipeline stage and to memory processing overall, and (2) the arithmetic intensity (FLOPs per byte of memory access, FLOPs/byte) of each stage, which characterizes whether the stage is compute-bound (high arithmetic intensity, limited by arithmetic throughput) or memory-bound (low arithmetic intensity, limited by memory bandwidth).

Measurement infrastructure: Latency is measured with "Python performance counters" for end-to-end request time and with "PyTorch Profiler (Paszke et al., 2019)" for per-kernel time fractions. The paper notes that PyTorch Profiler is used to "derive the latency breakdown without interfered by tracing overhead" (Section 6.1), meaning the breakdown percentages are derived from the profiler's kernel timing data rather than from instrumented code that would perturb the measurement. For the AMD GPU baseline, all methods use optimized implementations: DeepSeek Attention uses a modified vLLM codebase (Kwon et al., 2023) loading only the first layer due to MI210 memory constraints; SeerAttention-R uses TileLang (Wang et al., 2025) optimized kernels on Qwen 3 8B; LServe uses the LServe codebase with HIPIFY (ROCm Organization, 2026) to port CUDA kernels to HIP for the AMD GPU.

Key profiling results that drive design decisions:

Memory processing fraction grows with memory size (Figure 3): For sparse attention, memory processing accounts for 1-11% of decoding latency at 4K tokens, growing to 22-81% at 1M tokens. This monotonic growth justifies the core claim that longer contexts—the direction of the field—will make memory processing an increasingly dominant bottleneck. The variation across methods (22% for some, 81% for others at 1M tokens) reflects differences in how much of the attention computation is considered "memory processing" vs. "rest ops": DeepSeek Attention's indexer is a small fraction of the total MLA computation; LServe's page lookup and score computation are a larger fraction.
RAG memory processing is high even at moderate document counts (Figure 4): At 500K documents, memory processing accounts for 40-61% of request latency for single-stage RAG (DRAGIN, FLARE, FS-RAG). For two-stage RAG, the percentage is high even at 500K (because the reranker is computationally expensive) but grows more slowly with document count (because the reranker processes a fixed number of candidates, not all documents). This informs the mapping decision: the reranker dominates two-stage RAG's GPU execution, so offloading the first-stage retrieval to FPGA can only provide limited end-to-end speedup (1.1-2.1× for memory processing, 1.47-1.84× end-to-end, Figure 10).
Synthesized and parameterized memory are dominated by memory processing even at short contexts (Figure 5): MemAgent spends up to 97% of latency in Prepare Memory (LLM decoding), because generating 1024 tokens of synthesized memory is a full autoregressive generation pass. Titans/HMT spends a substantial fraction in Prepare Memory and Compute Relevancy because the segment projections, memory projections, and cross-attention are extra operations beyond the standard transformer forward pass. TTT/LaCT is dominated by Prepare Memory and Apply to Inference (forward and backward passes), which are compute-bound—hence the decision to not deploy TTT on the heterogeneous system.
Bottleneck stage varies by method type (Appendix B): For sparse attention and RAG, the bottleneck is Compute Relevancy and Retrieval, and "the proportion of latency is increasing as the memory size grows." For MemAgent, "the bottleneck is prepare memory, which is essentially LLM decoding." For Memory as Context, the bottleneck is Compute Relevancy and Retrieval, but "the proportion of latency grows slower than [sparse attention]." For TTT/LaCT, "the bottleneck is prepare memory and apply to inference, which is the LaCT block forward and backward pass." This variation in which stage is the bottleneck directly determines which stage gets offloaded: for sparse attention and RAG, Compute Relevancy + Retrieval is offloaded; for MemAgent, Prepare Memory (decoding) is offloaded; for Memory as Context, a fused Compute Relevancy + Retrieval is offloaded.
Arithmetic intensity quantification (Figures 13-14 in Appendix B, summarized in Table 2): The paper measures arithmetic intensity as orders of magnitude (exact numbers are in Appendix B, Figures 13-14, presented as bar charts with FLOPs/byte on the y-axis). Table 2 reports these as ranges (e.g., "10-100" for Prepare Memory in sparse attention, "1-10" for Compute Relevancy, "1" for Retrieval). The key insight: Prepare Memory and Apply to Inference are typically 1-2 orders of magnitude higher in arithmetic intensity than Compute Relevancy and Retrieval, meaning the former are limited by arithmetic throughput (GPU-friendly) while the latter are limited by memory bandwidth (FPGA-friendly with its customizable memory hierarchy).

How profiling results interact with mapping decisions: The paper uses profiling to make three types of decisions: (a) which methods to accelerate—TTT is excluded because profiling shows insufficient heterogeneity; (b) which stages to offload—Compute Relevancy and Retrieval for sparse attention/RAG, Prepare Memory for MemAgent; and (c) when to fall back to GPU-only—when sequence length exceeds 1M tokens for LServe/DeepSeek Attention, when batch size exceeds 2 for MemAgent. Profiling is the empirical foundation for every mapping claim.

3.4.3 Computational Heterogeneity Analysis

The paper analyzes computational properties along two axes: quantitative (arithmetic intensity) and qualitative (access patterns, data dependencies). Table 2 summarizes these for each method type; Appendix B provides detailed breakdowns (Figures 13-14).

Arithmetic intensity (FLOPs/byte): This is the standard roofline model metric (Williams et al., 2009): the ratio of floating-point operations to bytes of memory traffic. Higher values mean the operation is compute-bound (performance limited by arithmetic throughput); lower values mean it is memory-bound (performance limited by memory bandwidth). The paper reports orders of magnitude rather than exact values in Table 2, with exact measurements in Appendix B.

Prepare Memory: 10-100 FLOPs/byte for sparse attention (linear projections + RoPE are dense matrix multiplications), 1-100 for RAG (varies from tokenization at the low end to embedding model forward passes at the high end), 1-10 for synthesized memory (LLM decoding is fundamentally memory-bound because each weight must be loaded for each generated token, with little arithmetic reuse), >100 for Memory as Context and TTT (dense linear projections and model forward passes).
Compute Relevancy: 1-10 FLOPs/byte across sparse attention, RAG, Memory as Context, and TTT. These are skinny matrix-vector or matrix-matrix multiplications where each weight or key vector is accessed once per dot product, with limited arithmetic reuse—exactly the regime where memory bandwidth is the bottleneck.
Retrieval: ~1 FLOP/byte across all methods. Top-k selection, threshold comparison, max reduction, and BM25 lookup involve very few arithmetic operations per byte of data examined (a few comparisons and conditional moves per score value).
Apply to Inference: 10-100 FLOPs/byte for sparse attention (the actual attention computation is matrix-matrix multiply once key/value vectors are gathered), 0 for RAG (text concatenation is pure data movement with no arithmetic), >100 for synthesized memory and Memory as Context (prefilling and forward passes are compute-bound dense operations).

Access patterns (Table 2):

Regular: Prepare Memory, Apply to Inference, and the "Rest Ops in LLM" (standard transformer forward pass) exhibit consecutive, predictable memory accesses. Linear projections access weight matrices in a strided pattern that hardware prefetchers can anticipate. Attention computation accesses contiguous blocks of KV cache once the sparse indices are resolved.
Irregular: Compute Relevancy and Retrieval for sparse attention and RAG exhibit irregular access patterns. BM25 scoring accesses token-frequency histograms in an order determined by the query tokens, not by document layout in memory. Top-k selection maintains a running heap, where memory accesses depend on the values of incoming scores (an entry is either evicted or retained based on comparison with the current minimum). Embedding search for two-stage RAG accesses dense vectors stored in HBM, but the access order is determined by the query embedding's nearest-neighbor search, which involves irregular memory traversal.

Data dependencies (Table 2):

Local Memory: Operations that are independent across memory entries—each memory entry's score can be computed independently of others. Prepare Memory, Apply to Inference, and Compute Relevancy (in most methods) fall into this category, enabling high parallelism.
Across Memories: Operations that require comparing or aggregating across memory entries—the defining characteristic of the Retrieval stage. Top-k requires comparing each incoming score with the current k-th largest score, creating a data dependency across entries. Max reduction for LServe (finding the maximum score across logical pages) similarly aggregates across entries. Weighted sum for Memory as Context aggregates across memory embeddings. These across-memory dependencies are what make Retrieval difficult to parallelize on GPUs: the reduction tree or priority queue introduces synchronization and irregular data access.

Why this heterogeneity matters for hardware: GPUs are optimized for regular, compute-bound operations with high arithmetic intensity: their massive parallelism (thousands of threads) and deep memory hierarchies (L1, L2, HBM) are designed to hide latency by overlapping computation across many threads, but this works best when all threads follow similar execution paths and access memory in predictable patterns. When operations are irregular (threads take different paths, access different memory locations with no spatial locality), GPUs suffer from warp divergence (threads in a warp serialize on different branches), cache thrashing (irregular accesses evict useful data before it is reused), and low memory bandwidth utilization (individual 32-byte memory transactions are inefficient when many random locations are accessed).

FPGAs, by contrast, can be configured with custom memory hierarchies tailored to specific access patterns. The paper exploits this by building a three-tier memory system (Section 5.3, Figure 7) where key vectors are placed in the memory tier (BRAM, URAM, or HBM) based on how frequently they are accessed—using the FPGA's programmable interconnect to route data directly from the right memory to the compute engines without the fixed cache hierarchy that GPUs impose. This customization is what enables the 5× effective bandwidth advantage for the memory-bound stages.

3.4.4 The GPU-FPGA Heterogeneous System: Rationale and Mapping

The paper's central systems contribution is the mapping of the memory processing pipeline onto a GPU-FPGA system, governed by explicit criteria that balance computational fit against communication cost.

Why FPGA over GPU for specific operations (Section 5.1): The paper identifies three FPGA advantages, all of which are relevant to the memory-bound, irregular stages of the pipeline:

Larger SRAM capacity with higher bandwidth: The U55C FPGA provides BRAM (21.8 TB/s aggregate bandwidth) and URAM (10.4 TB/s) organized as a programmable scratchpad, totaling approximately 40 MB of on-chip storage. This is substantially larger than GPU L1 cache (typically 128 KB-256 KB per SM on MI210) and can be explicitly managed to hold the compressed key vectors for the entire input sequence (up to ~1M keys, each a few hundred bytes, fitting in the ~40 MB when compressed). By explicitly placing frequently accessed data (keys for earlier tokens, which are accessed by every query) in BRAM/URAM, the FPGA can sustain near-peak on-chip bandwidth for the inner product computations, avoiding the cache misses that plague GPU implementations.
Flexible data control with minimized scheduling overhead: FPGAs enable "streaming dataflow designs: executions are data driven, reducing explicit control overhead and time-consuming off-chip memory accesses" (Section 5.3). In a streaming dataflow, each functional unit (key loader, inner product engine, reduction unit, top-k retriever) processes data as it arrives and passes results downstream through FIFO buffers, without the instruction fetch, decode, and thread scheduling overhead that GPUs incur. This is particularly advantageous for the Retrieval stage, where the top-k retriever needs to maintain a running sorted list—an operation that on GPUs requires atomic operations or warp-level reductions with complex synchronization, but on FPGAs can be implemented as a dedicated hardware pipeline stage that accepts one score per cycle and updates the top-k list in constant time.
Low power consumption with competitive performance: The paper measures U55C power at 24.9-44.2 W across kernels, compared to 45-106 W for the MI210 GPU on the same operations (Appendix G). Combined with the kernel-level speedups, this yields the 1.11-4.66× geomean energy reduction reported in Table 3. The energy advantage is largest for MemAgent (4.66×) because LLM decoding is the most memory-bound operation and shows the largest gap between FPGA and GPU power-efficiency in this regime.

The two-criteria mapping policy (Section 5.2):

"Our general mapping criteria for memory processing steps and data are: 1) deploying steps based on the strengths of the FPGA and GPU (i.e., compute-bounded and regular data access on the GPU; irregular, data dependent, and memory-bound operations on the FPGA), and 2) balancing the trade-off between cross-device communication overhead and kernel-level speedup. We prioritize criterion 2 over 1 for minimal end-to-end latency."

This prioritization is critical and distinguishes the paper from naive "offload everything irregular" approaches. Criterion 2 means that even if an operation is memory-bound and irregular (FPGA-friendly), it stays on the GPU if the data it needs would be too expensive to transfer. The specific example given (Section 5.2):

"For example, although extracting the KV cache for top-k tokens is memory bound, we do not fuse it with top-k selection on the FPGA because the PCIe overhead outweighs the fusion benefit. Instead, we transfer only the top-k indices to minimize PCIe latency and perform KV cache extraction on the GPU."

This is a concrete instance of the tradeoff: the PCIe transfer of the full KV cache (which could be gigabytes for long sequences) would take milliseconds, while the FPGA kernel speedup for the extraction operation itself might save only microseconds. By transferring only the indices (kilobytes, ~12 µs for sparse attention per Appendix C.1), the communication cost is negligible, and the GPU performs the memory gathering locally—even though the GPU is less efficient at this irregular gather operation, the total end-to-end time is lower than including the PCIe transfer of full vectors.

The communication cost analysis (Appendix C.1): The paper quantifies PCIe overhead across all methods to justify that Criterion 2 does not prevent offloading in practice:

Sparse Attention: Transfers include new key indexing vectors (from GPU to FPGA, one compressed key vector per new token) and retrieved indices (from FPGA to GPU, top-k token indices). Total transfer: ~12 µs. GPU kernel latency for the corresponding operations: 128-2450 µs. Ratio: computation is 10-200× larger than communication. PCIe overhead is negligible.
RAG: Transfers include only retrieved indices (top-k document indices from FPGA to GPU). Total transfer: ~7 µs. Corresponding GPU kernel latency: 23-1596 ms. Ratio: computation is 3000-200,000× larger. Note the ms vs. µs unit difference—RAG retrieval processes millions of documents, making computation time enormous relative to the tiny index transfer.
Memory-as-Context: Transfers include memory embeddings, query embeddings, and retrieved embeddings for each segment (tens to hundreds of KB). Total transfer: 20-320 µs. Corresponding GPU kernel latency: 26-498 ms. Ratio: computation is 100-1000× larger.
MemAgent: Transfers include KV cache for the segment and token IDs for the synthesized memory (MB-scale). Total transfer: 14-218 ms. Corresponding GPU kernel latency: 17-534 seconds. Ratio: computation is 100-1000× larger.

The paper concludes: "These comparisons show that PCIe communication overhead remains small ( 1000x difference) relative to computation time across all methods, even when data transfer size increases." However, this statement is tempered for MemAgent: at 218 ms of transfer for 534 seconds of computation, the ratio is ~2500×, but the absolute transfer time (218 ms) is non-trivial and could become a bottleneck if the FPGA decoding were significantly faster—the communication-to-computation ratio would shrink.

The three deployment configurations (Figure 6):

Configuration 1: General Setup (Figure 6a) — Used for sparse attention and RAG. The GPU stores the original memory (full KV caches for sparse attention, document corpus for RAG) and runs Prepare Memory (generating compressed key vectors) and Apply to Inference (using retrieved indices to access memory and compute final results). The FPGA stores the processed memory (compressed key indexing vectors for sparse attention, document indices for RAG) and the query vector, and runs a fused Compute Relevancy + Retrieval kernel. The GPU transfers the compressed keys for the entire input sequence during prefilling (one transfer per input) and only the new token's compressed key during decoding (one transfer per token). After retrieval, the FPGA returns only the retrieved indices (not the full memory entries) to the GPU, which then performs local memory access to fetch the actual KV cache entries or documents.

Why fuse Compute Relevancy and Retrieval on the FPGA: These two stages have a producer-consumer relationship: the Retrieval stage consumes scores from the Compute Relevancy stage one at a time (for top-k) or in batches. By fusing them in a streaming dataflow, the FPGA avoids writing intermediate scores to off-chip memory and reading them back—the scores flow directly from the Inner Product Engine through FIFO buffers into the Top-K Retriever, eliminating the memory traffic that would be required on a GPU (where the score tensor would be written to HBM and then read back by a separate top-k kernel). This fusion is a key source of the FPGA's advantage: it exploits the data locality that exists between pipeline stages.

Configuration 2: Synthesized Memory / MemAgent (Figure 6b) — This follows the prefill-decode disaggregation paradigm from prior work (Yang et al., 2024a) but applied specifically to MemAgent's memory processing pipeline. The GPU runs LLM prefilling (Apply to Inference) on the input segment, producing the KV cache for that segment. The KV cache is transferred to the FPGA via PCIe. The FPGA runs LLM decoding (Prepare Memory) to generate the synthesized memory—this is a full autoregressive generation pass producing 1024 tokens of textual memory (per the default MemAgent hyperparameters in Section 6.1). The generated token IDs are transferred back to the GPU. The GPU concatenates the synthesized memory text with the next input segment and runs the next iteration's prefilling.

Why the FPGA runs decoding rather than prefilling: LLM decoding is memory-bound (each weight is loaded once per generated token, with limited reuse across tokens because the computation is sequential—each token depends on all previous tokens). Prefilling is compute-bound (all tokens in the input segment can be processed in parallel, with weight reuse across tokens). The FPGA's advantage is greatest for memory-bound operations (Case 3: Faster Decoding in Appendix F), while the GPU's massive parallelism is better suited for the compute-bound prefilling. This asymmetric mapping—FPGA for decode, GPU for prefill—exploits the differing computational characteristics of the two LLM inference phases.

Configuration 3: Memory as Context / Titans/HMT (Figure 6c) — The kernel mapping follows the General Setup, but with two modifications to the data placement strategy driven by the recurrent loop structure (the model processes input segments sequentially, with each segment accessing memory from all previous segments):

Revised data placement for locality: The retrieved memory embedding is transferred from FPGA to GPU after each segment, but "this only incurs communication overhead comparable to the retrieved index in the General Setup" because the memory embedding is a single vector per segment (similar in size to the retrieved index vector). Meanwhile, the full memory (all past memory embeddings) is stored only on the FPGA—the GPU does not maintain a copy because "GPU-side Prepare Memory only requires the retrieved memory and the next input segment to generate new memory." This saves GPU memory for model storage and reduces redundant data copies.
Kernel fusion on the FPGA: The FPGA fuses query generation (linear projection on the current segment embedding to produce a query vector) with cross-attention (scoring the query against all past memory embeddings) and selection (top-k or weighted sum to produce the retrieved memory embedding). These three operations naturally form a pipeline: query projection produces a vector → cross-attention scores it against memory → selection produces the output embedding. Fusing them avoids intermediate memory reads/writes, with the entire computation occurring in the FPGA's on-chip streaming dataflow.

Dynamic fallback decisions (Section 5.4 and Appendix F): The paper incorporates runtime decisions to avoid performance loss when the FPGA's advantage diminishes:

LServe and DeepSeek Attention beyond 1M tokens: "When the sequence length exceeds 1M tokens, LServe and DeepSeek Attention will experience a drop of speed on the FPGA due to accessing the HBM. Practically, the system can dynamically fall back to GPU-only execution to avoid a performance loss." This occurs because at very long sequences, the compressed keys no longer fit in BRAM+URAM (40 MB total) and must be partially stored in FPGA HBM (460 GB/s), which has lower bandwidth than GPU HBM (1.6 TB/s on MI210). At this crossover point, the FPGA's bandwidth advantage is lost, and the GPU's higher peak HBM bandwidth makes GPU-only execution preferable.
MemAgent with batch size > 2: "the system can dynamically select the optimal configuration. For example, when the batch size is larger than 2 in MemAgent, we switch to a GPU-centric deployment to avoid slowdown." This is because LLM decoding benefits from weight reuse with larger batches—the same weights are used for all batch samples—and GPUs achieve higher throughput for batched matrix-vector operations through thread-level parallelism across batch elements. The FPGA's advantage is greatest at batch size 1, where weight reuse is minimal and memory bandwidth is the sole bottleneck.

These dynamic fallback decisions are not independently profiled or evaluated in the paper; they are stated as design principles. The actual runtime switching mechanism (how the system detects the crossover point and triggers the fallback) is not described.

Why not use an ASIC or a more integrated platform? The paper acknowledges in the Conclusion that "a heterogeneous ASIC could further improve energy efficiency and eliminate cross-device communication overhead." The choice of FPGA is motivated by programmability (the system supports multiple methods by reconfiguring, not by fabricating new chips) and availability (off-the-shelf devices for demonstration). However, the paper also notes that the U55C is "fabricated in an older process technology than the GPU (16 nm vs. 6 nm) and costs half as much" (Section 6.1), and that "a newer FPGA model (e.g., AMD Versal V80) can further improve performance"—estimated at 1.6× additional geomean speedup (Appendix H.1). This positions the current results as a lower bound on what heterogeneous acceleration can achieve.

3.4.5 FPGA Kernel Design: The General Setup Kernel for Fused Compute Relevancy + Retrieval

The General Setup FPGA kernel (Figure 7) is the most detailed architectural contribution in the paper. It implements the fused Compute Relevancy and Retrieval stages for sparse attention and RAG methods. The paper uses DeepSeek Attention as the running example but notes that the same architecture applies to other methods with appropriate modifications to the scoring function and retrieval heuristic.

Kernel architecture overview: The kernel is a streaming dataflow design where functional modules are connected by FIFO channels, and execution is data-driven: each module processes data as it arrives from upstream and produces output for downstream. The paper contrasts this with GPU execution: "One advantage of FPGAs over GPUs is that users can build streaming dataflow designs: executions are data driven, reducing explicit control overhead and time-consuming off-chip memory accesses."

Three-level physical memory hierarchy (Figure 7, left side):

The key vectors for all past tokens are stored across three physical memory tiers, organized by access frequency:

BRAM (Block RAM): 21.8 TB/s aggregate bandwidth, used for the most frequently accessed key vectors. Specifically, "Key vectors with smaller token IDs are stored in faster memory to maintain high access speed." This is because earlier tokens have smaller token IDs and are accessed by every subsequent query during autoregressive decoding, accumulating vastly more accesses than later tokens. The paper allocates "token 0-16383" to BRAM (Figure 7): the first 16,384 tokens.
URAM (Ultra RAM): 10.4 TB/s aggregate bandwidth, used for moderately-accessed key vectors. The paper allocates "token 16384-65535" to URAM: tokens 16,384 through 65,535. These tokens are accessed by all queries from positions 16,384 onward, so they are accessed less frequently than the BRAM-stored tokens but more frequently than the tail.
HBM (High Bandwidth Memory): 460 GB/s bandwidth (significantly lower than on-chip SRAM), used for the least-accessed key vectors. The paper allocates "token >= 65536" to HBM. These tail tokens are accessed only by queries from positions beyond 65,535, which occur only for very long sequences.

The total on-chip memory (BRAM + URAM) is approximately 40 MB on the U55C. This can store compressed key vectors for up to 65,536 tokens (assuming each compressed key is ~600 bytes, typical for 64-head indexing vectors with partial RoPE). For sequences beyond 65,536 tokens, the overflow goes to HBM.

Why this tiered approach: It exploits the temporal locality inherent in autoregressive decoding. Earlier tokens are accessed more often because they are seen by all subsequent queries; later tokens are seen by fewer queries. By placing frequently-accessed data in the fastest memory, the FPGA maximizes the average bandwidth during the inner product computation—most of the computation time is spent streaming keys from BRAM/URAM through the inner product engine, with only the tail of the sequence incurring the lower HBM bandwidth. This is a form of explicit data placement that is not possible on GPUs, where the cache hierarchy (L1, L2) manages placement automatically based on access patterns, without knowledge of the monotonic relationship between token position and access frequency.

Write arbiter and key loader: The key loader is responsible for reading key vectors from their assigned memory tier and streaming them into the Compute Relevancy module. A write arbiter manages the initial placement of keys into the memory hierarchy when they are received from the GPU: keys for tokens 0-16,383 are written to BRAM, tokens 16,384-65,535 to URAM, and tokens ≥65,536 to HBM. The read arbiter (shown in Figure 7 as "Read Arbiter") coordinates reads from the three memory tiers, ensuring that the inner product engine receives a continuous stream of key vectors in token-ID order. Since the three memory tiers have different latencies and bandwidths, the arbiter must prefetch from HBM early enough to avoid stalls when the streaming transitions from URAM to HBM at token 65,536.

Compute Relevancy module: Inner Product Engine (Figure 7, center):

The Inner Product Engine computes the relevance scores between the query vectors and all key vectors. For DeepSeek Attention specifically, the query consists of 64 query heads (each a vector of the same dimension as the compressed key vector), and the computation is:

For each query head $h \in \{1, \ldots, 64\}$ and each key vector $k_j$ (for token $j$ ): $$s_{h,j} = q_h \cdot k_j$$ where $q_h$ is the query vector for head $h$ , $k_j$ is the key indexing vector for token $j$ , and $s_{h,j}$ is the scalar relevance score for head $h$ on token $j$ .

The engine streams key vectors from the memory hierarchy and broadcasts them to 64 parallel dot-product units (one per query head). The query vectors are loaded once from the FPGA's on-chip buffer (loaded from GPU via PCIe at the start of each decoding step) and held stationary in the dot-product units while all key vectors stream through. This stationary-query design maximizes the reuse of query vectors: each query vector is used for all $N$ key vectors (where $N$ is the sequence length), so keeping it in local registers reduces memory traffic by a factor of $N$ compared to reloading it for each key.

What it computes: For each token $j$ in the sequence, the engine produces 64 scalar scores $s_{1,j}, s_{2,j}, \ldots, s_{64,j}$ , one per query head. These 64 scores represent how relevant token $j$ is to each of the 64 query perspectives.

Why this form: Multi-headed scoring is necessary because DeepSeek Attention uses a weighted average of head scores to produce the final per-token score (the reduction step in Retrieval). Computing all 64 dot products in parallel avoids sequentializing the computation across heads, which would require re-streaming the key vectors 64 times—a 64× increase in memory bandwidth demand that would saturate even the FPGA's on-chip bandwidth. By keeping query vectors stationary and broadcasting key vectors to all 64 units, the engine achieves the parallelism without proportional bandwidth increase.

Retrieval module: Reduction + Top-K (Figure 7, right side):

The Retrieval module receives the 64 per-head scores for each token and produces the final top-k token indices. It consists of two chained sub-modules:

Reduction unit: For DeepSeek Attention, the 64 per-head scores are combined into a single per-token score through a weighted sum:

$$\text{score}_j = \sum_{h=1}^{64} w_h \cdot s_{h,j}$$

where $w_h$ are the query weights derived from the input token (these are part of the DeepSeek Attention mechanism and are included in the query data transferred from the GPU), and $s_{h,j}$ are the per-head dot product scores from the Inner Product Engine. For other sparse attention methods, the reduction may be simpler: SeerAttention-R uses a single score per block (no multi-head weighting), so the reduction unit becomes a pass-through or a simple max operation. LServe requires a max reduction across query heads before page-level score computation.

What it computes: A single scalar $\text{score}_j$ per token $j$ , representing the overall relevance of that token to the current query.

Why weighted sum: The weighting allows the attention mechanism to assign different importance to different heads for the current token—some heads may be more relevant than others depending on the token's position and content. An unweighted average would treat all heads equally and lose this head-importance signal.

Top-K Retriever: This sub-module maintains a running list of the top-k scores and their corresponding token indices, where $k = 2048$ for DeepSeek Attention (per the DeepSeek V3.2 Exp configuration, Section 6.1, Appendix D), $k$ equals the token budget (4096 for SeerAttention-R top-k mode), or $k$ equals the number of documents to retrieve (64 for single-stage RAG, 10 for two-stage RAG). The retriever processes one $(\text{score}_j, j)$ pair per cycle (as it streams from the Reduction unit) and updates its internal top-k state.

The implementation uses a parallel reduction tree (Figure 7, "Top-K Score/Index" block with > comparators): the incoming score is compared against the current minimum score in the top-k list using a comparator tree. If the new score exceeds the current minimum, it replaces the minimum entry, and the new minimum is found by a parallel reduction across all k entries. This processes one token per cycle in steady state, regardless of $k$ , because the comparison and update logic is fully pipelined.

What it produces: After all $N$ token scores have been processed, the retriever outputs the token indices of the $k$ highest-scoring tokens. These indices (an array of $k$ integers, ~8 KB for $k=2048$ at 4 bytes per index) are transferred back to the GPU via PCIe.

Why a parallel reduction tree rather than a heap: A heap-based top-k (common in CPU/GPU implementations) has O(log k) complexity per insertion because the heap property must be maintained. For $k=2048$ , log k ≈ 11 operations per token, which can become a bottleneck at high throughput. The parallel reduction tree, by contrast, uses hardware parallelism to perform all k comparisons in a single cycle (through a tree of comparators), accepting lower hardware utilization (most comparators are idle for tokens that don't make the cut) in exchange for deterministic, pipelineable, single-cycle throughput. This trades area for throughput—a tradeoff that is feasible on FPGAs where logic resources are abundant relative to the required computational throughput.

FIFO channels and backpressure: The dataflow between modules is managed through FIFO buffers (shown as arrows between modules in Figure 7). If the Top-K Retriever processes scores slower than the Inner Product Engine produces them (due to occasional stalls from the reduction tree on k-large updates), backpressure propagates upstream through the FIFOs, stalling the key loader and Inner Product Engine. This is the standard streaming dataflow control mechanism: no explicit scheduling, no thread synchronization, no deadlock risk—the hardware stalling logic handles flow control automatically.

How this design achieves >5× effective bandwidth for memory-bound stages (Appendix F, Case 1): The key is that for block-sparse attention methods (SeerAttention-R, LServe), the compressed key vectors fit entirely in on-chip memory (BRAM + URAM) for sequences up to ~65K tokens. The Inner Product Engine streams these keys at the aggregate on-chip bandwidth, which is ~32.2 TB/s (21.8 + 10.4). The GPU equivalent—streaming keys from L1/L2 cache—is limited by the effective SRAM bandwidth, which the paper estimates as ~5× lower. This is because GPU caches are designed for reuse (same address accessed multiple times by different threads) and write-back policies, not for streaming reads where each address is accessed exactly once. The FPGA's scratchpad memory, by contrast, is explicitly managed for streaming access: keys are read sequentially from BRAM/URAM, with no tag checks, no write-back, and no associativity conflicts—the hardware knows exactly which address to read next because the key loader controls the address sequence.

Why this advantage diminishes beyond 1M tokens (Appendix F): At sequence lengths beyond approximately 65K tokens, the compressed keys no longer fit in on-chip memory, and the kernel must read from HBM for the tail tokens. FPGA HBM bandwidth (460 GB/s) is lower than GPU HBM bandwidth (1.6 TB/s on MI210). For sequences where most key accesses are from HBM (e.g., at 1M tokens, the first 65K keys are in BRAM/URAM but the remaining 935K are in HBM), the GPU's higher peak HBM bandwidth and larger cache can achieve higher effective throughput, leading to the dynamic fallback decision.

Extensions for other methods:

SeerAttention-R: The Inner Product Engine computes dot products between pooled query and key vectors (one per block of 64 tokens), producing one score per block instead of one score per token. The Reduction unit is a pass-through (no multi-head weighting). The Top-K Retriever selects blocks up to the token budget (4096) or applies a threshold-based filter (scores > 5e-4).
LServe: The Inner Product Engine computes dot products between the query vector and the two extremal vectors (min and max) for each logical page. The Reduction unit finds the maximum score per physical page (max reduction across logical pages). The Top-K Retriever selects the highest-scoring physical pages.
RAG with BM25: The Inner Product Engine is replaced with a BM25 scoring module that computes term-frequency and inverse-document-frequency statistics. The Reduction unit is a pass-through (per-document score is the output of BM25). The Top-K Retriever selects the top-64 documents.
Two-stage RAG: The first stage uses an embedding model (run on GPU during Prepare Memory) and BM25 (run on FPGA during Compute Relevancy). The fusion of first-stage scores (embedding similarity + BM25) occurs on the FPGA. The second-stage reranker runs on the GPU because it is a full transformer forward pass (compute-bound, regular).

Memory as Context kernel (Appendix E, Figure 17): This is a separate FPGA architecture that fuses query generation with cross-attention. The key difference from the General Setup is that the query vector is generated on the FPGA (through a linear projection on segment embeddings) rather than received from the GPU, and the output is a weighted memory embedding (not indices). The modules are: Segment Loader (reads segment embeddings from HBM, streamed from CPU) → Query Linear Projection (dense matrix-vector multiply to produce query vector) → Memory Loader (reads past memory embeddings from HBM) → Cross Attention (dot product between query and each memory embedding, followed by weighted sum to produce retrieved memory embedding). Output is written back to HBM and delivered to GPU.

MemAgent kernel (Appendix E, Figure 18): This follows prior FPGA LLM accelerator designs (FlightLLM (Zeng et al., 2024), LUT-LLM (He et al., 2025c), GLITCHES (Yang et al., 2024a)) specialized for decoding. The architecture includes separate engines for Linear Projection (INT4 precision to match weight quantization), Attention (FP32 precision for accuracy), SwiGLU, and LayerNorm, connected through a global buffer. Data streams through each engine sequentially, with on-chip storage of weight matrices and intermediate activations. The specialization for decoding is that "the attention is a sequence of GEMV operations" (matrix-vector multiply, since the query is a single token) rather than GEMM (matrix-matrix multiply for prefilling), allowing higher parallelism in the hidden dimension.

3.4.6 Deployment Library and Reuse Strategy

The paper describes its implementation approach in Section 5.4, positioning it as a reusable infrastructure rather than a one-off implementation.

Library structure: "Traversing the methods in Table 1, we implement each step (standalone or fused) as reusable kernels to form a library." Each pipeline stage is implemented as a configurable kernel module: Prepare Memory kernels (linear projections, pooling, min/max computation, tokenization), Compute Relevancy kernels (inner product, BM25 scoring, embedding similarity), Retrieval kernels (top-k, threshold, max reduction, weighted sum), and Apply to Inference kernels (sparse attention, document concatenation, memory selection). These can be instantiated standalone or fused with adjacent stages (Compute Relevancy + Retrieval as in the General Setup, Query Generation + Cross Attention + Selection as in Memory as Context).

API for composition: The paper provides a GPU-FPGA communication API that users can use to "build new method by recombining kernels." The example given: "block-based RAG with BM25 and max-reduction kernels" can be constructed by combining the BM25 scoring module (from RAG's Compute Relevancy) with the max-reduction module (from LServe's Retrieval) and the standard top-k module. This composability requires that the kernel interfaces be standardized—the output format of Compute Relevancy modules must be compatible with the input format of Retrieval modules.

Boundary of provided vs. custom work: The paper explicitly acknowledges: "Arbitrary methods may require custom kernels. We plan to reduce this effort via design automation in our future work." The library covers the methods in Table 1, but novel methods with fundamentally different scoring functions or retrieval heuristics would require new FPGA kernel development. The paper does not describe the programming model for custom kernel development (e.g., whether users write HLS C++, use a domain-specific language, or use the provided kernels as templates to modify).

Host binaries for existing methods: Each method in Table 1 has a "provided host binary" that configures the specific pipeline—which kernels to instantiate, which memory tiers to use, the mapping to GPU vs. FPGA, and the communication schedule. These host binaries are the concrete instantiation of the paper's mapping decisions for each method.

Relationship to prior FPGA LLM work: For MemAgent and Memory as Context, the paper "adopt[s] the design paradigm of prior FPGA-based LLM accelerators (Yang et al., 2024a; Zeng et al., 2024; He et al., 2025b;c; Zhang et al., 2026)." Specifically:

FlightLLM (Zeng et al., 2024) provides the overall architecture pattern (separate engines for different operation types, streaming dataflow, on-chip weight storage).
LUT-LLM (He et al., 2025c) provides the separation of attention and linear projection engines for precision management.
GLITCHES (Yang et al., 2024a) provides the prefill-decode disaggregation mapping.
FlexLLM (Zhang et al., 2026) provides the composable HLS library approach (used for the Memory as Context kernel design).

The paper's novel FPGA contribution is primarily the General Setup kernel (fused Compute Relevancy + Retrieval) and the integration of these prior design paradigms into the specific memory processing pipeline context with the two-criteria mapping policy.

4. Key Insights and Innovations

Innovation 1: A Processing-Centric Unification of Diverse LLM Memory Optimizations

The paper's most intellectually distinctive contribution is not any specific hardware mapping or performance number—it is the conceptual reframing of LLM inference memory management from a storage taxonomy to a processing pipeline. Prior surveys (Wu et al., 2025; Zhang et al., 2025a) classified LLM memory by what it stores: parametric memory (model weights), contextual memory (KV caches), external memory (indexed vectors), and procedural memory (event stores). These taxonomies are storage-centric: they categorize based on representation, origin, and retention duration. What they miss—and what this paper makes central—is that across these storage-diverse mechanisms, the operations performed on memory during inference follow a remarkably consistent four-stage pattern: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference.

This is a fundamental reframing, not an incremental taxonomy extension. A storage classification tells you that DeepSeek Attention and Titans both maintain "contextual memory"; a processing classification tells you that despite this superficial similarity, DeepSeek Attention's bottleneck is in the Compute Relevancy stage (inner products on compressed keys, invoked per token) while Titans' bottleneck spans a fused Compute Relevancy + Retrieval stage (cross-attention on segment and memory embeddings, invoked per segment). Conversely, two methods that appear algorithmically unrelated—DeepSeek Attention (sparse attention with an indexer) and DRAGIN (dynamic RAG with BM25 retrieval)—share the same stage-level computational profile: a memory-bound, irregular-access Compute Relevancy followed by a memory-bound, data-dependent Retrieval. This is the insight that enables a single hardware mapping strategy to apply across them.

The significance of this reframing extends beyond this paper's evaluation. It provides a shared vocabulary for the field: researchers developing new memory optimizations can describe their work in terms of which pipeline stages they modify and what computational properties those stages exhibit, rather than inventing bespoke descriptions. It provides a diagnostic framework: profiling a new method through the four-stage lens immediately reveals where the bottleneck is and what kind of computation it involves, guiding optimization effort. And it provides a hardware-targeting interface: the stage decomposition maps naturally to heterogeneous hardware decisions, because each stage's computational properties (arithmetic intensity, access pattern, data dependency) are consistent across method instantiations. The pipeline abstraction is validated empirically by Table 1, which shows that all representative long-context methods—including those the authors did not develop—fit into the four-stage decomposition without forcing. This is a strong signal that the pipeline captures something fundamental about how LLM inference processes memory, not something idiosyncratic to the methods studied.

Innovation 2: Computational Heterogeneity as the Diagnostic Lens for Hardware Mapping

The paper's second conceptual contribution is elevating computational heterogeneity—the systematic variation in arithmetic intensity, access pattern, and data dependency across pipeline stages—from an observation to the primary diagnostic for hardware offloading decisions. Prior FPGA-based LLM accelerators (Zeng et al., 2024; Yang et al., 2024a; He et al., 2025c) offloaded operations to FPGAs based on heuristics: LLM decoding is memory-bound, so it goes on the FPGA; attention is compute-bound, so it stays on the GPU. This paper makes the analysis systematic: it quantifies arithmetic intensity (FLOPs/byte) for every stage of every method (Table 2, Figures 13-14), classifies access patterns (regular vs. irregular) and data dependencies (local vs. across-memory), and then maps stages based on a two-criteria policy where computational fit is explicitly prioritized against communication cost.

What distinguishes this from prior work is the cross-method scope. Prior FPGA accelerators targeted specific LLM operations in isolation; this paper profiles qualitatively different memory mechanisms and shows that despite their algorithmic diversity, the stage-level computational properties are consistent—Compute Relevancy is 1-10 FLOPs/byte with irregular accesses across sparse attention, RAG, and Memory as Context; Retrieval is ~1 FLOP/byte across all methods. This consistency makes the mapping policy generalizable: the specific IP core for inner-product scoring may differ between DeepSeek Attention and BM25, but the decision to map it to FPGA follows from the same computational profile. The paper's exclusion of TTT/LaCT from the heterogeneous system is equally important: it demonstrates that the framework can diagnose when not to offload—TTT's memory processing is dominated by compute-bound forward and backward passes that lack the heterogeneity that would justify FPGA mapping. This is a negative result with practical implications: it prevents futile engineering effort.

The innovation here is not the roofline model itself (Williams et al., 2009) or the general observation that GPUs struggle with irregular memory-bound operations (Boutros et al., 2020)—it is the application of these concepts to a newly unified problem domain and the demonstration that the resulting mapping decisions are both principled (based on explicit criteria, not heuristics) and cross-method generalizable. This shifts the conversation from "FPGAs can accelerate LLMs" (a point demonstration) to "LLM memory processing has computational structure that makes it systematically suitable for heterogeneous acceleration" (a domain characterization).

Innovation 3: A Cost-Aware Offloading Policy That Prioritizes Communication Over Kernel Speedup

The paper's systems contribution—the offloading policy itself—is distinguished by an explicit ordering of criteria that, while stated simply, represents a departure from common practice in accelerator system design. The policy (Section 5.2) states: deploy operations based on computational fit (Criterion 1), but prioritize minimizing communication overhead (Criterion 2) over kernel-level speedup. The specific, concrete consequence is that memory-bound operations are deliberately left on the GPU when the PCIe transfer cost of their operands would exceed the FPGA's computational advantage.

This is not a theoretical contribution, but it is a design insight with implications for how heterogeneous accelerator systems should be architected. The dominant narrative in heterogeneous computing—particularly in FPGA-accelerated machine learning—emphasizes identifying "FPGA-friendly" operations (sparse, irregular, memory-bound) and offloading them, with data movement treated as a secondary concern to be optimized afterward. The paper inverts this: communication cost is the primary determinant; kernel fit is secondary. The concrete example—KV cache extraction for top-k indices remains on the GPU despite being memory-bound because transferring the full KV cache would outweigh the FPGA's kernel advantage—makes the principle tangible. The extensive profiling in Appendix C.1, which quantifies PCIe overhead relative to computation time across methods (showing 10-200,000× ratios), provides empirical validation that Criterion 2 does not prevent offloading in practice for the studied methods, but the principle itself is independent of this validation: it is a policy, not merely an observation.

This innovation matters beyond this paper because it provides a template for heterogeneous system design. As accelerators diversify (FPGAs, ASICs, CXL-attached memory, near-memory compute) and interconnects evolve (PCIe 5.0/6.0, CXL, UALink), the central design tension will increasingly be the choice between computational locality and computational specialization. The paper's two-criteria policy, with its explicit prioritization, offers a framework for navigating this tension that does not depend on the specific hardware evaluated. The fact that the paper's prototype uses an older-process FPGA (16 nm U55C) connected via PCIe 3.0—arguably the worst-case communication-to-compute ratio—and still achieves net speedups strengthens the case for the policy's generality: if it works with this interconnect, it will work with better ones.

Innovation 4: Memory Processing Overhead Quantified as a Scaling Bottleneck That Grows With Context Length

The paper's empirical finding that memory processing accounts for 22-97% of inference latency (Figures 3-5), with the fraction monotonically increasing with memory size, is a diagnostic contribution that changes the cost model for long-context LLM inference. Prior to this work, the field understood that long contexts incur overhead—KV cache memory pressure, attention compute scaling—but the dominant narrative focused on attention computation itself (quadratic in prefill, linear in decode) as the primary bottleneck. This paper shows that when advanced memory optimizations are applied (sparse attention, RAG, compressed memory), the non-attention memory management operations—the indexing, scoring, retrieval, and integration steps—can dominate total latency, reaching 81% of decoding time for sparse attention at 1M tokens and 97% for synthesized memory.

This finding is significant because it redirects optimization effort. If attention is the bottleneck, the response is to build faster attention kernels (e.g., FlashAttention) or sparsify attention further. If memory processing is the bottleneck, the response is to accelerate indexing, scoring, and retrieval—operations that are architecturally distinct from attention and require different hardware support. The paper's profiling (Section 3.2, Figures 3-5) establishes that the latter is the case for modern optimized inference, and that the problem will only become more acute as context lengths continue to scale (128K → 1M → beyond). The monotonic growth trend in Figure 3 (1-11% at 4K tokens → 22-81% at 1M tokens) is the key evidence: it is not a static overhead but a fundamental scaling bottleneck that widens with context length.

What distinguishes this from typical profiling studies is its cross-method scope. A paper on DeepSeek Attention profiling DeepSeek Attention's overhead is an implementation study. This paper profiles representative methods across four categories (sparse attention, RAG, compressed memory, TTT) and finds the same pattern—memory processing fraction grows with memory size, with the bottleneck stage varying by method type but always within the four-stage pipeline. This systematic evidence transforms the finding from an implementation detail to a domain principle: memory processing is an architectural bottleneck of long-context LLM inference, not an artifact of any particular optimization technique.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on multiple workloads using the same datasets as the original works for each method (Section 6.1). For sparse attention, per-token latency is measured under varying past-token lengths. For RAG, the Wikipedia dump following DRAGIN (Su et al., 2024) is used, with two-stage RAG following the RAG-EDA setup (Pu et al., 2024) with document counts up to 20M. For Memory as Context and MemAgent, total request latency is measured under different input sequence lengths using default hyperparameters from the original works. For TTT/LaCT, the LaCT codebase (Zhang et al., 2025b) is used for profiling.
Base model(s). The experiments span multiple model families, chosen to match each method's original implementation. DeepSeek Attention uses DeepSeek V3.2 Exp (Liu et al., 2025), with vLLM modified to load only the first layer due to MI210 GPU memory constraints (64 GB HBM). SeerAttention-R uses Qwen 3 8B (Yang et al., 2025a) with a block size of 64. LServe uses Llama 3.1 8B (Grattafiori et al., 2024) ported via HIPIFY. RAG methods (DRAGIN, FLARE, Fixed-sentence RAG) use Llama 2 7B (Touvron et al., 2023) as the generator model. Two-stage RAG uses Llama 3.1 8B. Memory as Context uses an open-source HMT implementation modified to replicate Titans via linear projection of the summarization step. MemAgent uses Qwen 2.5 7B (Team et al., 2024). TTT/LaCT uses the LaCT codebase. This diversity of models reflects the paper's goal of demonstrating that the pipeline abstraction and heterogeneous mapping apply across model architectures, not just to a single model family.
Metrics. Two primary metrics are reported. End-to-end latency (in milliseconds for sparse attention per-token, in seconds for RAG/MemAgent/Memory as Context per request) measured with Python performance counters. Latency fraction attributable to memory processing, derived from PyTorch Profiler kernel timing data—the paper explicitly notes this is used "to derive the latency breakdown without interfered by tracing overhead" (Section 6.1), meaning the profiler's sampling-based timing is used rather than instrumented code that would perturb measurement. Speedup is computed as the ratio of baseline latency to GPU-FPGA system latency for both memory processing alone and end-to-end inference. Energy efficiency is measured in Joules per token (sparse attention) or Joules per request (RAG, MemAgent, Memory as Context), with geomean improvement reported across sequence lengths or document counts (Table 3). For batch experiments (Table 4), geomean speedup across batch sizes is reported.
Baselines. The primary baseline is GPU-only execution on the same GPU model (AMD MI210) using optimized implementations: vLLM for DeepSeek Attention, TileLang-optimized kernels for SeerAttention-R, HIPIFY-ported CUDA kernels for LServe, and BM25S (Lù, 2024) as the retrieval backend for single-stage RAG (replacing the slower ElasticSearch used in original work). For two-stage RAG, the baseline includes bge-large-en-v1.5 for first-stage embedding plus bge-reranker-large for second-stage reranking (Xiao et al., 2023). For MemAgent, the baseline is GPU-only execution with Qwen 2.5 7B. For Memory as Context, the baseline is GPU-only execution using the modified HMT/Titans implementation. The paper also provides estimated baselines for NVIDIA A100 in Appendix H (Figures 25-26) derived by "aggregating the measured latency components of the FPGA, GPU, and PCIe communication, while profiling kernel execution latency separately" (Appendix H), though these lack physical co-located measurement.
Generation budget / compute accounting. For sparse attention, the relevant metric is sequence length (past tokens processed), swept from 4K to 1M tokens, with per-token latency measured. For RAG, the metric is document count (number of documents in the retrieval corpus), swept from 500K to 20M documents, with per-request latency measured. For MemAgent, segment length (5000 tokens) and memory size (1024 tokens) are fixed per the original work, with output max token length set to 32. For Memory as Context, segment length (1024 tokens) and output sequence length (32 tokens) are fixed. For batch experiments (Table 4), batch sizes range from 1 to 32 for sparse attention and RAG, 1 to 32 for Memory as Context, and 1 to 32 for MemAgent. The paper does not use FLOPs or generation counts as a universal compute metric; rather, each method's natural scaling parameter (sequence length, document count) is varied. The FPGA and GPU kernels are both included in latency measurement, and PCIe transfer time is included in the heterogeneous system's latency (Appendix C.1 quantifies transfer times separately to show they are negligible).
Cross-validation / statistical protocol. No formal cross-validation or statistical significance testing is reported. The paper presents single-run measurements across a sweep of scaling parameters (sequence lengths, document counts). The profiling breakdowns (Figures 3-5) are based on PyTorch Profiler traces, which provide deterministic per-kernel timing on single runs. The latency measurements use Python performance counters, presumably averaged over multiple iterations, but the number of iterations is not specified. The paper does not report variance, confidence intervals, or error bars on any measurement. For the dynamic fallback decisions (switching to GPU-only beyond 1M tokens for LServe/DeepSeek Attention, beyond batch size 2 for MemAgent), no evaluation of the switching mechanism is provided—these are stated as capabilities but not experimentally validated. The A100 estimates in Appendix H are derived from separate profiling (kernel execution on A100 plus FPGA timing from MI210 system, with PCIe estimates) rather than co-located measurement, introducing potential timing inaccuracies from platform-specific interactions (CPU scheduling, PCIe topology differences).

Main Quantitative Results

The paper organizes results around three mechanisms of FPGA advantage—large on-chip memory, pipelined flexible datapath, and faster decoding—each corresponding to a different deployment configuration and method category.

Case 1: Large On-Chip Memory Advantage for Block-Sparse Attention

The headline finding is that the FPGA's on-chip memory (BRAM at 21.8 TB/s + URAM at 10.4 TB/s = ~32.2 TB/s aggregate) provides approximately 5× higher effective bandwidth than GPU SRAM for the memory-bound Compute Relevancy and Retrieval stages in block-sparse attention methods, yielding 1.8–4.9× kernel-level speedup and 1.04–1.49× end-to-end speedup.

SeerAttention-R: Figure 9 shows memory processing speedup on the GPU-FPGA system over GPU baseline for both top-k mode (token budget 4096) and threshold mode (threshold 5e-4). For top-k mode, speedup ranges from 1.8–2.2× across sequence lengths. For threshold mode, speedup is higher at 2.6–4.9× because threshold-based selection involves more irregular data-dependent filtering that benefits more from the FPGA's flexible datapath. These kernel-level speedups translate to 1.04–1.25× end-to-end speedup (Figure 8), with the lower multiplier reflecting that Prepare Memory and Apply to Inference (dense projections and block-sparse attention) remain on GPU and are not accelerated.

The scaling trend in Figure 9 is flat for SeerAttention-R: speedup remains roughly constant as sequence length increases from 4K to 1M tokens. This is because the compressed key vectors (block-level pooled representations) are small enough to fit entirely in on-chip memory even at 1M tokens (block size 64 means 1M tokens produce ~15,625 blocks, each block's pooled key vector is a few hundred bytes, total ~3-6 MB, well within the 40 MB BRAM+URAM capacity). The key vectors are never evicted to HBM, so the FPGA consistently operates at on-chip bandwidth regardless of sequence length.

LServe: Figure 9 shows a different pattern for LServe: speedup starts high at 5.6× for 4K tokens, remains elevated at 2.5–4.5× for 64K–256K tokens, then drops to 1.2× at 1M tokens. This is because LServe's paged organization involves larger key vector representations (min and max vectors per logical page) that exceed on-chip capacity as sequence length grows. At 4K tokens, all page metadata fits in BRAM+URAM. At 256K tokens, a significant fraction spills to FPGA HBM (460 GB/s). At 1M tokens, the majority of page metadata is in FPGA HBM, where bandwidth is lower than GPU HBM (1.6 TB/s on MI210), and the FPGA's advantage nearly vanishes. This is the motivation for the dynamic fallback: "the system can dynamically fall back to GPU-only execution to avoid a performance loss" (Section 6.2, Appendix F) for sequences beyond 1M tokens. The end-to-end speedup (Figure 8) follows the same pattern: 1.15–1.49× for sequences up to 256K, declining toward 1.0× at 1M tokens.

DeepSeek Attention: Figure 9 shows 1.3–2.2× memory processing speedup, translating to 1.1–1.2× end-to-end speedup (Figure 8). The speedup is lower than SeerAttention-R and LServe at short sequences because DeepSeek Attention's Compute Relevancy involves multi-headed dot products (64 heads) with weighted average reduction, which has higher arithmetic intensity than single-score block attention—more of the computation is arithmetic-limited rather than memory-limited, reducing the FPGA's bandwidth advantage. Additionally, DeepSeek Attention's key indexing vectors (with partial RoPE embeddings) are larger per token than block-pooled representations, causing earlier HBM spill at around 65K tokens where on-chip capacity is exceeded.

The DeepSeek Attention results also demonstrate the impact of HBM access: the speedup curve in Figure 9 shows a slight decline as sequence length increases from 4K to 256K (from ~2.2× to ~1.3×), then a sharper decline beyond 1M tokens, consistent with the transition from primarily on-chip to primarily HBM access. The paper notes (Section 6.2): "When the sequence length exceeds 1M tokens, LServe and DeepSeek Attention will experience a drop of speed on the FPGA due to accessing the HBM."

A key detail in the reported numbers: The end-to-end speedups (1.04–1.49×) are substantially lower than the memory processing speedups (1.8–4.9×) because memory processing is only 22–81% of total latency at these sequence lengths (Figure 3). By Amdahl's Law, even an infinite speedup on the memory processing fraction x yields at most a 1/(1-x) end-to-end speedup. For x = 0.50 (50% memory processing), the maximum end-to-end speedup is 2×. The paper's results are consistent with this bound: at sequence lengths where memory processing is ~50% of latency, the 2–5× memory processing speedup translates to ~1.3–1.5× end-to-end.

Case 2: Pipelined and Flexible Datapath for Sparse Attention, RAG, and Memory as Context

The headline finding is that the FPGA's streaming dataflow architecture, which fuses Compute Relevancy and Retrieval into a single pipelined kernel with fine-grained overlap of communication and computation, provides 1.1–6.6× memory processing speedup across methods that involve irregular scoring and selection, translating to 1.1–2.2× end-to-end speedup.

DeepSeek Attention (pipelining aspect): In addition to the on-chip memory advantage discussed above, DeepSeek Attention benefits from fusion of the multi-headed inner product computation with the weighted-average reduction and the top-k selection. On the GPU, these would execute as separate kernels (inner product → reduce → top-k), each reading the score tensor from HBM and writing it back. The FPGA eliminates these intermediate HBM accesses by streaming scores directly from the Inner Product Engine through the Reduction unit into the Top-K Retriever via FIFO channels (Figure 7). This is the source of the 1.3–2.2× speedup in Figure 9 for sequence lengths where on-chip memory is not the primary bottleneck (i.e., at longer sequences where HBM access dominates both GPU and FPGA, the fusion advantage persists while the bandwidth advantage diminishes).

RAG (BM25-based methods): Figure 10 (right) shows memory processing speedup for single-stage RAG methods (DRAGIN, FLARE, Fixed-sentence RAG) of 5.1–6.6× over the BM25S baseline. This is the largest memory processing speedup in the paper. The reason is that BM25 scoring involves irregular accesses to token-frequency histograms across documents, with the access order determined by query tokens rather than document layout—a pattern that causes severe cache thrashing on GPUs. The FPGA's custom datapath pipelines the BM25 computation with the top-k selection, avoiding the intermediate score tensor that BM25S writes to memory.

This 5.1–6.6× memory processing speedup translates to 1.14–1.47× end-to-end speedup for DRAGIN, 1.19–1.55× for FLARE, and 1.26–1.58× for Fixed-sentence RAG (Figure 10, left), with speedup increasing with document count because memory processing becomes a larger fraction of total latency as the retrieval corpus grows (Figure 4).

A notable detail: the paper replaced the original ElasticSearch retriever with BM25S as the GPU baseline, stating BM25S provides "orders of magnitude faster lexical search" (Lù, 2024). This is a strong baseline choice—the FPGA is not compared against a naive CPU implementation but against an optimized GPU lexical search. The 5.1–6.6× speedup is over an already-optimized baseline.

Two-stage RAG: Figure 10 (right) shows 1.1–2.1× memory processing speedup for two-stage RAG, translating to 1.16–1.84× end-to-end speedup (Figure 10, left). The speedup is substantially lower than single-stage RAG because the second-stage reranker (bge-reranker-large) dominates execution time and runs entirely on the GPU. The FPGA only accelerates the first-stage retrieval (embedding search + BM25 fusion + top-64 selection), which is a diminishing fraction of total latency as the reranker cost grows. The paper notes (Section 6.2): "two-stage RAG is limited to 1.1–2.1× due to reranker dominance." This is a clear demonstration of the Criterion 2 limitation: even though second-stage retrieval could theoretically be offloaded, the reranker—a full transformer forward pass—runs more efficiently on GPU because it is compute-bound and regular, and its output is required on the GPU for context assembly regardless.

Memory as Context (Titans/HMT): Figure 11 (right) shows 3.1–4.0× memory processing speedup, translating to 1.3–1.6× end-to-end speedup. The FPGA fuses query generation (linear projection on current segment embedding) with cross-attention (scoring against all past memory embeddings) and weighted-sum retrieval, as described in Appendix E, Figure 17. This is the third distinct kernel architecture (after General Setup and MemAgent decoding), demonstrating the library-based approach. The speedup is high (3.1–4.0×) because the Memory as Context pipeline is entirely memory-bounded: query generation, cross-attention scoring, and weighted sum all involve streaming through memory embeddings with limited arithmetic intensity, making the FPGA's streaming dataflow highly effective. The end-to-end speedup is moderate (1.3–1.6×) because memory processing constitutes 40–60% of total latency in this method (Figure 5, right), and the GPU's model forward pass (applying the retrieved embeddings to subsequent segment processing) is not accelerated.

Case 3: Faster Decoding for Synthesized Memory (MemAgent)

The headline finding is that under prefill-decode disaggregation, the FPGA running LLM decoding (Prepare Memory) achieves a consistent 1.8× end-to-end speedup over GPU-only execution for MemAgent (Figure 12), the highest end-to-end speedup among all methods for batch size 1.

Why 1.8× and not higher: MemAgent's memory processing pipeline is dominated by Prepare Memory (LLM decoding), which constitutes up to 97% of total latency (Figure 5, left). The FPGA's decoding advantage comes from its ability to sustain higher effective HBM bandwidth utilization for memory-bound matrix-vector operations (GEMV per generated token) compared to GPUs, where peak HBM bandwidth is often underutilized during decoding due to the lack of weight reuse across tokens. Prior work (Zeng et al., 2024; He et al., 2025c) has demonstrated this decoding advantage on FPGAs, and the paper builds on those designs (Appendix E, Figure 18).

The 1.8× end-to-end speedup is constant across segment lengths (Figure 12, left) because the decoding time scales with the number of generated memory tokens (fixed at 1024 per segment, per MemAgent defaults) and the model size, not with the input segment length. The GPU runs Apply to Inference (prefilling on the input segment + synthesized memory), which is compute-bound and benefits from GPU parallelism—this is why prefilling stays on the GPU in the disaggregated mapping.

The PCIe cost for this configuration: The KV cache transfer from GPU to FPGA and token ID transfer back take 14–218 ms per segment (Appendix C.1), while the FPGA decoding takes 17–534 seconds. The ratio is ~2500×, meaning communication is negligible relative to computation. However, this is the largest absolute PCIe transfer time in the paper (up to 218 ms), and the paper acknowledges that if FPGA decoding throughput were substantially higher (e.g., through multi-FPGA or ASIC acceleration), the communication-to-computation ratio could shrink enough that PCIe becomes a bottleneck.

Batch Inference Scaling

Table 4 presents geomean speedup of the GPU-FPGA system over GPU-only baselines across batch sizes (BS=1, 2, 4, 8, 32) for each method. The patterns reveal how batching affects the relative advantage of heterogeneous execution:

Sparse attention speedup increases with batch size: For SeerAttention-R threshold mode, geomean speedup grows from 1.12× at BS=1 to 1.60× at BS=32. For LServe, from 1.19× to 1.83×. The paper explains this (Section 6.4): "KV cache and latent indexing embeddings are not shared across samples within a batch... high batch size does not improve data reuse for score computations in sparse attention on GPUs." In other words, the memory-bound Compute Relevancy and Retrieval stages do not benefit from batching on GPUs because each sample has its own distinct key vectors and query—there is no weight reuse and minimal data sharing. Dense components (linear projections, feedforward layers) do benefit from weight reuse across batch samples, so as batch size increases, a larger fraction of total latency is attributable to the (unaccelerated) memory processing stages, amplifying the FPGA's advantage.

RAG speedup increases with batch size: DRAGIN geomean speedup grows from 1.14× (BS=1) to 1.92× (BS=32). FLARE from 1.19× to 2.11×. Fixed-sentence RAG from 1.26× to 2.10×. The reasoning is similar: BM25 scoring and document retrieval are input-dependent and cannot be shared across batch samples, so batching does not improve their throughput on GPUs. Dense components (the generator model's forward pass) do benefit from batching, shifting the latency bottleneck increasingly toward retrieval. Two-stage RAG shows more modest scaling (1.16× to 1.37×) because the reranker benefits from weight reuse across samples in the batch, moderating the shift in bottleneck fraction.

Memory as Context speedup decreases with batch size: Geomean speedup drops from 1.48× (BS=1) to 1.15× (BS=32). The paper attributes this (Section 6.4) to the offloaded cross-attention containing linear projections that can benefit from weight reuse on GPUs at higher batch sizes: "With larger batch sizes, these linear projections achieve higher weight reuse and improved GPU utilization. This reduces the relative advantage of offloading." However, the paper notes that "since the memory embeddings remain independent across samples, FPGA acceleration still provides benefits in long-sequence regimes for computing the cross attention score and perform selection over the memory embeddings," explaining why speedup remains above 1.0× even at BS=32.

MemAgent speedup reverses to slowdown at larger batches: Geomean speedup is 1.85× at BS=1, drops to 1.65× at BS=2, then becomes a slowdown: 0.93× at BS=4, 0.49× at BS=8, and 0.13× at BS=32. This is the most dramatic batch sensitivity in the paper. The reason (Section 6.4): "Under batching, the decode stage significantly benefits from weight reuse on GPUs. Given the lower compute throughput of FPGAs for dense operations, this leads to performance degradation as batch size increases." LLM decoding with batch size >1 transforms the memory-bound GEMV per token into a batched GEMM where weight matrices are reused across batch samples, dramatically improving GPU utilization and throughput. The FPGA, with lower raw compute throughput (the U55C has fewer DSP slices than the MI210 has tensor cores), cannot match this batched throughput improvement. The paper's dynamic fallback—"when the batch size is larger than 2 in MemAgent, we switch to a GPU-centric deployment to avoid slowdown" (Section 6.4)—is necessary for practical deployment but highlights a fundamental limitation: the FPGA decoding advantage is inherently limited to small-batch, latency-sensitive inference, not throughput-oriented batched serving.

Energy Efficiency

Table 3 reports geomean energy per token (sparse attention) or per request (RAG, MemAgent, Memory as Context) for both the GPU-FPGA system and the GPU baseline.

Sparse attention energy improvement: DeepSeek Attention: 1.61× geomean reduction (15.86 vs. 25.62 J/token). SeerAttention-R top-k: 1.11× (0.32 vs. 0.36 J/token). SeerAttention-R threshold: 1.14× (0.30 vs. 0.34 J/token). LServe: 1.43× (0.29 vs. 0.43 J/token). The variation reflects differences in FPGA power consumption (Appendix G: 24.9–26.4 W across sparse attention kernels) relative to GPU power (45–55 W for the same operations). The energy reduction for SeerAttention-R is modest (1.11–1.14×) because the absolute energy per token is already very low (0.30–0.36 J) and the FPGA operating power (24.9 W) is not dramatically lower than the GPU kernel power (45 W) for these operations. The larger reduction for DeepSeek Attention (1.61×) reflects the greater fraction of latency offloaded and the higher GPU power (55 W) for the more complex multi-headed scoring.

RAG energy improvement: DRAGIN: 1.10× geomean reduction (328.16 vs. 362.57 J/request). FLARE: 1.14× (241.99 vs. 275.53). Fixed-sentence RAG: 1.21× (259.88 vs. 315.25). Two-stage RAG: 1.07× (150.33 vs. 160.68). The reductions are modest because RAG energy is dominated by the generator model's forward pass (Llama 2 7B or Llama 3.1 8B running on GPU at 106 W), and the FPGA acceleration (29.7 W for BM25 and retrieval) affects only the retrieval component. Two-stage RAG shows the smallest improvement (1.07×) because the reranker dominates energy consumption and runs on GPU unmodified.

Synthesized memory and Memory as Context: MemAgent achieves 4.66× geomean energy reduction (3,202 vs. 13,662 J/request)—the largest in the paper. This is because LLM decoding is the dominant energy consumer (97% of latency), and the FPGA (44.2 W) is substantially more energy-efficient for this memory-bound operation than the GPU (99 W). Memory as Context achieves 1.65× (16.55 vs. 27.31 J/request), reflecting the 42.6 W FPGA vs. 94 W GPU for the fused cross-attention and retrieval.

Energy scaling with memory size: The paper notes (Section 6.3): "energy efficiency improvements generally increase with memory size, except for DeepSeek Attention and LServe due to the decreasing performance after 1M tokens for HBM access (stop at 1.43× and 1.07× respectively)." This means that for methods where the FPGA maintains its advantage at scale (SeerAttention-R, RAG, Memory as Context), the energy benefit grows with context length because the offloaded fraction of work increases. For methods where FPGA performance degrades at scale (DeepSeek Attention, LServe), the energy benefit plateaus or reverses.

Extended A100 Results

Appendix H (Figures 25-26) provides a case study using DeepSeek Attention to estimate performance on an NVIDIA A100 + U55C system. Since the paper lacks physical access to a co-located A100 + U55C platform, results are derived by "aggregating the measured latency components of the FPGA, GPU, and PCIe communication, while profiling kernel execution latency separately" (Appendix H). The key findings:

The MI210+U55C system can outperform an A100-only system in certain configurations (Figure 26), despite the A100 generally being faster than the MI210.
When the A100 replaces the MI210 as the GPU in the heterogeneous system, the speedup over A100-only is maintained (Figures 25-26), showing that the heterogeneous advantage "is largely agnostic to the specific GPU model" (Appendix H).
The paper estimates that migrating from U55C (16 nm) to a modern FPGA (AMD Versal V80, 7 nm) would provide "an additional 1.6× geomean end-to-end speedup" (Appendix H.1), suggesting the current prototype underrepresents the potential of heterogeneous acceleration due to the older FPGA process technology.

These A100 results are estimates, not measurements, and the paper acknowledges this limitation. The methodology—separate profiling of components and summation—does not capture potential interactions like PCIe contention between A100 and U55C DMA engines, NUMA effects in CPU-mediated P2P transfers, or thermal throttling in a co-located configuration.

Ablation Studies and Robustness Checks

The paper does not include formal ablation studies in the traditional sense (e.g., removing or varying components of the proposed system and measuring impact). However, several analyses serve as robustness checks and sensitivity analyses for specific design choices.

Arithmetic intensity measurement (Appendix B, Figures 13-14): The paper quantifies arithmetic intensity (FLOPs/byte) for each pipeline stage across methods, showing that Compute Relevancy and Retrieval consistently fall in the 1-10 FLOPs/byte range (memory-bound) while Prepare Memory and Apply to Inference are 10-100+ FLOPs/byte (compute-bound). This measurement validates the core mapping decision—if the stages did not exhibit this systematic heterogeneity, the heterogeneous system would provide no advantage. The figures also show that for two-stage RAG, the reranker (second-stage relevancy computation) has high arithmetic intensity (>100 FLOPs/byte), consistent with the decision to keep it on GPU despite being part of memory processing.

PCIe transfer latency vs. computation time (Appendix C.1, Figure 16): The paper quantifies transfer latency for varying data sizes and compares to the corresponding GPU kernel latency. For sparse attention, transfer is ~12 µs vs. GPU kernel latency of 128–2,450 µs. For RAG, transfer is ~7 µs vs. GPU kernel latency of 23–1,596 ms. For Memory as Context, transfer is 20–320 µs vs. GPU kernel latency of 26–498 ms. For MemAgent, transfer is 14–218 ms vs. GPU kernel latency of 17–534 seconds. The paper concludes that PCIe overhead is " 1000x difference" from computation time, but this characterization is accurate for RAG and sparse attention (3-4 orders of magnitude) and less so for MemAgent (2-3 orders of magnitude: 218 ms vs. 534 s is ~2,500×). This analysis validates Criterion 2 of the mapping policy—that communication overhead does not outweigh kernel speedup—but only for the current FPGA and GPU combination. A faster FPGA (e.g., Versal V80) or slower interconnect (e.g., PCIe 3.0 x4) could shift this balance.

Dynamic fallback behavior (Section 6.2, Appendix F): The paper describes dynamic fallback for LServe and DeepSeek Attention at >1M tokens and for MemAgent at batch size >2, where the system reverts to GPU-only execution. These fallback decisions are motivated by performance analysis (FPGA HBM bandwidth limitation at extreme sequence lengths; GPU batched throughput advantage) but are not experimentally validated with an actual runtime switching mechanism. The system's ability to detect the crossover point and trigger the switch is mentioned as capability ("the system can dynamically fall back") but not implemented or measured.

HIPIFY porting for LServe (Section 6.1): The LServe CUDA kernels are ported to HIP for the AMD GPU using HIPIFY (ROCm Organization, 2026). This introduces a potential performance difference from the original NVIDIA-optimized CUDA implementation. The paper does not compare HIP-ported performance against the original CUDA implementation or discuss whether the porting introduced overhead that could affect the baseline measurement. Given that the speedup claims are relative to this HIP baseline, any porting inefficiency would inflate the apparent FPGA advantage.

BM25S as optimized baseline (Section 6.1): By replacing ElasticSearch with BM25S for the RAG baseline, the paper strengthens its claims—the 5.1–6.6× speedup is over the state-of-the-art BM25 implementation, not a naive baseline. However, the paper does not provide a comparison against the original ElasticSearch baseline to quantify how much of the improvement is FPGA acceleration vs. baseline optimization. This is appropriate given the goal of fair comparison, but it means the practical speedup over the exact DRAGIN/FLARE implementations from prior work is not directly measurable from the paper's numbers.

Kernel power measurements (Appendix G): The paper reports FPGA kernel power (24.9–44.2 W across methods) and GPU kernel power (45–106 W for the same operations), providing the basis for the energy efficiency claims in Table 3. However, the methodology of power measurement is not described—whether this is board-level power (including HBM, PCIe PHY, and other peripherals) or chip-only power, whether it is measured through hardware power sensors or estimated through tools, and whether it is average or peak power. Without this information, the energy efficiency comparisons (Table 3) should be interpreted as approximate.

Library composability demonstration (Section 5.4): The paper states that "block-based RAG with BM25 and max-reduction kernels" can be constructed by recombining existing kernels, but does not evaluate this composed method's performance. This is a claim about the library's generality, not an experimental result. The evaluation covers only the seven methods explicitly listed in Table 1, all of which have dedicated host binaries. The composability claim is unvalidated.

Critical Assessment

The paper's core claims, as articulated in Section 1 (Claims 1-3), are largely supported by the profiling and measurement data, but with important caveats about scope, methodology, and the strength of inference.

Claim 1 (Memory processing pipeline unifies diverse optimizations) is strongly supported by the mapping in Table 1, which shows all representative methods fitting into the four-stage decomposition without forcing. The profiling data (Figures 3-5) further validates that memory processing is a significant latency fraction (22-97%) across all methods, confirming the pipeline's practical relevance. However, the claim is established through classification, not through a counterfactual demonstration—the paper does not show that a non-pipeline-based analysis would produce worse results. The evidence is the comprehensiveness of the mapping, not a controlled experiment. Additionally, the TTT/LaCT exclusion demonstrates an important boundary: methods where memory processing is not the bottleneck or where it lacks sufficient computational heterogeneity do not benefit from heterogeneous deployment. This is a strength—the pipeline framework identifies when not to offload—but the TTT exclusion is based on profiling the LaCT implementation specifically, and different TTT variants with different computational profiles might behave differently.

Claim 2 (Computations in memory processing are heterogeneous) is convincingly supported by the arithmetic intensity measurements in Appendix B (Figures 13-14) and the qualitative characterizations in Table 2. The 1-2 order of magnitude difference in arithmetic intensity between Prepare Memory/Apply to Inference and Compute Relevancy/Retrieval is consistent across method categories. The quantification is done through profiling on GPU, which is appropriate since the claim is about the operations themselves, not their FPGA implementation. However, the "irregular access pattern" and "across-memory dependency" characterizations in Table 2 are qualitative and not measured—there is no quantification of cache miss rates, memory divergence, or degree of irregularity. The claim would be stronger with GPU performance counter data showing, for example, L1/L2 cache hit rates for these stages to quantify "irregular" vs. "regular" empirically rather than through code inspection.

Claim 3 (Heterogeneous systems accelerate memory processing) is supported by the measured speedups (Figures 8-12, Tables 3-4) but bounded by specific conditions that the paper is partially transparent about. The strongest results—2.2× end-to-end speedup for RAG, 1.8× for MemAgent, 1.49× for sparse attention—are conditional on single-batch, latency-optimized inference. As Table 4 shows, the advantage for MemAgent reverses entirely at batch size >2, and for Memory as Context it diminishes substantially at higher batch sizes. The paper's dynamic fallback mechanism addresses this but is not experimentally validated. The speedup claims are also relative to a GPU-only baseline on the same AMD MI210 GPU; as Appendix H shows, an A100-only system can outperform the MI210+U55C system in some configurations, meaning the heterogeneous system's absolute performance may not exceed what a better GPU alone could achieve. This is not a flaw in the approach—the paper argues the heterogeneous concept generalizes to A100+U55C, and provides estimates to support this—but the claims about absolute speedup over "GPU baselines" should be understood as speedup over the specific GPU used, not over all possible GPUs.

Specific weaknesses and missing evidence:

Single-run measurements without variance: The paper reports no error bars, confidence intervals, or measurement variance for any latency, speedup, or energy number. PyTorch Profiler provides deterministic kernel timing, so per-kernel measurements may have low variance, but end-to-end latency measurements using Python performance counters are subject to system noise (CPU scheduling, PCIe contention, thermal effects). Without variance reporting, it is unclear whether, for example, the 1.04× end-to-end speedup for SeerAttention-R at some sequence lengths (Figure 8) is statistically distinguishable from 1.0×.
No ablation of pipeline stage mapping choices: The paper maps Prepare Memory to GPU and Compute Relevancy + Retrieval to FPGA based on the two-criteria policy, but never evaluates the alternative mappings. What if Compute Relevancy alone were offloaded and Retrieval stayed on GPU? What if the entire pipeline were on FPGA (feasible for smaller models)? Without these ablations, the contribution of the mapping policy itself—as opposed to the contribution of FPGA acceleration in general—is not isolated. The paper demonstrates that the chosen mapping works, not that it is optimal among possible mappings.
Limited model scale diversity: The evaluated models span 7B-8B parameters (Llama 2 7B, Llama 3.1 8B, Qwen 3 8B, Qwen 2.5 7B) plus one much larger model (DeepSeek V3.2 Exp, though only one layer is loaded due to memory constraints). The paper does not evaluate on small models (e.g., 1B parameters) where the absolute memory processing overhead would be smaller, or on very large models (70B+) where GPU memory pressure from KV caches might change the tradeoffs. The claim that the heterogeneous system is effective "across multiple LLM inference optimizations" is supported across method types but not across model scales.
No evaluation of the dynamic fallback mechanism: The paper states—appropriately—that the system falls back to GPU-only execution for LServe/DeepSeek Attention at >1M tokens and for MemAgent at batch size >2, but does not measure the latency of the fallback itself (detection overhead, data migration if KV caches or model weights need to be copied back to GPU). In a production system, frequent fallback transitions could introduce jitter and reduce the practical benefit.
Energy measurement methodology not specified: The energy efficiency results (Table 3, Appendix G) are presented without describing the measurement apparatus (hardware power sensors? software estimation? onboard telemetry?), the measurement duration, or whether idle power is included. The 4.66× energy reduction for MemAgent is the largest single result in the energy section, and the mechanism (FPGA at 44.2 W vs. GPU at 99 W for decoding) suggests the FPGA's process technology disadvantage (16 nm vs. 6 nm) would make this gap larger on a process-normalized basis. But without measurement methodology, the absolute energy numbers are difficult to interpret or reproduce.
No comparison against CPU-offloaded baselines: For RAG, a common deployment pattern is CPU-based retrieval (BM25 on CPU, dense retrieval on CPU) with GPU only for generation. The paper mentions (Appendix F, Case 2) that "some methods (e.g., RAG) adopt CPU offloading as a baseline to accelerate these operations relative to GPU execution," and argues the U55C is faster due to "3.5× higher peak TOPs and substantially higher HBM bandwidth compared to system DRAM." However, no CPU baseline numbers are reported. The EPYC 7v13 CPU is available in the system (Table 5) and could have been profiled as an alternative offload target. Without this comparison, the case for FPGA specifically (rather than any non-GPU accelerator) is incomplete.
Library composability is unvalidated: The claim that users can "build new method by recombining kernels" (Section 5.4) is illustrated only by an example in prose (block-based RAG with BM25 and max-reduction). No composed method is built and evaluated. The development effort for composing kernels—interface compatibility, FIFO width matching, memory tier assignment—is not quantified.

What would strengthen the paper:

Variance reporting for all latency and energy measurements, even if minimal (standard deviation across 5-10 runs).
Ablation of the mapping policy: compare the chosen mapping against a naive offload-everything-to-FPGA and a GPU-only baseline to quantify the contribution of the two-criteria policy specifically.
CPU-offload baselines for RAG: measuring BM25 retrieval latency on the EPYC 7v13 with the same BM25S library to quantify the FPGA advantage over a realistic CPU-offload deployment.
Multi-batch end-to-end results: evaluating the dynamic fallback for MemAgent at batch size transitions (1→2→4) with measurement of transition latency.
Model scale sweep: evaluating at least one method (e.g., SeerAttention-R) across model sizes (1B, 3B, 8B, 70B) to determine whether the memory processing fraction—and thus the heterogeneous speedup—scales with model size.
Energy measurement specification: document how power is measured (onboard sensors, external power meter, vendor tools) and whether idle/static power is included or only dynamic kernel power.

Despite these weaknesses, the experimental section achieves its primary goal: it demonstrates that a GPU-FPGA heterogeneous system can accelerate the memory processing pipeline of LLM inference across diverse method categories with practical end-to-end speedups. The results are internally consistent with Amdahl's Law (end-to-end speedups are lower than kernel speedups in proportion to the offloaded fraction), the batch scaling behavior is explained by the shift between memory-bound and compute-bound regimes, and the energy results align with the kernel power measurements. The paper's transparency about conditions where the advantage diminishes or reverses (long sequences for LServe, large batches for MemAgent) is a strength that provides practitioners with guidance on when to deploy this approach.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Excluded From the Efficiency Calculation

The assumption or constraint. The paper's profiling-based hardware mapping methodology relies on offline analysis—specifically, generating 2048 samples per question and measuring arithmetic intensity and latency breakdown—to determine which pipeline stages are memory-bound versus compute-bound and thus which should be offloaded to the FPGA. This analysis is treated as a one-time cost that is amortized across all subsequent inference requests for a given model and method. However, the paper also proposes dynamic fallback mechanisms—switching from FPGA to GPU-only execution when sequence length exceeds 1M tokens for LServe and DeepSeek Attention, or when batch size exceeds 2 for MemAgent—which require runtime monitoring and decision-making. The cost of this runtime profiling is never measured or included in the reported speedup numbers. The paper acknowledges this only indirectly when discussing future work: "The system can dynamically fall back to GPU-only execution to avoid a performance loss" (Section 6.2), without specifying how the fallback is triggered, what data is monitored, or what latency the detection mechanism introduces.

The consequence. The reported speedups assume perfect, zero-cost knowledge of when to offload and when to fall back. In practice, the system must either (a) make static decisions based on offline profiling (sacrificing optimality when workload characteristics deviate from the profiled conditions), or (b) dynamically monitor sequence length, batch size, or FPGA HBM utilization at runtime (incurring monitoring overhead that reduces net speedup). For the static case, a request with a sequence length just above the 1M-token fallback threshold for DeepSeek Attention would either be processed suboptimally on the FPGA (where HBM bandwidth limits performance) or be conservatively routed to the GPU (losing the FPGA benefit for token ranges where it still helps). For the dynamic case, monitoring latency—particularly for detecting HBM spill conditions, which requires tracking whether on-chip memory capacity is exceeded mid-computation—adds overhead that is proportional to the frequency of monitoring checks. Neither the static misprediction cost nor the dynamic monitoring cost is quantified.

What evidence exists in the paper. The paper presents a static analysis showing LServe speedup declines from 5.6× at 4K tokens to 1.2× at 1M tokens (Figure 9) and DeepSeek Attention memory processing speedup declines from ~2.2× at 4K to ~1.2× at 1M tokens. These curves provide the rationale for the fallback threshold, but the paper does not measure any system's ability to detect the crossover point in real time, nor does it include monitoring overhead in the latency numbers. The batch scaling results (Table 4) similarly motivate the MemAgent fallback at batch size >2, but no transition measurement is provided. Section 5.4 and Appendix F describe the fallback as a capability but do not evaluate it.

Mitigation status. Not addressed. The paper states the fallback as a design principle ("Practically, the system can dynamically fall back to GPU-only execution," Section 6.2) but does not implement or measure it. Future work on dynamic difficulty estimation or runtime monitoring is not explicitly proposed, though the paper mentions design automation (Section 5.4) as a general future direction.

6.2 Single Hardware Platform Evaluation Limits Generality of Heterogeneous Advantage Claims

The assumption or constraint. All latency and energy measurements are conducted on a single hardware configuration: an AMD MI210 GPU (6 nm process, 64 GB HBM at 1.6 TB/s) paired with an AMD Alveo U55C FPGA (16 nm process, 16 GB HBM at 460 GB/s) connected via PCIe 3.0 P2P (peak 32 GB/s). The paper explicitly notes this limitation: "We do not have physical access to a system with both NVIDIA A100 and U55C on the same node, but we provide estimation based on the real-system profiling in Appendix H to demonstrate the generalizability of our system" (Section 6.1). The A100 estimates in Appendix H are derived by aggregating separately profiled component latencies, not measured on a co-located system. The paper also acknowledges that the U55C is fabricated in an older process technology (16 nm) than the GPU (6 nm) and costs half as much, and that "a newer FPGA model (e.g., AMD Versal V80) can further improve performance" (Section 6.1).

The consequence. The reported speedup magnitudes—1.04–2.2× end-to-end, 1.1–4.7× energy reduction—are specific to this particular GPU-FPGA combination and interconnect. Three hardware-dependent factors cannot be separated from the results:

FPGA process technology: The U55C's 16 nm process limits its clock frequency, logic density, and on-chip memory capacity relative to what a modern 7 nm FPGA (Versal V80) could achieve. The paper estimates a 1.6× additional geomean speedup from migrating to V80 (Appendix H.1), but this is a synthesis-based estimate, not a measurement. The actual benefit of a modern FPGA would depend on whether the memory hierarchy sizes (BRAM/URAM) scale proportionally, whether the clock frequency improvement translates to linear throughput gains for the specific streaming dataflow designs, and whether the power efficiency (Joules per operation) improves with the process node.
Interconnect bandwidth: The PCIe 3.0 P2P link (32 GB/s peak) is sufficient for the current prototype because compute time dominates communication time by 10–200,000× (Appendix C.1). However, if the FPGA kernels were accelerated (e.g., through process scaling or multi-FPGA parallelism), the communication-to-computation ratio would shrink proportionally. At some acceleration factor, PCIe would become the bottleneck, and the two-criteria mapping policy would force more operations to remain on the GPU despite being FPGA-friendly. The paper does not characterize the FPGA speedup ceiling before PCIe saturation, making it impossible to determine whether a next-generation FPGA (with, say, 3× the U55C's throughput) would still see net speedups or would be communication-limited.
GPU model and memory bandwidth: The MI210's 1.6 TB/s HBM bandwidth is competitive with the A100 (2.0 TB/s) but substantially lower than the H100 (3.35 TB/s). A GPU with higher HBM bandwidth would reduce the FPGA's bandwidth advantage for memory-bound stages, potentially changing which pipeline stages benefit from offloading. The A100 estimates in Appendix H (Figures 25-26) partly address this, but they are derived rather than measured, and do not cover H100 or B200 class GPUs. The paper's claim that the heterogeneous approach "is largely agnostic to the specific GPU model" (Appendix H) is supported only for A100 vs. MI210, not for the broader GPU landscape.

What evidence exists in the paper. Appendix H (Figures 25-26) provides estimated results for A100+U55C using DeepSeek Attention, showing that the heterogeneous speedup is maintained but with different absolute magnitudes. Appendix H.1 provides a synthesis-based estimate of V80 improvement (1.6× additional speedup) but no measurement. The PCIe transfer analysis in Appendix C.1 quantifies current communication overhead as negligible but does not project at what FPGA speedup factor communication would become a bottleneck. The dynamic fallback discussion (Section 6.2, Appendix F) addresses FPGA HBM bandwidth limitations at extreme sequence lengths but does not analyze scenarios where GPU bandwidth improvements shift the crossover point to shorter sequence lengths.

Mitigation status. Partially addressed through estimation and explicit acknowledgment. The paper is transparent about its hardware limitations and provides projections for both GPU upgrade (A100) and FPGA upgrade (V80). However, these are estimates, not measurements, and the lack of a co-located A100+U55C platform means that practical integration issues—PCIe topology differences, NUMA effects, DMA engine contention, thermal throttling—are not captured. The paper does not provide a sensitivity analysis showing how speedup varies with GPU HBM bandwidth, FPGA process technology, or interconnect bandwidth, which would allow practitioners to assess applicability to their specific hardware configuration.

6.3 The Pipeline Abstraction Is Validated Through Classification, Not Counterfactual Experiment

The assumption or constraint. The paper's central analytical contribution—the four-stage memory processing pipeline (Prepare Memory, Compute Relevancy, Retrieval, Apply to Inference)—is validated by showing that all representative methods in Table 1 can be decomposed into these stages. This is a classification exercise: the paper demonstrates that the pipeline can describe the methods, not that this description produces better outcomes than alternative decompositions. The implicit claim is that the four-stage decomposition is the right abstraction for hardware mapping—that grouping operations along these stage boundaries rather than others (e.g., grouping by operation type: all matrix multiplications together, all reductions together) leads to optimal heterogeneous mapping. This claim is never tested against an alternative decomposition.

The consequence. A practitioner cannot determine whether the achieved speedups are attributable to (a) the specific four-stage pipeline taxonomy, (b) the general idea of offloading memory-bound irregular operations to FPGA (which could be identified through standard roofline analysis without the pipeline abstraction), or (c) simply careful FPGA kernel engineering on the identified bottlenecks, regardless of how they are categorized. If alternative groupings—for example, treating the entire sparse attention mechanism (Prepare + Compute Relevancy + Retrieval + Apply) as a monolithic offload target, or splitting it along operation-type boundaries (all linear projections together, all scoring together, all selection together)—would produce comparable or better speedups, then the pipeline abstraction's value as a mapping methodology (as opposed to a descriptive framework) is not empirically established.

This matters for the paper's claim that the pipeline supports reusability: "Existing methods use provided host binaries, while users can build new method by recombining kernels and interfacing them through our GPU-FPGA communication API" (Section 5.4). If the pipeline stages do not actually represent optimal partition boundaries for hardware mapping, then recombining kernels along these boundaries may produce suboptimal results for new methods, and the library's value as a general acceleration platform is limited.

What evidence exists in the paper. The paper provides no ablation comparing the four-stage mapping against alternative mappings. The closest indirect evidence is the arithmetic intensity measurement (Figures 13-14 in Appendix B), which shows that the four stages have distinct computational characteristics. This supports the claim that the stages are computationally meaningfully different, but does not demonstrate that grouping operations this way is superior to other groupings. The library composability claim is mentioned but not experimentally validated; no methods beyond the seven in Table 1 are implemented by recombining kernels, so there is no evidence that the pipeline-based kernel interfaces actually support efficient composition.

Mitigation status. Not addressed. The paper treats the pipeline abstraction as a validated contribution based on its descriptive completeness (Table 1 covers all methods) and its consistency with the hardware mapping decisions (stages with similar computational properties receive similar mappings). The missing counterfactual—what if operations were grouped differently?—is a conceptual gap that limits the strength of the methodological claim. Future work on design automation (Section 5.4) could explore this by automatically searching over possible operation partitionings, but this is not proposed in the paper.

6.4 Batch Size Sensitivity Renders the Approach Unsuitable for Throughput-Oriented Serving

The assumption or constraint. The paper's evaluation focuses primarily on batch-size-1 (BS=1) inference, which is the latency-sensitive regime. While Table 4 reports geomean speedup across batch sizes 1–32, the trends reveal a fundamental tension: the heterogeneous system's advantage is inversely correlated with batch size for Memory as Context (declining from 1.48× at BS=1 to 1.15× at BS=32), and reverses entirely for MemAgent (from 1.85× speedup at BS=1 to 0.13×—an 8.7× slowdown—at BS=32). The paper acknowledges this for MemAgent: "The system can dynamically select the optimal configuration. For example, when the batch size is larger than 2 in MemAgent, we switch to a GPU-centric deployment to avoid slowdown" (Section 6.4).

The consequence. The heterogeneous system's applicability is sharply bounded: it benefits latency-sensitive serving (batch size 1–2) but provides no benefit—or actively harms performance—in throughput-oriented serving where batching is essential for cost efficiency. This is a structural limitation, not a surmountable engineering challenge. LLM decoding is memory-bound at batch size 1 because each weight byte is used for a single computation per token; at larger batch sizes, the same weight byte is used across multiple batch samples, shifting the operation toward compute-bound territory where GPUs excel. FPGAs, with lower raw arithmetic throughput (fewer DSP slices, lower clock frequency, no tensor-core equivalents), cannot match this batched throughput scaling. The result is that the heterogeneous system addresses only one deployment regime (interactive, low-concurrency serving) but not the other (batch processing, high-throughput serving), and even in the interactive regime, the advantage depends on the concurrency model—if multiple requests are batched to improve hardware utilization, the FPGA advantage diminishes.

For RAG and sparse attention methods, the speedup increases with batch size (Table 4), which appears to contradict this limitation. However, this is because the offloaded operations (BM25 scoring, inner product with compressed keys) do not benefit from batching on GPUs—the key vectors and documents are sample-specific and cannot be shared. As batching improves the dense GPU operations' throughput, the unaccelerated memory processing stages become a larger fraction of total latency, amplifying the FPGA's relative contribution. The net end-to-end speedup is still positive at large batches, but the absolute latency per request is dominated by the (unaccelerated) memory processing, meaning the system is fast relative to the GPU baseline but still bottlenecked on the same operations. In other words, the FPGA offloads scale with batch size for these methods, but they do not solve the batch-throughput problem—the absolute throughput remains limited by the offloaded operations' throughput on the FPGA (for single-FPGA systems) or by the GPU's memory bandwidth (for the non-offloaded portions).

What evidence exists in the paper. Table 4 provides geomean speedup across batch sizes for all methods. The MemAgent reversal is stark: BS=1 (1.85×), BS=2 (1.65×), BS=4 (0.93×), BS=8 (0.49×), BS=32 (0.13×). Memory as Context declines monotonically: 1.48× → 1.47× → 1.45× → 1.38× → 1.15×. Sparse attention and RAG increase with batch size, but the paper does not report absolute throughput (requests per second) or latency-at-batch, only relative speedup, making it impossible to assess whether absolute performance is acceptable for production serving. The dynamic fallback for MemAgent is stated but not measured (no transition latency, no batch-size detection mechanism described).

Mitigation status. Partially addressed through explicit acknowledgment and the proposed dynamic fallback mechanism. The paper does not claim to solve throughput-oriented serving and is transparent about the batch limitations. However, the headline speedup numbers (Section 1, Abstract) emphasize batch-size-1 results ("up to 2.2× faster and 4.7× energy reduction") without qualification, which could mislead practitioners who deploy in batched serving environments. The dynamic fallback for MemAgent is described but not validated, and no fallback is proposed for Memory as Context despite its declining speedup with batch size. The paper does not discuss hybrid strategies (e.g., running some batch samples on FPGA, some on GPU, or using multiple FPGAs to improve throughput for the offloaded stages) that could partially address the throughput limitation.

6.5 Energy Measurement Methodology and Reproducibility Are Under-Specified

The assumption or constraint. The paper reports energy efficiency as a primary result: 1.11–1.61× geomean energy reduction for sparse attention, 1.07–1.21× for RAG, 4.66× for MemAgent, and 1.65× for Memory as Context (Table 3), with per-kernel power measurements listed in Appendix G (FPGA kernels: 24.9–44.2 W; GPU kernels: 45–106 W). However, the paper does not describe how power is measured: whether through onboard hardware sensors, external power meters, vendor software tools (e.g., ROCm-SMI for AMD GPUs, Xilinx Board Support Package for FPGAs), or estimation from thermal design power (TDP). The measurement granularity (instantaneous vs. average over the kernel execution), inclusion of idle/static power (the FPGA board, GPU board, and CPU system draw power even when idle), and measurement of PCIe PHY and HBM power (which contribute to total system power but may not be attributed to specific kernels) are all unspecified.

The consequence. The energy efficiency numbers in Table 3 cannot be reproduced, compared against other work, or reliably used for cost estimation in deployment planning without knowing the measurement methodology. Three specific uncertainties arise:

Static power inclusion: If only dynamic (kernel-execution) power is measured, the energy numbers understate the total system energy because idle power during non-memory-processing stages (GPU running rest-of-LLM operations, FPGA idle, PCIe link active) is excluded. If static power is included, the 4.66× MemAgent improvement may partly reflect the FPGA's lower idle power rather than its computational efficiency, and the per-Joule improvement would not scale proportionally if the FPGA were loaded more heavily.
Board-level vs. chip-level measurement: The U55C FPGA board includes the FPGA chip, HBM stacks, PCIe PHY, power regulators, and cooling—all drawing power. The MI210 GPU similarly includes the GPU die, HBM, and board peripherals. If power is measured at the board level (e.g., through PCIe slot power or external power supply), the numbers include peripherals that scale differently with workload. If measured at the chip level (through on-die sensors), they exclude board-level overhead.
Process technology normalization: The U55C is fabricated at 16 nm while the MI210 is at 6 nm—roughly two process nodes apart. A process-normalized comparison (energy per operation at equivalent technology) would show a much larger FPGA advantage, but the reported numbers are raw measurements that conflate architectural efficiency with process technology. Practitioners considering a future 6 nm FPGA cannot directly extrapolate from these results without knowing the technology contribution.

What evidence exists in the paper. Appendix G lists kernel power as point values (e.g., "U55C: 26.4 W, MI210: 55 W" for DeepSeek Attention) without units of measurement, instrumentation description, measurement duration, or uncertainty bounds. Table 3 reports geomean energy efficiency (J/request or J/token) as aggregate numbers without decomposition into dynamic vs. static, kernel vs. board-level, or computational vs. data-movement energy. The paper states (Section 6.3): "The energy reduction does not solely come from the speedup: the FPGA kernels have a lower operating power than the corresponding GPU kernels," implying that power is measured separately from latency, but the measurement apparatus is not described.

Mitigation status. Not addressed. This is a methodological gap that affects the reproducibility and interpretability of the energy results. Standard practice in systems papers is to specify the measurement methodology (e.g., "board power measured via PCIe slot power sensors sampled at 1 kHz" or "GPU power reported via ROCm-SMI at 1-second intervals, averaged over 10 runs"). The paper provides none of this information. The energy results should be interpreted as approximate, and the 4.66× MemAgent improvement—the largest single result in the energy section—should be treated with particular caution given the unspecified methodology.

6.6 Evaluation Scope Is Limited to Representative Methods, Not Exhaustive Coverage of the Pipeline Space

The assumption or constraint. The paper evaluates seven specific methods across four categories (sparse attention, RAG, synthesized memory, Memory as Context), selected as "representative" of their categories (Section 6.1). The pipeline abstraction claims to generalize across "existing and emerging methods" (Table 1 caption: "Summary of LLM inference optimizations and the computations in their memory processing pipeline"), and the library-based deployment approach claims that "users can build new method by recombining kernels and interfacing them through our GPU-FPGA communication API (e.g., block-based RAG with BM25 and max-reduction kernels)" (Section 5.4). However, the pipeline space is not exhaustively characterized: the paper does not evaluate methods that differ in their stage composition (e.g., methods with multiple retrieval stages, methods with iterative Prepare–Compute–Retrieval loops), methods that use fundamentally different scoring functions (e.g., learned neural scorers other than cross-attention or inner product), or methods that blur the pipeline stage boundaries (e.g., end-to-end learned retrieval where scoring and selection are not separable). The library composition claim—that new methods can be built by recombining existing kernels—is unvalidated by any example beyond the seven explicitly implemented methods.

The consequence. A practitioner with a method outside the evaluated set cannot determine whether the heterogeneous system will provide speedup, or whether the existing kernel library can support their method without custom FPGA development. The paper's implicit claim is that if a new method fits the four-stage decomposition and its stages have computational characteristics similar to those profiled (memory-bound/irregular for Compute Relevancy and Retrieval, compute-bound/regular for Prepare and Apply), then the same mapping strategy will work, and the existing kernels can be reused. But this claim assumes that the kernel interfaces—the data formats, memory layouts, and communication patterns—are compatible across methods within the same stage. The paper does not specify these interfaces, so a practitioner cannot assess whether their method's stage outputs match the library's stage inputs without implementing it.

Concrete example: A method that uses learned binary hash codes for Compute Relevancy rather than inner product scores would still fit the pipeline abstraction (Prepare: generate hash codes; Compute Relevancy: Hamming distance computation; Retrieval: threshold-based selection; Apply: standard attention), and would still be memory-bound and irregular, but would require a completely different FPGA kernel (popcount-based distance rather than dot-product-based scoring). The existing inner-product kernel would not compose with this method, contradicting the library composability claim. The paper does not address this boundary: which variations within a stage can be handled by parameterization of existing kernels, and which require new kernel development.

What evidence exists in the paper. Table 1 provides the pipeline decomposition for seven methods plus TTT. The evaluation (Section 6) covers these seven methods. The library composability claim (Section 5.4) is illustrated only by a prose example and is not experimentally validated. No methods beyond those in Table 1 are implemented by recombining kernels. The paper acknowledges this implicitly: "Arbitrary methods may require custom kernels. We plan to reduce this effort via design automation in our future work" (Section 5.4). The TTT/LaCT exclusion demonstrates a principled boundary (insufficient heterogeneity), but this is a decision not to deploy, not a demonstration of the library's coverage or reusability.

Mitigation status. Partially addressed by explicit acknowledgment of the custom-kernel requirement for "arbitrary methods" and the call for future design automation. However, the paper's core marketing—the pipeline as a unifying abstraction, the library as a reuse platform—implies a generality that the evaluation does not support. The seven evaluated methods span the major categories of long-context LLM memory optimization, but they do not sample the space of pipeline variations systematically (e.g., all scoring functions are either inner product or BM25; all retrieval heuristics are top-k, threshold, or weighted sum; all memory representations are either compressed vectors or full text). A more comprehensive evaluation or a clearly specified interface contract for each kernel module would strengthen the composability claim, but neither is provided.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper initiates a shift from storage-centric taxonomies of LLM memory toward a processing-centric framework for LLM memory acceleration. Prior work (Wu et al., 2025; Zhang et al., 2025a) classified LLM memory by what it stores—parametric memory in weights, contextual memory in KV caches, external memory in vector databases—drawing parallels to human cognition and organizing around representation form and retention duration. This paper argues, through systematic profiling and hardware mapping, that the more operationally useful lens is what computation is performed on memory during inference: the four-stage pipeline of Prepare → Compute Relevancy → Retrieve → Apply to Inference. This reframing is not a taxonomy refinement but a diagnostic methodology shift: it moves the field from asking "what kind of memory does this method use?" to asking "which pipeline stages does this method execute, and what are their computational properties?"

The magnitude of this shift is incremental but catalytic. The pipeline abstraction does not reveal previously unknown computational properties—the arithmetic intensity measurements in Appendix B confirm patterns that experienced systems researchers would have intuited (scoring is memory-bound, retrieval is irregular). Rather, its value is unifying: it provides a shared vocabulary and diagnostic framework that reveals that DeepSeek Attention and DRAGIN, despite algorithmic dissimilarity, share the same stage-level computational bottlenecks (memory-bound Compute Relevancy + Retrieval) and therefore benefit from the same hardware mapping (GPU for the dense rest, FPGA for the fused scoring and selection). Prior to this work, each method was optimized in isolation; after this work, a single heterogeneous mapping strategy is shown to accelerate seven qualitatively different methods across four categories. This is the landscape change: what appeared to be method-specific engineering challenges are revealed as instantiations of a common computational pattern, amenable to a common acceleration strategy.

The paper also reconciles an implicit contradiction in the FPGA-for-LLM literature. Prior FPGA-based LLM accelerators (FlightLLM (Zeng et al., 2024), LUT-LLM (He et al., 2025c), GLITCHES (Yang et al., 2024a)) demonstrated that FPGAs can accelerate LLM decoding, but their scope was limited to standard transformer forward passes—they did not address the emerging class of long-context memory optimizations that are increasingly dominant in production inference. A practitioner could reasonably ask: "FPGAs help with standard decoding, but what about sparse attention? RAG? Compressed memory?" This paper answers that question empirically across the major categories, showing that the FPGA advantage extends beyond decoding into the memory processing pipeline—but only when the pipeline stages exhibit sufficient computational heterogeneity and communication overhead remains bounded. The exclusion of TTT/LaCT (Section 4) is the critical boundary that prevents overgeneralization.

The paper also shifts the narrative around what bottlenecks long-context LLM inference. The dominant prior narrative focused on attention computation itself—quadratic scaling in prefill, linear scaling with KV cache size in decode—as the primary challenge for long contexts. This paper's profiling (Figures 3-5) demonstrates that when advanced memory optimizations are applied (sparse attention, RAG, compressed memory), the non-attention memory management operations—indexing, scoring, retrieval, context assembly—can dominate total latency, reaching 81% of decoding time for sparse attention at 1M tokens and 97% for synthesized memory. This redirects optimization effort: building faster attention kernels addresses a shrinking fraction of total latency, while accelerating the memory processing pipeline addresses a growing fraction. The implication is that future long-context LLM systems should be designed with heterogeneous hardware from the start, treating memory processing as a first-class computational workload rather than an afterthought to be handled by the same GPU that runs matrix multiplications.

What becomes more attractive as a research direction: (1) Verifier or scoring model design that explicitly targets hardware efficiency—since Compute Relevancy is the dominant bottleneck across sparse attention, RAG, and Memory as Context, designing scoring functions that map naturally to FPGA or ASIC dataflow architectures (e.g., inner products with structured sparsity, hash-based scoring, quantized similarity) could yield algorithmic-hardware co-design wins. (2) Near-memory compute architectures that eliminate the PCIe bottleneck entirely—since the paper's two-criteria mapping policy prioritizes communication over kernel fit, architectures that colocate computation with memory (processing-in-memory for KV caches, near-storage retrieval for RAG) could expand the set of pipeline stages that are profitable to offload. (3) Automated design-space exploration for heterogeneous pipeline mapping—the paper's manual mapping based on profiling and two criteria is effective but does not guarantee optimality; an automated search over mappings (which stages to offload, which fusion boundaries to use, which memory tiers to assign) could discover configurations that outperform the paper's manually designed ones.

What becomes less attractive: (1) GPU-only optimization of memory processing operations—the paper demonstrates, across seven methods, that GPUs systematically underutilize their hardware on memory-bound, irregular stages, and that the gap widens with memory size. Incremental GPU kernel improvements (e.g., better caching, warp-level reductions) are unlikely to close the 5-6× gap shown for BM25 retrieval (Figure 10) or the 3-4× gap for Memory as Context cross-attention (Figure 11). (2) Unified accelerator architectures that treat all LLM inference operations as homogeneous—the computational heterogeneity documented in Table 2 is structural (different arithmetic intensities, different access patterns, different dependency structures), not accidental. An accelerator designed for one profile (e.g., dense matrix multiplication) will underperform on the other (irregular top-k retrievers), and vice versa. The paper's two-criteria mapping policy formalizes this: the optimal system is heterogeneous by design.

Follow-Up Research This Work Enables

Automated stage-boundary search for optimal pipeline partitioning. The paper manually partitions the memory processing pipeline at stage boundaries (Prepare on GPU, Compute Relevancy + Retrieval fused on FPGA, Apply on GPU), arguing that these are the natural boundaries because they correspond to shifts in computational properties. However, the paper never tests alternative partitionings—for example, fusing Prepare Memory with Compute Relevancy on FPGA (which would avoid the PCIe transfer of compressed keys but require the linear projections to run on the FPGA's lower-throughput DSPs), or splitting Retrieval across devices (top-k on FPGA, KV cache gathering on GPU, which is the paper's actual approach but justified by Criterion 2 rather than explored as a design choice). A follow-up study could implement a design-space exploration framework that automatically evaluates all feasible partitionings of the four-stage pipeline across GPU and FPGA, measuring end-to-end latency including communication, and determine which partitionings are Pareto-optimal under different sequence length, batch size, and model size regimes. The paper's kernel library (Section 5.4) provides the building blocks; what is needed is an automated search over how to compose them. The strong hypothesis—implicit in the paper but untested—is that the stage boundaries are the optimal partition points. A negative result (finding that intra-stage partitioning, e.g., splitting a single linear projection across devices, yields better performance) would refine the pipeline abstraction's role from a mapping methodology to a descriptive taxonomy that does not prescribe optimal hardware boundaries.

Scoring function co-design for FPGA-native relevance computation. The paper's General Setup FPGA kernel (Figure 7) implements inner-product and BM25 scoring, both of which are well-suited to streaming dataflow architectures because they decompose into parallel dot products with minimal control flow. However, the paper does not explore whether the scoring function itself can be designed to be more FPGA-friendly without sacrificing retrieval quality. A follow-up study could benchmark retrieval accuracy (e.g., recall@k on standard RAG benchmarks) against hardware efficiency (e.g., latency and energy per query on FPGA vs. GPU) for a range of scoring functions: (a) inner product with full-precision vectors (the paper's current approach), (b) inner product with binary or ternary quantized vectors (replacing multipliers with popcount or addition), (c) learned hash-based scoring where relevance is approximated by Hamming distance between compact codes, (d) small neural scorers (2-3 layer MLPs) that could fit entirely in FPGA on-chip memory. The empirical question is whether the accuracy gap between these FPGA-optimized scorers and the full-precision baseline is small enough that the hardware efficiency gain (potentially 10-100× higher throughput for binary scoring on FPGA) makes them Pareto-optimal for latency-sensitive deployment. This would extend the paper's insight—that computational heterogeneity matters for hardware mapping—into algorithmic co-design, where the scoring function is chosen because its computational profile matches the target hardware.

Multi-FPGA scaling for batch throughput and memory capacity. The paper's batch scaling results (Table 4) reveal a fundamental tension: the heterogeneous system's advantage is largest at batch size 1 and diminishes or reverses at larger batches for Memory as Context and MemAgent. For sparse attention and RAG, speedup increases with batch size (because the offloaded stages do not benefit from batching on GPUs), but the paper does not measure absolute throughput or explore whether multiple FPGAs could handle the offloaded stages at higher batch sizes. A follow-up study could deploy 2, 4, or 8 U55C (or Versal V80) FPGAs connected to a single GPU, partitioning the memory processing workload across FPGAs by sample (each FPGA handles a subset of the batch) or by memory shard (each FPGA stores a partition of the key vectors or document corpus). The key measurement would be throughput scaling—does adding FPGAs improve batch throughput linearly, or does the GPU become the bottleneck? For MemAgent specifically, where batch size >2 causes FPGA slowdown, could two FPGAs running decoding in parallel match the GPU's batched throughput at batch size 4, making heterogeneous deployment viable at higher concurrencies? This would directly address the paper's primary deployment limitation (Section 6.4).

Online difficulty and workload estimation for dynamic offloading decisions. The paper proposes dynamic fallback—switching from FPGA to GPU-only when sequence length exceeds 1M tokens for LServe/DeepSeek Attention, or when batch size exceeds 2 for MemAgent—but does not implement or measure the fallback mechanism (Section 6.2, Appendix F). A critical missing component is the decision procedure: what signals should the system monitor to decide when to offload vs. fall back, and what is the monitoring overhead? A follow-up study could implement a runtime system that monitors (a) current batch size, (b) sequence length, (c) FPGA on-chip memory utilization (fraction of BRAM/URAM consumed by current key vectors), and (d) FPGA HBM bandwidth utilization (measured via hardware performance counters on the U55C), and uses these signals to make per-request offloading decisions with a simple threshold-based policy. The key evaluation metrics would be: (1) decision accuracy—does the runtime correctly predict when FPGA execution would be slower than GPU-only? (2) decision overhead—what is the latency cost of monitoring and decision-making relative to the end-to-end request latency? (3) robustness—does the policy work for unseen methods (e.g., a new sparse attention variant not in the training set) or does it overfit to the profiled methods? This would transition the paper's static profiling-based mapping to a practical runtime system.

Pipeline-aware model architecture design for heterogeneous hardware. The paper evaluates on fixed model architectures and inference optimizations, treating the pipeline stages as given and optimizing their execution. A more ambitious follow-up would co-design the model architecture and the hardware mapping: for example, when training a sparse attention indexer, add a hardware-aware regularization term that penalizes scoring functions with high arithmetic intensity (which would be GPU-bound) or irregular memory access patterns (which cause GPU cache thrashing), steering the learned indexer toward FPGA-friendly computational profiles. Concretely, for DeepSeek Attention, the indexer uses 64 query heads with weighted-average reduction; an architecture with fewer heads (e.g., 8 heads) would reduce the arithmetic intensity of the Compute Relevancy stage, making it even more memory-bound and thus more FPGA-advantaged, potentially at the cost of retrieval accuracy. The follow-up would measure the accuracy-efficiency Pareto frontier: how much retrieval accuracy is sacrificed for how much latency reduction, and whether hardware-aware training discovers architectures that the paper's fixed-model evaluation misses. This extends the paper's core thesis—that memory processing has computational structure amenable to heterogeneous acceleration—into the model design phase itself.

Practical Applications and Downstream Use Cases

Latency-critical long-context serving for interactive applications. The most direct application is deploying the GPU-FPGA system for interactive LLM services that require low-latency responses on long contexts—for example, a coding assistant that needs to retrieve relevant code snippets from a 1M-token repository (using sparse attention or RAG), a document Q&A system that queries a 20M-document corpus (using two-stage RAG), or a creative writing tool that maintains compressed memory of a novel-length narrative (using Memory as Context). For these applications at batch size 1, the paper's results translate directly: 1.04-1.49× lower per-token latency for sparse attention (Figure 8), 1.14-1.84× lower per-request latency for RAG (Figure 10), and 1.3-1.6× lower latency for Memory as Context (Figure 11). The key deployment consideration is that the FPGA must be co-located with the GPU (same node, same PCIe root complex) and the FPGA's HBM must be large enough to hold the compressed memory for the target sequence length (40 MB BRAM+URAM suffices for ~65K tokens of DeepSeek-style compressed keys; beyond that, HBM capacity of 16 GB on U55C becomes the limit).

Energy-efficient LLM serving for cost-sensitive deployments. The paper reports 1.11-4.66× geomean energy reduction per request across methods (Table 3), with the largest gain for MemAgent (4.66×). For cloud LLM providers where energy cost is a significant fraction of total serving cost, replacing a fraction of GPU servers with GPU-FPGA heterogeneous nodes could reduce the energy bill proportionally for workloads dominated by memory processing. The specific scenario: a provider serving MemAgent-based long-context summarization at batch size 1-2 could achieve approximately 4.66× lower energy per request, or equivalently serve 4.66× more requests within the same energy budget. The caveat (Section 6.4) is that this only holds at low batch sizes—if the provider batches requests to improve GPU utilization, the MemAgent advantage reverses, and the system should fall back to GPU-only. This makes the deployment economically viable only for interactive (low-concurrency) serving, not batch processing.

Custom hardware roadmapping for LLM inference ASICs. The paper's profiling and heterogeneity analysis provides quantitative justification for including FPGA-like programmable logic or specialized memory-processing units in future LLM inference ASICs. Chip designers considering a heterogeneous LLM accelerator can use the paper's arithmetic intensity breakdowns (Table 2, Appendix B) and latency fraction analysis (Figures 3-5) to estimate the die area and power budget that should be allocated to memory-processing units versus dense matrix-multiplication units. Specifically: for a chip targeting 1M-token sparse attention inference at batch size 1, memory processing accounts for 22-81% of latency (Figure 3), and the Compute Relevancy + Retrieval stages are 1-10 FLOPs/byte (memory-bound), suggesting that a dedicated on-chip unit with high-bandwidth local SRAM (similar to the FPGA's BRAM+URAM) and streaming inner-product and top-k datapaths would provide end-to-end speedup roughly proportional to the fraction of latency offloaded from the main tensor cores. The paper's U55C power measurements (24.9-44.2 W, Appendix G) provide a baseline for estimating the power budget of such a unit, and the Versal V80 projection (1.6× additional speedup, Appendix H.1) provides a technology-scaling estimate.

When to Prefer This Method

The paper does not position the GPU-FPGA heterogeneous system against a specific named alternative (e.g., CPU-offloading, pure-ASIC, multi-GPU) with an explicit tradeoff analysis. The implicit comparison is against GPU-only execution, and the paper's own results bound the conditions under which heterogeneous execution is preferable. Based on the paper's empirical findings, the decision rule is:

Prefer the GPU-FPGA heterogeneous system when: (1) the workload runs at low batch size (1-2 for MemAgent, 1-4 for Memory as Context, any batch size for sparse attention and RAG per Table 4), (2) the memory processing fraction exceeds approximately 20% of end-to-end latency (since Amdahl's Law bounds end-to-end speedup as 1/(1 - fraction), and the paper's measured kernel speedups of 2-6× yield net speedups above 1.1× only when the offloaded fraction is above 15-20%), (3) the sequence length or document count is large enough that memory processing is the bottleneck (1M tokens for sparse attention, 500K+ documents for RAG, per Figures 3-4), but not so large that FPGA HBM bandwidth becomes the limit (beyond 1M tokens for LServe/DeepSeek Attention, Section 6.2), and (4) the FPGA process technology is competitive with the GPU's (the paper shows advantage even with a 16 nm FPGA paired with a 6 nm GPU, suggesting this condition is permissive but not guaranteed for all FPGA/GPU pairings).
Prefer GPU-only execution when: (1) the workload runs at batch size >2 for MemAgent (>4 for Memory as Context, per Table 4 trends), (2) the memory processing fraction is below ~15% (e.g., short-context sparse attention at 4K tokens where memory processing is 1-11% of latency, Figure 3—Amdahl's Law limits end-to-end speedup to <1.1×), (3) the sequence length exceeds the FPGA on-chip memory capacity and the GPU's higher HBM bandwidth makes GPU-only execution faster (the dynamic fallback condition in Section 6.2, demonstrated for LServe at 1M tokens where speedup drops to 1.2×, Figure 9), or (4) the method's memory processing lacks sufficient computational heterogeneity—specifically, if the bottleneck stages are compute-bound regular operations (as with TTT/LaCT, Section 4), the GPU's raw arithmetic throughput dominates and FPGA offloading provides no benefit.