Hierarchical SVG Tokenization: Learning Compact Visual Programs for Scalable Vector Graphics Modeling

ArXiv: 2604.05072

🎯 Pitch

Treating SVG coordinates as digit-by-digit text fragments causes large language models to hallucinate and wastes over half the training tokens—HiVG cuts this bloat by 63% while raising generation quality, simply by tokenizing complete geometric commands like Bézier curves as single units.

1. Executive Summary

This paper proposes HiVG, a hierarchical SVG tokenization framework for autoregressive vector graphics generation that decomposes raw SVG strings into structured atomic tokens and then compresses executable command–parameter groups into geometry-constrained segment tokens, substantially reducing sequence length while preserving syntactic validity. The approach is evaluated using Qwen2.5-VL-3B-Instruct on a merged corpus of 2.45M SVG samples, and combines three named mechanisms: Structure Segment Learning (merging frequent command–coordinate groups into reusable geometric primitives, such as compressing a cubic Bézier curve with its six coordinate parameters into a single <SEG> token), Hierarchical Mean–Noise (HMN) initialization (injecting numeric ordering signals into new token embeddings via a Gaussian–polynomial basis, so that coordinate tokens like <P_20> and <P_21> preserve spatial proximity in embedding space), and a three-stage curriculum training paradigm that progressively increases SVG program depth. HiVG achieves 62.7%–63.8% sequence compression (2.68×–2.76× fewer tokens) and reaches comparable generation quality with approximately 2.7× fewer training tokens, establishing that aligning tokenization with executable geometric structure—rather than treating SVG as flat byte streams—provides a principled foundation for reliable vector graphics modeling, though this compression operates on problems within the base model's capability range as measured by rendering fidelity on icon and logo-style graphics.

2. Context and Motivation

The Core Problem: Tokenization Is the Bottleneck in SVG Generation

The fundamental question this paper tackles is deceptively simple: what is the right unit of representation for generating vector graphics with language models? Every autoregressive model—whether generating natural language, code, or structured data—must first convert its target output into a sequence of discrete tokens. For most text-to-SVG and image-to-SVG systems built on LLMs, this tokenization step inherits directly from natural language processing: the SVG string is fed into a generic byte-level tokenizer (e.g., the BPE tokenizer from Qwen or another pretrained LLM), which treats the program as flat text and splits it into subword fragments.

The paper argues—and provides extensive empirical evidence—that this inherited approach is fundamentally misaligned with the geometric nature of vector graphics. Raw SVG strings are dominated by numeric coordinate sequences: a simple cubic Bézier curve like C 20 -20 50 30 80 0 becomes fragmented into individual digits and punctuation marks (e.g., C, 2, 0, -, 2, 0...). A modest icon containing a few dozen drawing commands can easily balloon into hundreds of tokens, most of which encode low-level coordinate digits rather than semantically meaningful geometric operations. This creates three interconnected problems that the paper identifies as the key barriers to reliable SVG generation:

Problem 1: Destroyed Spatial Relationships. When 100 is tokenized as three separate tokens—"1", "0", "0"—the tokenizer has no mechanism to represent that these digits form a contiguous spatial value, let alone that 100 is numerically close to 101 but far from 50. The model must learn to reconstruct spatial continuity from discrete, unordered symbols. This is not a minor inconvenience: the paper explicitly connects it to the well-documented phenomenon of coordinate hallucination in LLM-based SVG generation (citing Huang et al., 2024 and OmniSVG by Yang et al., 2025), where generated SVGs contain geometrically implausible or self-intersecting paths because the model treats coordinate values as arbitrary text tokens rather than continuous spatial quantities.

This is worth unpacking: in natural language, subword tokenization is not just acceptable but beneficial—"running" and "runner" sharing the token "run" captures morphological relationships, while the fact that "1999" and "2000" tokenize differently is irrelevant because language models don't need to understand numeric proximity for most NLP tasks. But in SVG generation, numeric proximity is the task. A Bézier curve control point at coordinate 100 should behave similarly to one at coordinate 101, and the model should smoothly interpolate between them. Standard tokenization provides no inductive bias toward this behavior, forcing the model to learn spatial continuity entirely from data—an expensive proposition that the paper's training efficiency results (Fig. 1b, showing 2.7× less training data needed with structured tokenization) quantify directly.

Problem 2: Severe Token Redundancy. The paper quantifies this in Figure 1a: converting from raw-string tokenization to HiVG reduces sequence length by 62.7%–63.8%, corresponding to 2.68×–2.76× compression. This is not merely an efficiency concern—though reducing training and inference costs by nearly 3× is significant—but a representational one. When a single cubic Bézier curve consumes roughly 10–15 raw tokens, a moderate-length SVG with 20 curves already approaches 200–300 tokens just for path data. In an autoregressive framework where the model attends to all previous tokens, this means that structural tokens (opening/closing tags, element boundaries) become vanishingly sparse relative to coordinate noise. The model's attention budget is dominated by low-level numeric tokens, making it harder to learn long-range dependencies—like ensuring that two paths don't overlap inappropriately or that a gradient stop aligns with a path endpoint.

The authors frame this in Section 1 as a contradiction between SVG's appeal and its realization:

"even a simple SVG shape may be represented by a long string of drawing commands and coordinates, which may expand to tens or even hundreds of tokens after tokenization. Such redundancy contradicts the structural compactness that makes SVG appealing in the first place."

The irony is sharp: SVG's primary selling point over raster formats is its compact, resolution-independent representation, yet the tokenization pipeline inflates it into something far less efficient than it needs to be.

Problem 3: Structural Validity Is Not Guaranteed. In a byte-level tokenizer, nothing prevents the model from generating a syntactically invalid SVG fragment—a path command with the wrong number of parameters, unclosed tags, or coordinates outside the canvas bounds. The model must learn SVG syntax implicitly from training data, and failures manifest as rendering errors or unparseable output. The paper's "Render" metric in Tables 5 and 6 captures this directly: even at the final training stage, the baseline achieves only 87.69%–90.41% render success on the hardest length bucket, meaning roughly 10% of generated SVGs cannot be rendered at all.

This is a particularly acute problem because SVG generation sits at the intersection of two domains—program synthesis and visual generation—where correctness has different meanings. In program synthesis, a program either compiles/runs or it doesn't; validity is binary. In visual generation, a generated image can be "mostly right" even with minor artifacts. SVG generation inherits both constraints: an invalid SVG renders to nothing (program synthesis failure), but even a valid SVG can produce a visually incoherent result (generation failure). The challenge is to design a tokenization scheme that preserves both executable validity and visual-semantic structure.

Why This Problem Matters Now

The paper's timing is motivated by a significant shift in how the field approaches vector graphics generation. The authors trace this evolution in Section 2.1:

Pre-LLM era: differentiable rendering and optimization. Methods like CLIPDraw (Frans et al., 2022), DiffSketcher (Xing et al., 2023), and Clipasso (Vinker et al., 2022) treated SVG generation as a continuous optimization problem: start with random path parameters, render through a differentiable rasterizer, compute a loss against a target image or text embedding, and backpropagate to update the parameters. These approaches produce high-fidelity results but suffer from fundamental limitations: they optimize one SVG at a time (no amortized inference), they scale poorly with the number of primitives, and they struggle to model compositional structure—the optimizer finds a visually plausible arrangement of strokes but doesn't understand that a "house" should decompose into a roof, walls, and a door as separable semantic parts. Moreover, the optimization process is slow, typically requiring thousands of iterations per image.

LLM era: autoregressive program synthesis. The paper positions itself within a newer paradigm where SVG is treated as executable code generated by a pretrained LLM finetuned on SVG corpora. Representative works include StarVector (Rodriguez et al., 2025), LLM4SVG (Xing et al., 2025), and OmniSVG (Yang et al., 2025). These approaches amortize the generation cost (one forward pass produces the entire SVG) and naturally inherit the LLM's understanding of compositional structure from its pretraining. The catch—and this is the paper's central motivating observation—is that every one of these methods still uses the pretrained LLM's generic text tokenizer to represent SVG.

The authors explicitly state:

"Alongside this architectural shift, we observe a concerning trend: existing methods inherit coordinate representations from the pre-trained LLM, which often leads to coordinate hallucination."

This suggests that the field has been rapidly improving the model architectures and training recipes for SVG generation while neglecting the fundamental representation problem. The paper's contribution is to argue that tokenization is not a trivial preprocessing detail but a first-class design decision that determines how efficiently the model can learn geometric relationships, how compact the generated sequences become, and how reliably the output preserves structural validity.

The real-world stakes are spelled out through the evaluation: HiVG achieves 4.06 on a 5-point usability scale and wins 58.9%–70.8% of pairwise comparisons against baselines in human evaluation (Fig. 1c). This matters because SVG generation is not merely an academic benchmark—it has practical applications in icon design, logo generation, UI prototyping, and data visualization, where the generated output needs to be editable in tools like Adobe Illustrator. The paper's usability review (Table S1) demonstrates that professional practitioners rate HiVG's generated SVGs as significantly more semantically layered, editable, and less redundant than competing methods. This is a tangible measure of whether the generated code is actually useful downstream, not just whether it renders correctly.

Where Prior Approaches Fall Short

The paper identifies three categories of prior work, each with specific limitations that HiVG is designed to address:

1. Generic Text Tokenizers for SVG (the Dominant Paradigm)

The default approach in LLM-based SVG generation—used by StarVector, OmniSVG, and InternSVG—is to feed the raw SVG string through the pretrained model's BPE tokenizer. This treats SVG as unstructured text. The paper illustrates the consequence in Figure 2a: a simple path element <path d="M 10,20 c 20,-20 50,30 80,0"> expands to 10 tokens, with coordinate digits split across multiple subword units. The paper's characterization is blunt:

"tokenized coordinates fail to reflect their underlying geometric relationships."

Some prior works have attempted to mitigate the worst symptoms through preprocessing. LLM4SVG (Xing et al., 2025) converts coordinates to relative form to reduce vocabulary fragmentation. OmniSVG (Yang et al., 2025) flattens coordinates to reduce nesting depth. But the paper argues these are surface-level patches that don't address the root cause:

"the fundamental problem remains unresolved... the devil is in the token compression."

Specifically, these preprocessing steps don't change the fact that each coordinate digit remains a separate token with no inherent spatial relationship to neighboring digits. A relative coordinate like d_-20 in LLM4SVG's scheme might tokenize as d, _, -, 2, 0—still five tokens, still fragmented. The structural compression benefit is minimal.

2. SVG-Aware Tokenization Without Hierarchical Compression

A step up from generic text tokenizers are approaches that explicitly tokenize SVG structural elements: opening tags, closing tags, attribute names, and path commands become separate tokens. This is illustrated in Figure 2b and is the approach used by both LLM4SVG and InternSVG. The improvement is real but insufficient: the tokenizer now "understands" that <path> is a structural boundary and that M is a move command, but geometric primitives remain fragmented into many numeric coordinate tokens. In Figure 2b, the same path element produces 7 tokens—better than 10, but the 4 path-parameter tokens still consume more than half the sequence length.

The paper's critique of this category is that it stops halfway: it introduces domain awareness at the structural level but not at the geometric primitive level. Commands and their coordinate parameters remain decoupled, so the model must still learn to associate a command token with the correct number of subsequent coordinate tokens—a long-range dependency that becomes harder as sequences grow longer.

3. Implicit or Continuous Representations (Non-Autoregressive)

Earlier work like DeepSVG (Carlier et al., 2020) and StrokeNUWA (Tang et al., 2024) projected SVG commands and coordinates into continuous latent spaces for learning compact implicit representations. While these approaches achieve compactness, they are incompatible with the autoregressive LLM paradigm that has become dominant. They don't produce discrete token sequences that can be predicted left-to-right, making them unsuitable for integration with pretrained language models.

The paper positions HiVG as sitting at the intersection of these prior approaches: it inherits the discrete, autoregressive compatibility of SVG-aware tokenization while achieving the compactness benefits of implicit approaches through hierarchical compression. The key insight is that the hierarchical compression is structure-preserving: unlike an autoencoder that might compress arbitrary byte sequences into an opaque latent code, HiVG's segment tokens correspond to explicit, renderable geometric primitives that can be decoded back to valid SVG commands deterministically. This preserves executability—a generated segment token always decodes to a syntactically complete drawing operation—while dramatically reducing sequence length.

The DeepSVG, LLM4SVG, and Cad-GPT Connection

The paper does not emerge from a vacuum. It builds on specific technical foundations that are important to understand:

DeepSVG (Carlier et al., 2020): Established the idea of representing SVG paths as sequences of drawing commands with associated parameters. While DeepSVG used a hierarchical VAE rather than autoregressive modeling, its decomposition of SVG into command-type and coordinate-value streams provided the conceptual blueprint that HiVG extends. The innovation in HiVG is to take this decomposition and apply it to token merging—identifying which command–coordinate groups recur frequently enough to warrant their own vocabulary entries.

LLM4SVG (Xing et al., 2025): Shares several authors with HiVG and provides the immediate predecessor. LLM4SVG serializes SVG elements into textual command tokens for autoregressive generation, establishing the baseline that HiVG improves upon. The key difference is that LLM4SVG operates at the level of individual commands and coordinates—each M command and each coordinate value is a separate token. HiVG's contribution is the segment-level merging that groups these into composite tokens.

Cad-GPT (Wang et al., 2025): Not an SVG method per se but conceptually parallel: Cad-GPT compresses 3D spatial parameters and 2D sketch coordinates into a 1D linguistic token space for CAD generation, demonstrating that geometric token compression improves spatial reasoning. HiVG applies a similar philosophy to the SVG domain, but with a crucial difference: the compression is geometrically constrained—only complete command–parameter groups that form renderable units are merged, ensuring that compression never sacrifices syntactic validity.

How HiVG Positions Itself

The paper's positioning can be understood along three axes:

Methodological: From flat text to executable geometry. The core claim is that the tokenizer should "natively align with the underlying properties of vector graphics" (Section 1). This means producing tokens that correspond to renderable geometric operations rather than character-level fragments. The analogy to natural language is instructive: BPE tokenization in NLP merges frequently co-occurring characters into subword units like "ing" or "tion" that carry morphological meaning. HiVG applies the same philosophy—frequent co-occurrences should be merged—but with a domain-specific constraint: the merges must respect SVG's syntactic structure. You can't merge half a command with its neighbor because the result wouldn't be executable. The paper calls this Structure Segment Learning (SSL) to emphasize that the learning is over segments (complete command–parameter units), not arbitrary token pairs.

Theoretical: Compression as representation quality. The paper invokes a deeper principle that has gained traction in the language modeling community (Section 2.2):

"A longstanding belief holds that compression is closely connected to intelligence, with some researchers suggesting that they are fundamentally equivalent."

This is a reference to work by Delétang et al. (2023) and Huang et al. (2024), which argue that better compression implies better modeling of the underlying data distribution. By this logic, a tokenization scheme that achieves 2.7× compression of SVG sequences isn't just more efficient—it's a better representation because it captures the recurring geometric patterns that make SVG programs structured rather than random. The paper's segment tokens (illustrated in Figure 4: Bézier curves, elliptical arcs, orthogonal corners, semicircular arcs) are essentially a learned vocabulary of geometric primitives that appear frequently across the training corpus, analogous to how BPE learns a vocabulary of common subwords in natural language.

Practical: Enabling smaller models for SVG generation. A striking aspect of the evaluation is that HiVG uses a 3B-parameter model (Qwen2.5-VL-3B-Instruct) and competes with or outperforms 7B and 8B models (SVGen-7B, OmniSVG-8B, InternSVG-8B) on multiple metrics. The human evaluation in Figure 1c shows HiVG-3B beating all larger baselines, and Table 1 shows it achieving the best SSIM (0.896 vs. next best 0.790), LPIPS (0.114 vs. next best 0.205), and CLIP-S (0.957 vs. next best 0.930) on image-to-SVG reconstruction. The paper implicitly argues that tokenization quality can substitute for model scale—a message with practical significance because it suggests that organizations without access to massive compute can still achieve competitive SVG generation by investing in better representation design rather than larger models.

The Specific Technical Gap: No Principled Token Merging for Structured Graphics

To be precise about what the paper adds: prior work had explored BPE-style merging for text (Sennrich et al., 2016), for continuous robot actions (FAST by Pertsch et al., 2025), and for 3D meshes (FreeMesh by Liu et al., 2025), but no prior work had applied structural token merging to SVG under geometric validity constraints. The paper's specific technical contribution is the constraint that merges can only occur over complete command–parameter pairs (Eq. 2: $s = (\texttt{<cmd>}, c_1, \ldots, c_k)$ with $k$ fixed by the SVG specification), which guarantees that every merged segment token remain executable. This is more constrained than BPE (which can merge any adjacent tokens) but more expressive than simple vocabulary expansion (which just adds domain tokens without compression).

The paper contrasts this with CAD-GPT's approach, which compresses spatial parameters but does so without the geometric validity guarantee—a CAD sketch coordinate can be merged with an unrelated token, potentially producing a token that doesn't correspond to a valid CAD operation. HiVG's constraint is SVG-specific and is what allows the deterministic decode: a <SEG> token always maps to a complete drawing operation with the correct number of parameters.

Connecting Back to the Executive Summary

The prior sections' executive summary notes that HiVG achieves 62.7%–63.8% sequence compression and 2.7× training token efficiency. This context section explains why those numbers matter: they're not just efficiency metrics but indicators that the tokenization has captured the geometric structure of SVG programs. The compression comes from recognizing that a cubic Bézier curve is a semantic unit that should be a single token, not from lossy compression that might sacrifice rendering fidelity. The training efficiency comes from the model not having to learn spatial continuity of coordinates from scratch—the HMN initialization and the segment-level structure provide inductive biases that make the learning problem easier, reducing the amount of data needed to reach a given quality level.

3. Technical Approach

3.1 Reader Orientation

HiVG is a hierarchical tokenization pipeline that converts raw SVG markup into a compact sequence of discrete tokens—first by parsing SVG into structured atomic tokens that separate commands, coordinates, and attributes, and then by compressing frequent command–coordinate groups into single segment tokens—so that an autoregressive language model can generate valid, high-quality vector graphics from text descriptions or reference images. The problem it solves is that standard LLM tokenizers fragment SVG's geometric structure into digit-level pieces (destroying spatial relationships and bloating sequence length), and HiVG's solution is to realign the unit of tokenization with the unit of rendering: each token either represents a complete, executable drawing operation or a structural/attribute marker, never a partial geometric quantity.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that operate sequentially from raw SVG ingestion to model training:

1. Atomic Tokenizer — parses a preprocessed SVG string and emits a sequence of atomic tokens drawn from four disjoint vocabularies: structure tokens (tags like <svg>, <path>), command tokens (path operators like <cmd_M>, <cmd_c>), coordinate tokens (quantized absolute positions like <P_100> and relative offsets like <d_-20>), and attribute tokens (colors, opacities). This decomposition guarantees that every token has a well-defined geometric or structural role, unlike generic byte-level tokenization where a single digit token could mean nothing on its own.

2. Structure Segment Learner (SSL) — ingests a large corpus of atomic-token sequences, identifies the most frequently co-occurring command–parameter groups (e.g., a cubic Bézier command followed by its six coordinate parameters), and merges them into composite segment tokens (e.g., <SEG_161>) under a strict geometric validity constraint: only groups that form a syntactically complete and executable drawing operation are eligible for merging. The output is a fixed vocabulary of segment tokens—essentially a library of reusable geometric primitives—that replaces lengthy coordinate sequences with single tokens during both training and inference.

3. Hierarchical Mean–Noise (HMN) Initialization — constructs the initial embedding vectors for all newly introduced tokens (both atomic and segment) by blending four signals: the mean embedding of the pretrained vocabulary, a small Gaussian perturbation, a semantic prior derived from the token's textual description via frozen model weights, and (for coordinate tokens only) a numeric embedding that encodes the scalar coordinate value through a Gaussian–polynomial basis projection. The purpose is to give the model spatial awareness from epoch zero—tokens for adjacent coordinates start near each other in embedding space—rather than requiring it to learn numeric ordering from scratch.

4. Extended LLM Backbone — the base autoregressive model (Qwen2.5-VL-3B-Instruct) with its embedding matrix expanded to include the new atomic and segment tokens. The vision encoder and multimodal projector remain frozen; only the language model parameters and the new token embeddings are trained. The model is finetuned with supervised learning across two tasks: text-to-SVG token generation and image-to-SVG token generation, both using unified instruction-style prompts.

5. Curriculum Training Scheduler — splits the training process into three sequential stages that progressively increase the maximum SVG sequence length: Stage 1 (30–326 tokens) aligns the new embeddings, Stage 2 (326–605 tokens) activates segment tokens and shifts learning to composite geometric units, and Stage 3 (605–1000 tokens) introduces full-length programs requiring long-range layout coherence. The model sees all three difficulty levels at each stage (to avoid catastrophic forgetting) but the distribution of lengths shifts progressively toward more complex sequences.

Information flow during training: raw SVG → data cleaning (filtering, baking transforms, quantizing coordinates) → atomic tokenizer → structure segment learner (offline, produces segment vocabulary) → hierarchical token sequence (mix of atomic + segment tokens) → HMN-initialized embeddings → Qwen2.5-VL-3B-Instruct with curriculum scheduler → autoregressive training.

Information flow during inference: text prompt or input image → instruction prompt template → Qwen2.5-VL-3B-Instruct with extended vocabulary → autoregressive generation of hierarchical tokens → deterministic detokenization back to SVG string → rasterization for viewing.

3.3 Roadmap for the Deep Dive

• First, the atomic tokenization scheme — how raw SVG strings are decomposed into four disjoint token categories and how coordinates are quantized, because this defines the foundational units from which everything else is built.
• Second, structure segment learning (SSL) — how the atomic token sequences are compressed into segment tokens via constrained BPE-style merging, because this is the core compression mechanism and the paper's primary technical innovation.
• Third, the HMN initialization strategy — the full mathematical construction of the four-component embedding initialization, because this is how the model acquires spatial awareness without wasting training compute on learning numeric ordering from data.
• Fourth, the curriculum training paradigm — the three-stage progression through increasing sequence complexity, what each stage optimizes, and the stage-to-stage transitions, because this is how the paper achieves stable optimization despite injecting hundreds of new tokens and doubling effective sequence length.
• Fifth, the training and inference setup — hyperparameters, prompt templates, multi-task formulation, and the separation of frozen versus trainable components, because these implementation details determine reproducibility and practical feasibility.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a representation design paper whose core idea is that SVG tokenization should operate at the granularity of complete, executable drawing operations rather than individual characters or even individual commands, and that this hierarchical representation—combined with spatially-informed embedding initialization and progressive curriculum training—enables a 3B-parameter model to generate SVGs that are both more compact and more structurally valid than those produced by larger models using generic text tokenization.

Atomic SVG Tokenization: The Four-Vocabulary Decomposition

The paper's first design decision is to replace byte-level text tokenization with a structured decomposition of SVG into four disjoint token categories. This is not "how to tokenize SVG better" so much as "what are the fundamental semantic types present in an SVG program, and how do we give each type a dedicated vocabulary so the model never has to infer type from context?"

The formal definition. The atomic vocabulary is the union of four disjoint sets:

$\mathcal{V}_{\text{atomic}} = \mathcal{V}_{\text{struct}} \;\cup\; \mathcal{V}_{\text{cmd}} \;\cup\; \mathcal{V}_{\text{coord}} \;\cup\; \mathcal{V}_{\text{attr}}$

where $\mathcal{V}_{\text{struct}}$ contains structure tokens that define SVG elements and hierarchical layout (e.g., <svg>, <path>, </svg>, </path>), $\mathcal{V}_{\text{cmd}}$ contains path operators (e.g., <cmd_M> for move-to, <cmd_c> for relative cubic Bézier curve), $\mathcal{V}_{\text{attr}}$ contains visual attributes (e.g., fill=#9f616a, opacity=0.5), and $\mathcal{V}_{\text{coord}}$ contains geometric positions encoded as discrete tokens.

What it computes: the decomposition takes a preprocessed SVG string (which has already undergone the cleaning pipeline described in Appendix A: inlining CSS, baking transforms, normalizing viewBox to 784×784, quantizing coordinates) and outputs a sequence where every token belongs to exactly one of the four categories. There is no ambiguity: a digit can only appear as part of a coordinate token, never as a raw character. A path command is always an explicit command token, never confused with attribute text.

Why this form: the key property is disjointness — the model never faces the problem of deciding whether the digit "2" is part of a coordinate, a color hex code, or a path count. In a generic BPE tokenizer, the token 2 could appear in all three contexts, forcing the model to use surrounding context to disambiguate. By separating vocabularies, the authors inject explicit type information into the token stream: <cmd_c> is unambiguously a drawing operation, <P_100> is unambiguously an absolute coordinate, and they appear in predictable syntactic patterns (a command token is always followed by exactly $k$ coordinate tokens, where $k$ is fixed by the SVG specification).

Coordinate vocabulary design. The coordinate vocabulary $\mathcal{V}_{\text{coord}}$ is further split into two sub-vocabularies: absolute position tokens (<P_0> through <P_794>) and relative offset tokens (<d_-794> through <d_794>). Given a canvas of size $784 \times 784$, raw coordinates are first normalized to the canvas range (so a coordinate value of 392, 392 would be the center), then uniformly quantized into discrete integer bins, each mapped to a single token. The 795 bins for absolute coordinates correspond to integer positions from 0 to 794 (the extra 11 bins beyond 784 account for out-of-bound tolerance, as described in Appendix A: "clamp minor numerical overflow within a tolerance range of ±10").

The relative coordinate vocabulary is larger (1589 tokens ranging from -794 to +794) because relative offsets can span the full canvas width in either direction. The paper notes that relative coordinates are used for all path parameters except the first command in each path, which uses absolute coordinates to establish the starting position. This choice is explicitly motivated:

"This representation reduces global translation variance and exposes recurring geometric patterns across SVG programs. As a result, relative coordinates tend to increase the frequency of repeated command–coordinate groups in the corpus, which facilitates the discovery of reusable geometric primitives during segment learning."

This is an important design insight: the choice of relative coordinates is not just for numerical stability but specifically to increase the statistical frequency of reusable patterns, making the downstream SSL step more effective. If every path used absolute coordinates, a cubic Bézier curve at position (100, 200) with control points (120, 180) and (150, 230) would look completely different—in token space—from the same curve shape at position (500, 300). But with relative coordinates, both curves produce the same command–parameter group: <cmd_c> <d_20> <d_-20> <d_30> <d_30> <d_0> <d_0> (or similar offsets), enabling the SSL algorithm to discover the repeating pattern.

Atomic vocabulary size. At the fixed canvas resolution of $784 \times 784$, the atomic vocabulary contains 2,450 tokens total, broken down as 2,384 coordinate tokens (795 absolute + 1589 relative) and 66 non-coordinate tokens. The non-coordinate set includes 42 structure tokens (21 SVG elements × 2 for paired open/close tags), 20 path-command tokens (10 SVG path commands × 2 for absolute/relative variants), and 4 arc-flag tokens (large-arc and sweep-flag, each with binary values 0 and 1). The explicit <large_{0,1}> and <sweep_{0,1}> tokens handle the elliptical arc command's flag parameters, which in raw SVG are compact binary flags that generic tokenizers would either split across tokens or absorb into adjacent numbers.

What happens to the SVG string, step by step. Given a cleaned SVG like <svg viewBox="0 0 784 784"><path fill="#9f616a" d="M 10,20 c 20,-20 50,30 80,0"></path></svg>, the atomic tokenizer produces: <svg> viewBox <path> fill=#9f616a <cmd_M> <P_10> <P_20> <cmd_c> <d_20> <d_-20> <d_50> <d_30> <d_80> <d_0> </path> </svg>. The sequence length is 14 atomic tokens. Compare this to a generic BPE tokenizer (Figure 2a) which would produce something like 25–30 subword tokens for the same SVG—the structural decomposition alone provides significant compression before any segment merging occurs.

The fixed-arity constraint. Every SVG path command has a fixed parameter arity defined by the specification: a move-to command (M or m) requires 2 coordinates, a line-to (L or l) requires 2 coordinates, a cubic Bézier curve (C or c) requires 6 coordinates (3 control/endpoint pairs), a quadratic Bézier (Q or q) requires 4 coordinates, an elliptical arc (A or a) requires 7 parameters (2 radii + x-axis-rotation + large-arc-flag + sweep-flag + 2 endpoint coordinates), and a close-path (Z or z) requires 0 coordinates. This fixed arity is what makes the concept of a "segment"—a command together with all of its required parameters—well-defined and is the foundation for the SSL algorithm that follows.

Structure Segment Learning (SSL): Token Merging Under Geometric Constraints

This is the core compression mechanism and the paper's primary technical contribution. The key insight is that BPE-style frequency-based token merging can be applied to SVG token sequences, but only if the merge candidates respect SVG's syntactic structure—specifically, merges must occur at segment boundaries, grouping a complete command with all its coordinate parameters into a single token.

Formal definition of a segment. A segment is defined as a command token together with all of its coordinate parameters:

$s = (\texttt{<cmd>}, c_1, \ldots, c_k)$

where $\langle\texttt{cmd}\rangle$ is a specific command token (e.g., <cmd_M>, <cmd_c>, <cmd_A>), $c_1, \ldots, c_k$ are the coordinate tokens that follow it, and $k$ is uniquely determined by the command type according to the SVG specification. For example, a move-to segment is (<cmd_M>, <P_x>, <P_y>) with $k=2$; a cubic Bézier segment with relative coordinates is (<cmd_c>, <d_dx1>, <d_dy1>, <d_dx2>, <d_dy2>, <d_dx3>, <d_dy3>) with $k=6$.

What the algorithm does. The SSL procedure operates on the multiset of segments extracted from atomic token sequences across the training corpus:

1. Extraction: For every SVG in the training corpus, the atomic tokenizer produces a sequence. The algorithm scans this sequence, identifies each command token, and groups it with the subsequent $k$ coordinate tokens (where $k$ is the known arity for that command) to form a segment $s$. These segments form a multiset $\mathcal{S} = \{s_1, s_2, \ldots\}$.

2. Iterative merging: At iteration $t$, the algorithm examines all adjacent segment pairs $(s_i, s_j)$ that appear consecutively in the corpus, counts their co-occurrence frequency, and selects the pair with the highest count:

$(s_i^*, s_j^*) = \arg\max_{(s_i, s_j)} \mathrm{count}(s_i, s_j)$

If the frequency of the selected pair exceeds a minimum threshold $f_{\min}$, the pair is replaced throughout the corpus with a new composite segment token, and a new vocabulary entry is created. The process repeats for $M$ merging iterations, after which the final segment token vocabulary is frozen.

3. Geometric validity constraint: Critically, merging is restricted to whole segments—the algorithm never merges across segment boundaries into adjacent structure or attribute tokens, and it never merges partial segments (e.g., a command token with only 3 of its 6 required coordinates). This constraint is what distinguishes SSL from standard BPE, which merges arbitrary adjacent token pairs regardless of their syntactic role. As the paper states:

"merging is restricted to segment boundaries, while structure and attribute tokens remain unchanged. As a result, all learned tokens correspond to renderable segment groups... ensuring syntactic validity and geometric coherence."

4. Cleaning degenerate segments: As part of the extraction process (discussed in Section 4.4E and detailed in Appendix D.1), the algorithm identifies and removes structurally degenerate segments that contribute no meaningful geometry: zero-move line segments (<cmd_L> <d_0> <d_0>), zero-radius arcs (<cmd_A> <d_0> <d_0> ...), zero-move Bézier curves (<cmd_C> <d_0> <d_0> <d_0> <d_0> <d_0> <d_0> <cmd_Z>), and redundant horizontal/vertical moves (<cmd_h> <d_0> <cmd_v> <d_0>). As shown in Figure 10(a), these cleaning operations remove approximately 0.86 commands per sample on average, with line-related commands (<l> at ~0.34 per sample) and cubic curves (<c> at 0.22 per sample) being the largest categories of removed noise.

What emerges from SSL. Figure 4 shows representative learned segment tokens and their interpretations:

• <SEG_1>: A composite pattern encoding an elliptical arc command with its 7 parameters, discovered because this specific arc configuration appears 16,288 times in the training corpus.
• <SEG_2>: A horizontal-line-to followed by a vertical-line-to with specific offsets, forming an orthogonal corner primitive (frequency 19,786).
• <SEG_3>: A cubic Bézier curve with its 6 coordinate parameters, representing a specific curve shape that recurs across many SVGs (frequency 15,229).
• <SEG_4>: An elliptical arc encoding a semicircular arc (frequency 19,712).

The frequencies indicate that these are genuinely common primitives in the icon/logo/UI domain, not rare one-off patterns. Figure 10(c) shows that cubic Bézier curves (<c>) are the dominant command type within learned segments, particularly in the mid-frequency bucket (51–200 occurrences) where they reach a 40% share. High-frequency segments (Top 50) also exhibit strong representations of arcs (<a>, 24%) and smooth curves (<s>, 22%).

Segment token length consistency. Figure 10(d) shows that the atomic token length of learned segments (i.e., how many atomic tokens the segment replaces) is remarkably stable across frequency tiers, with a median of approximately 9 atomic tokens. This means that a single <SEG> token is, on average, replacing roughly 9 atomic tokens—which themselves were already more compact than raw text tokens. The stability of this length across frequency buckets suggests that SSL is not over-merging rare patterns into giant opaque tokens but is consistently identifying compact, geometrically coherent units.

Corpus scale effects on SSL. Table 3 shows how the SSL corpus size affects downstream generation performance. The key finding: increasing the SSL corpus from 50k to 500k samples significantly improves metrics (e.g., CLIP-S jumps from 0.803 to 0.889, a 10.7% relative improvement on image-to-SVG), but further scaling to 1.5M samples yields diminishing returns (CLIP-S improves only from 0.889 to 0.910). The atomic-to-segment compression ratio (AT→ST column) remains stable around 1.03×–1.05× regardless of corpus scale, indicating that the segment vocabulary quality—not the compression ratio itself—is what improves with more data. Interestingly, the average token count actually decreases as the SSL corpus grows (from 618 at D500k to 552 at D1.5M), suggesting that larger corpora enable discovery of more broadly applicable segment patterns that replace rarer, more specific ones.

The 10→7→2 token compression in Figure 2. The paper's running example in Figure 2 illustrates the full compression pipeline for a specific path: <path d="M 10,20 c 20,-20 50,30 80,0">. With a generic text tokenizer (Figure 2a), this produces 10 subword tokens. With an SVG-aware tokenizer that separates elements and attributes but keeps coordinates as individual tokens (Figure 2b), it produces 7 tokens (the M 10,20 c 20,-20 50,30 80,0 coordinate string is partially grouped but still fragmented). With HiVG's segment-level compression (Figure 2c), the same path element becomes just 2 tokens: a structural <path> token and a single <SEG_161> token that encodes the entire M 10,20 + c 20,-20 50,30 80,0 sequence as one learned geometric primitive.

Why SSE is not just standard BPE applied to SVG. The crucial difference is the geometric validity guarantee. Standard BPE applied to atomic SVG token sequences might produce merges like <cmd_c> <d_20> (a command plus only its first coordinate parameter), which would break the fixed-arity structure and produce a token that doesn't correspond to any renderable operation. SSL's segment-boundary constraint prevents this: only complete command–parameter groups are eligible for merging, and merges always combine whole groups. The paper frames this explicitly:

"Different from the free-form combination or coordinate-level discretization seen in these prior works, we identify renderable units under geometric constraints to achieve structural token compression for SVG generation."

The consequence is that the detokenization step is deterministic and always produces syntactically valid SVG: a <SEG> token maps to a known sequence of atomic tokens which maps to a known sequence of SVG commands and coordinates. There's no ambiguity or generation-time validity checking needed—the tokenization scheme encodes executability by construction.

Hierarchical Mean–Noise (HMN) Initialization: Injecting Spatial Structure Into Token Embeddings

When a pretrained language model's vocabulary is extended with domain-specific tokens, the new embeddings must be initialized before training begins. A naive approach—random initialization from a Gaussian distribution—gives the model no prior information about the relationships among the new tokens. The paper's HMN strategy is designed to answer a specific question: how should we initialize embeddings for SVG tokens such that coordinate tokens for nearby spatial positions are already close to each other in embedding space, and segment tokens that represent similar geometric primitives are already distinguishable?

The embedding initialization equation. For each newly added token $t$, its embedding vector $\mathbf{e}_t$ is initialized as:

$\mathbf{e}_t = \lambda_\mu \boldsymbol{\mu} + \lambda_n \boldsymbol{\epsilon} + w_{\text{sem}}\, \phi(\mathrm{desc}_t) + w_{\text{num}}\, \mathbf{d}_t$

where $\boldsymbol{\mu} \in \mathbb{R}^d$ is the mean embedding vector computed over the original pretrained vocabulary $\mathcal{V}_0$, $\boldsymbol{\epsilon} \sim \mathcal{N}(0, \sigma^2 \mathbf{I})$ is isotropic Gaussian noise with per-dimension variance $\sigma^2$, $\phi(\mathrm{desc}_t)$ maps a textual description of token $t$ into the pretrained embedding space using frozen model weights, and $\mathbf{d}_t \in \mathbb{R}^d$ is a numeric embedding that encodes the scalar coordinate value for coordinate tokens (zero for non-coordinate tokens). The coefficients $\lambda_\mu = 0.8$, $\lambda_n = 0.02$, $w_{\text{sem}} = 0.1$, and $w_{\text{num}} = 0.08$ control the relative contribution of each component (from Table S2).

What it computes: the embedding for each new token is a weighted sum of four vector contributions. The mean anchor $\lambda_\mu\boldsymbol{\mu}$ provides distributional alignment with the pretrained vocabulary—it places the new token roughly in the center of the existing embedding space, avoiding extreme values that could cause optimization instability. The noise term $\lambda_n\boldsymbol{\epsilon}$ adds small random perturbations that differentiate tokens from each other (without this, all new tokens would start identically at the mean). The semantic prior $w_{\text{sem}}\phi(\mathrm{desc}_t)$ uses the model's own text encoder—frozen during initialization—to embed a natural-language description of the token (e.g., "Curve with two control points and endpoint" for a cubic Bézier segment token) into the embedding space, giving the model a head start on understanding what the token represents. The numeric embedding $w_{\text{num}}\mathbf{d}_t$ provides a structured perturbation for coordinate tokens that varies smoothly with the coordinate value.

Why this form: the additive decomposition directly addresses three distinct failure modes of simpler initializations:

1. Pure noise initialization (embedding ∼ $\mathcal{N}(0, \sigma^2\mathbf{I})$): places new tokens far from the pretrained vocabulary distribution, requiring the model to "pull" them into alignment during training—a slow and potentially unstable process, especially when the embedding matrix is large and gradients are sparse.
2. Mean-only initialization (embedding = $\boldsymbol{\mu}$): all new tokens start identically, providing no basis for differentiation and forcing the model to learn token identity from scratch, which slows early training.
3. Semantic-only initialization (embedding = $\phi(\mathrm{desc}_t)$): provides semantic differentiation but no spatial structure—two adjacent coordinate tokens like <P_20> and <P_21> would have unrelated embeddings if their textual descriptions are simply "the coordinate 20" and "the coordinate 21", because text encoders don't have a built-in notion of numeric proximity at that granularity.

The HMN form addresses all three: the mean anchor ensures distributional alignment, the noise ensures differentiability, the semantic prior provides semantic grouping (all coordinates have similar semantic descriptions, all commands have distinct ones), and the numeric embedding provides the fine-grained spatial structure that makes <P_20> and <P_21> neighbors in embedding space while keeping <P_20> and <P_200> separated by a distance proportional to their value difference.

The numeric embedding construction for $\mathbf{d}_t$. This is the most technically involved component and deserves detailed explanation. For a coordinate token with normalized scalar value $v_t \in [0, 1]$ (the raw coordinate value divided by 784):

1. Basis expansion: The scalar $v_t$ is encoded using a combination of $K = 16$ Gaussian radial basis functions (RBFs) and low-order polynomial features. The Gaussian RBFs, following Rahimi and Recht (2007), compute:

$\psi_j(v_t) = \exp\left(-\frac{(v_t - c_j)^2}{2\sigma_{\text{rbf}}^2}\right)$

for $j = 1, \ldots, K$, where the centers $c_j$ are spaced uniformly across [0, 1] and $\sigma_{\text{rbf}}$ controls the width of each basis function. This encoding captures local smoothness: small changes in $v_t$ produce small changes in the basis function outputs, so tokens for nearby coordinate values receive similar representations. The polynomial features capture global ordering: higher-degree polynomial terms encode that 200 is "further from 0" than 20, providing large-scale monotonic structure.

The combined basis representation is a vector in a low-dimensional space (dimension determined by $K$ plus the number of polynomial features), which is then projected to the model embedding dimension $d$ using a fixed random projection matrix—a Johnson–Lindenstrauss (J-L) transform that approximately preserves pairwise distances between the basis vectors while expanding them to full embedding dimensionality.

2. Normalization: The projected vector is normalized to unit length, producing $\mathbf{d}_t$, which serves as a small directional perturbation (scaled by $w_{\text{num}} = 0.08$) rather than a dominant component of the embedding.

Why Gaussian–polynomial basis + random projection? The authors could have used a simple learned embedding table for coordinate tokens—this is what most models do for new discrete tokens. The HMN approach is specifically designed to inject smoothness priors that would take the model many training steps to learn from data. The Gaussian RBFs ensure local continuity (values at 100 and 101 have similar embeddings), while the polynomial features ensure that the representation respects the total order of real numbers (100 is not just similar to 101 but is also strictly between 50 and 150 in some meaningful sense). The random projection preserves these properties while mapping to the high-dimensional embedding space without adding learnable parameters that would need to be trained alongside the model—the projection matrix is fixed at initialization and never updated.

Non-coordinate tokens omit the numeric branch. For structure, command, attribute, and segment tokens, $w_{\text{num}}\mathbf{d}_t$ is set to zero. These tokens receive only the mean anchor, noise, and semantic prior contributions. The semantic prior is particularly important for segment tokens: a <SEG> token that encodes a cubic Bézier curve gets a description like "Curve with two control points and endpoint," while one encoding an elliptical arc gets "Elliptical arc segment." These descriptions are processed through the frozen text encoder to produce embeddings that capture the functional category of the geometric primitive, even before the model has been trained on any SVG data.

Empirical validation of HMN (Table 4). The ablation study in Table 4 systematically removes components of HMN and measures the effect after 1 epoch of training across both image-to-SVG and text-to-SVG:

• Rows 1–3 (no semantic, no numeric): Starting from pure noise (Row 1: LPIPS 0.226, SSIM 0.440), adding mean anchoring (Row 2) actually hurts some metrics (SSIM drops to 0.244), likely because all new tokens start at the same point and need to be differentiated through training. Adding noise to the mean anchor (Row 3: Mean+Noise) recovers partially (SSIM 0.523).
• Rows 4–5 (semantic, no numeric): Adding semantic priors (Row 4) improves over noise-only (SSIM 0.477 vs. 0.440), and adding noise to semantic priors (Row 5: SSIM 0.550) further improves differentiation.
• Rows 6–7 (semantic + numeric): This is the critical comparison. Adding numeric priors via simple linear interpolation between text embeddings of "0" and "784" (Row 6: "Lerp") improves SSIM to 0.680 and LPIPS to 0.182. Switching to the full Gaussian–polynomial basis with J-L projection (Row 7: "HMN (J-L)") provides the best results: SSIM 0.720, LPIPS 0.170, CLIP-S 0.880—substantial improvements over row 6. This confirms that the structured numeric encoding (Gaussian RBFs capturing local smoothness, J-L projection preserving pairwise distances) matters over a naive linear interpolation of text embeddings.

The progressive improvement from Row 1 to Row 7 shows that each component of HMN contributes additively to final performance, with the numeric encoding providing the largest single gain among the four components.

A subtle point about the "Lerp" baseline: The paper explicitly notes that a naive linear interpolation between text embeddings of "0" and "784" (Row 6) does not work well because these are BPE-tokenized digit strings—the text encoder doesn't inherently understand numeric magnitude. The HMN approach bypasses this by constructing a basis that explicitly encodes numeric relationships, then projecting into the embedding space via a random matrix that approximately preserves these relationships.

Three-Stage Curriculum Training Paradigm

Training an LLM with an expanded vocabulary and hierarchical token representation is not trivial. The newly introduced tokens start with HMN-initialized embeddings that are close to the pretrained distribution but not perfectly aligned; the model must learn to predict long-range geometric dependencies; and the effective sequence complexity increases as segment tokens compress long coordinate sequences into single tokens that carry richer semantic meaning. The paper's curriculum training addresses these challenges by progressively increasing the structural complexity of the training data across three stages.

What each stage optimizes (Section 3.3):

• Stage 1: Embedding Alignment. Training begins with atomic SVG tokens only (no segment tokens) and moderate-length sequences (30–326 tokens). The primary objective is to align the newly introduced atomic token embeddings with the pretrained embedding manifold while the model learns local geometric transitions—essentially, learning the conditional distribution $P(\text{next coordinate} \mid \text{command}, \text{previous coordinates})$ at the single-command level.
• Stage 2: Structural Abstraction. Segment tokens are activated, replacing frequently co-occurring atomic command–coordinate groups with single composite tokens. The dependency horizon expands—the model now predicts segment tokens that each represent 9+ atomic tokens of geometry—while the token-space vocabulary remains the same size (atomic tokens are still available for rare patterns, but common patterns are compressed). This stage teaches the model to think in geometric units rather than coordinate digits.
• Stage 3: Global Composition. Full-length SVG programs (605–1000 tokens) are introduced. The model focuses on layout coherence, multi-path composition, and long-range dependencies—ensuring that element A and element B don't overlap inappropriately, that gradients align with their target shapes, and that the overall canvas composition is balanced.

Critical details from the paper:

• Not a data split but a data schedule: Each stage expands the training distribution without discarding earlier regimes. The model sees sequences from all three length buckets at every stage, but the proportion of long sequences increases progressively. This prevents catastrophic forgetting: the model doesn't lose the ability to generate short, simple SVGs when it starts training on long, complex ones.
• Sequence length partitions (from Tables 5 and 6): The training corpus is partitioned into three levels based on SVG token count: L1 covers 30–326 tokens, L2 covers 326–605 tokens, and L3 covers 605–1000 tokens. Evaluation is performed on all three levels at the end of each stage to measure how generalization changes as training progresses.
• Stage transitions are measured, not just declared: Tables 5 and 6 report "delta rows" showing the change from one stage to the next for each metric. For example, the image-to-SVG transition from Stage 1 to Stage 2 (in Table 5) shows that all L1/L2/L3 metrics improve substantially: CLIP-S jumps from 0.9360 to 0.9540 on L1 (+4.9% relative), from 0.8884 to 0.9320 on L2 (+4.9%), and from 0.8750 to 0.9210 on L3 (+5.3%). This confirms that Stage 2's segment token activation benefits all complexity levels, not just the longer sequences.
• Stage 2 is the inflection point: The largest improvements occur at the Stage 1→Stage 2 transition. In the image-to-SVG results (Table 5), SSIM on L1 improves from 0.8052 to 0.8152, LPIPS drops from 0.1716 to 0.1611, and ImageReward (ImgR) flips from negative (-0.0764) to slightly positive (0.0294)—indicating that human-preference scores improve dramatically once the model starts operating on segment tokens. This is consistent with the paper's central thesis: compressing coordinates into geometric primitives doesn't just reduce sequence length; it fundamentally changes what the model learns, allowing it to focus on compositional structure rather than coordinate-by-coordinate prediction.
• L3 remains challenging throughout: Even at Stage 3, the hardest sequences (L3, 605–1000 tokens) show lower render success rates (87.69%–90.41%) and higher LPIPS (0.2185–0.2405) compared to L1 (94.53%–95.60% render success, LPIPS 0.1611–0.1716). This is expected—longer SVGs are inherently more complex—but the curriculum training ensures that the performance gap between L1 and L3 narrows across stages, indicating improved generalization to complex compositions.
• Text-to-SVG shows parallel trends (Table 6): The text-conditioned generation follows the same pattern, with Stage 2 activation of segment tokens providing the largest gains. The transition from Stage 1 to Stage 2 improves CLIP on L1 from 0.2346 to 0.2356 (modest) and DINO diversity on L1 from 0.2949 to 0.2931 (stable), but more importantly, it stabilizes the performance on longer sequences—without curriculum training, performance on L2 and L3 degrades more severely with length.

Why curriculum training is necessary: The paper does not explicitly compare against a non-curriculum baseline (e.g., training all sequences from the start with segment tokens activated), but the stage-transition data in Tables 5 and 6 strongly implies that simultaneous activation of segment tokens and exposure to long sequences would destabilize training. At Stage 1, when only atomic tokens are used, the model learns the basic transition probabilities between commands and coordinates. If segment tokens were introduced simultaneously with long sequences at epoch 0, the model would face a combinatorial explosion of possible token sequences—both atomic and segment tokens are in the vocabulary, and it would need to learn when to use each. The staged approach decomposes this: Stage 1 establishes the atomic foundation, Stage 2 introduces compression while keeping sequences manageable, and Stage 3 pushes sequence length while the model already understands segment tokens.

The progressive depth analogy: The paper frames the curriculum through the lens of "program depth," drawing an implicit analogy to how human programmers learn: first understand individual statements (atomic tokens), then learn to combine them into functions (segment tokens), then compose functions into complete programs (global composition). This framing is not just pedagogical—it reflects the real structure of SVG programs, where a drawing is genuinely a hierarchical composition of geometric primitives into shapes, shapes into elements, and elements into scenes.

Training and Inference Configuration

The implementation details from Appendix C establish the experimental conditions under which all results were obtained. These are critical for reproducibility and for understanding the practical constraints of the approach.

Model architecture. The base model is Qwen2.5-VL-3B-Instruct, a 3-billion-parameter vision-language model. The vision encoder and multimodal projector are kept frozen throughout training; only the language model parameters and the newly introduced SVG token embeddings (2,450 atomic + 500 segment = ~2,950 tokens) are optimized. This means the core visual understanding capabilities come from the pretrained model—the finetuning only teaches the language model to decode those visual features into SVG tokens. The paper does not modify the model architecture; it only extends the embedding matrix and (implicitly) the output projection layer to accommodate the new tokens.

Extended vocabulary size. Starting from Qwen2.5-VL's native vocabulary (which the paper doesn't specify but is standard for the Qwen2.5 family), HiVG adds approximately 2,950 new tokens: 2,450 atomic tokens (from the four-vocabulary decomposition) plus approximately 500 segment tokens (from $M = 500$ SSL merging iterations, as confirmed in Table 3's caption: "All models use M=500 merges"). The total vocabulary expansion is nontrivial—adding nearly 3,000 tokens to a vocabulary that's typically in the 100k–150k range means roughly 2–3% new tokens, which is large enough to require careful initialization but small enough to avoid catastrophic interference with the pretrained token representations.

Training hyperparameters (from Table S2 and Section 4.1):

• Optimizer: AdamW
• Learning rate: $1 \times 10^{-5}$
• Weight decay: 0.2
• Warmup ratio: 0.2 (changed from 0.1 in some supplementary configurations) over the total training steps
• Global batch size: 128
• Training epochs: 2 full epochs over the 2.45M-sample corpus
• Maximum context length per stage: 1,792 tokens (Stage 1), 2,176 tokens (Stage 2), 2,432 tokens (Stage 3)
• Training precision and hardware: not explicitly stated, but implied by the 3B parameter scale to be standard mixed-precision training

The 2-epoch setting is notable: compared to typical LLM finetuning which often uses 1–3 epochs, two full passes over 2.45M examples is substantial. The paper's claim of "2.7× fewer training tokens" (Figure 1b) refers to the fact that HiVG reaches a given quality level with fewer total tokens processed during training—not fewer epochs but fewer tokens because each SVG is compressed into fewer tokens per sample. If a baseline using raw text tokenization requires 3 epochs to converge, and HiVG requires 2 epochs with 2.7x compression, the total training tokens are reduced by approximately 4–5×.

Multi-task training formulation. The model is trained on two tasks simultaneously using unified instruction-style templates (Figure S5):

For text-to-SVG (T2ST):

<|im_start|>user
[T2ST] Generate SVG tokens from text. {text description}
<|im_end|>
<|im_start|>assistant
<svg>viewBox=0 0 784 784<cmd_M><P_242><P_674><cmd_c><d_7><d_-195>...
</path></svg>
<|im_end|>

For image-to-SVG (I2ST):

<|im_start|>user
[I2ST] <image>. Generate SVG tokens from image.
<|im_end|>
<|im_start|>assistant
<svg>viewBox=0 0 784 784<cmd_M><P_283><P_567><cmd_c><d_0><d_0>...
</path></svg>
<|im_end|>

The task prefix [T2ST] and [I2ST] serve as explicit task identifiers, while the SVG output is always in the hierarchical token format (mix of atomic and segment tokens, not raw SVG strings). For image-conditioned generation, the <image> placeholder is processed by the frozen vision encoder, and the model learns to map visual features to SVG token sequences autoregressively. The unified format means the model sees both tasks interleaved during training, sharing the same SVG representation across modalities.

Inference configuration (from Table S2):

• Decoding strategy: standard autoregressive generation
• Temperature: 0.7
• Top-p (nucleus sampling): 0.9
• Top-K: 50
• Repetition penalty: 1.0 (no penalty)
• Evaluation rendering resolution: $512 \times 512$ (different from the training canvas size of $784 \times 784$; the paper doesn't explain this discrepancy, but it likely reflects downscaling for fair comparison with baselines that output at 512px resolution)

What "training on SVG tokens" means mechanistically: The training objective is standard next-token prediction (causal language modeling) applied to sequences of hierarchical SVG tokens. For a sequence $\mathbf{x} = (x_1, x_2, \ldots, x_T)$ where each $x_t$ is either an atomic token or a segment token, the loss is:

$\mathcal{L}_{\text{SFT}} = -\sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t}, \text{condition})$

where $\theta$ includes the full language model parameters plus the new token embeddings, and condition is either the text prompt embedding or the image embedding from the frozen vision tower. The model is trained on the assistant's response tokens only (the user prompt tokens are masked from the loss, following standard instruction tuning practice, though this detail is inferred from the template structure rather than stated explicitly).

Why supervised finetuning rather than RL or differentiable rendering: The paper explicitly positions HiVG within the autoregressive program synthesis paradigm rather than the differentiable rendering paradigm of earlier work. The choice of SFT over reinforcement learning (even though RL has been applied to SVG generation, as in SVGen and Reason-SVG) is consistent with the paper's focus on representation design: the goal is to show that better tokenization improves any autoregressive training objective, and SFT is the simplest and most reproducible baseline for this claim. The paper does not claim that SFT is optimal—only that with the right tokenization, even simple SFT on a 3B model can outperform larger models trained with more sophisticated objectives.

Design Choices and Their Justifications: A Consolidated View

Several cross-cutting design decisions emerge from the technical breakdown that are worth making explicit:

Why separate the atomic and segment vocabularies rather than using only segment tokens. The paper keeps both atomic tokens and segment tokens available during training and inference. This means the model can choose—dynamically, during autoregressive generation—whether to emit a sequence of atomic tokens or a single segment token for a given geometric primitive. The rationale is flexibility: common patterns get compressed into segment tokens, but rare or novel patterns can still be expressed using atomic tokens. If the vocabulary were limited to only segment tokens, the model could only generate patterns that appeared frequently enough in the training corpus to warrant a segment vocabulary entry—a severe limitation for diverse SVG generation.

Why 500 merges. The paper uses $M = 500$ merging iterations (Table 3 caption). This number is a hyperparameter that trades off between compression ratio and vocabulary granularity. More merges would compress further but at the risk of creating overly specific segment tokens that represent rare patterns and don't generalize. Fewer merges would leave more patterns expressed as atomic sequences. The choice of 500 appears to be based on empirical performance (the paper doesn't ablate $M$ directly, but the segment vocabulary size of 500 tokens—roughly 20% of the atomic vocabulary size—is a reasonable balance).

Why 784×784 canvas with ±794 coordinate range. The canvas size of $784 \times 784$ is typical for SVG generation benchmarks (it maps cleanly to common icon sizes). The coordinate range extending from -794 to +794 (rather than exactly 0–784) accounts for out-of-bound tolerance during the preprocessing pipeline: when transforms are baked into coordinates, some values may slightly exceed the nominal canvas, and clipping/clamping to ±10 tolerance (Appendix A) produces coordinates in the extended range. The symmetrical range for relative coordinates enables any vector to be expressed as an offset from any starting position anywhere on the canvas.

Why freeze the vision encoder. The vision tower and multimodal projector of Qwen2.5-VL-3B are kept frozen, with only the language model and token embeddings trained. This is a practical choice that dramatically reduces training cost (vision encoders are typically 30–40% of the parameters in VLMs) and avoids the risk of degrading pretrained visual representations during SVG-specific finetuning. The assumption is that Qwen2.5-VL's pretrained visual features are already sufficient for the level of visual understanding needed for SVG reconstruction—identifying shapes, colors, and spatial layout—and that the only missing piece is the ability to decode those features into valid SVG tokens.

Why not use on-the-fly rendering feedback during training. Some prior SVG generation methods (e.g., SVGen's rendering-aware RL) use differentiable rendering or rendered-image feedback to improve geometric accuracy. HiVG does not—the training is purely autoregressive next-token prediction with no rendering-based loss. This is a deliberate simplification that keeps the training pipeline purely text-like and compatible with standard LLM training infrastructure. The paper implicitly argues that good enough tokenization makes rendering feedback unnecessary for competitive performance, though the ablation results on hard sequences (L3 performance plateauing at ~87–90% render success) suggest there may still be a gap that rendering feedback could fill.

4. Key Insights and Innovations

Innovation 1: Tokenization Should Encode Executable Geometry, Not Character Fragments

The dominant assumption in LLM-based SVG generation—inherited unexamined from NLP—is that whatever byte-level tokenizer came with the pretrained model is good enough. StarVector, OmniSVG, and InternSVG all feed SVG strings through the same BPE tokenizer that handles natural language, operating on the implicit premise that SVG is "just text" and that the model will figure out geometric structure from context. This paper makes a clean break from that premise.

The conceptual move is to redefine the unit of representation from subword tokens to complete, renderable geometric operations. The paper doesn't ask "how can we make the existing tokenizer work better for SVG?" It asks "what is the natural atomic unit of vector graphics, and how should a tokenizer reflect that?" The answer—that a cubic Bézier curve with its six coordinate parameters is a semantic unit, not seven independent tokens—is simple in retrospect but represents a fundamental shift in how the problem is framed.

What makes this distinctive is that it's not an architectural innovation. The model remains a standard autoregressive transformer. The training objective remains next-token prediction. The innovation is purely representational: change what the tokens are, and the learning problem changes qualitatively. The sequence compression (62.7%–63.8% in Figure 1a) and training efficiency gains (2.7× fewer training tokens in Figure 1b) are not the primary contribution—they are symptoms of the deeper insight that the tokenizer should match the domain's intrinsic structure.

This is a fundamental reframing, not an incremental improvement. Standard BPE and its SVG-aware variants both treat SVG as a linear string from which structure must be inferred. HiVG treats SVG as a hierarchical program whose structure is encoded in the token types themselves. The fixed-arity constraint that produces segments—each command token is always followed by exactly $k$ coordinate tokens—is not a heuristic hack but a deliberate design choice that makes syntactic validity a property of the tokenization scheme rather than something the model must learn.

The evidence that this matters beyond compression: Tables 5 and 6 show render success rates climbing from ~90% in Stage 1 (atomic tokens only) to ~96% in later stages with segment tokens, and the SSIM/LPIPS metrics in Table 1 show HiVG-3B substantially outperforming larger models (SSIM 0.896 vs. Gemini-2.5-pro's 0.790). A 3B model beating 8B+ models on geometric fidelity is not explicable by sequence compression alone—it suggests that the structured representation enables more sample-efficient learning of geometric relationships.

Innovation 2: The "Renderable Unit" Constraint Redefines Token Merging for Structured Data

BPE-style token merging is well-established for text—merge frequently co-occurring character pairs, build a subword vocabulary. Recent work has extended this idea to continuous robot actions (FAST) and 3D mesh coordinates (FreeMesh). But all prior merging operates under a weak constraint: merge whatever co-occurs frequently. The paper's distinctive contribution is introducing domain-specific validity constraints into the merging process.

The key idea is that not all frequent co-occurrences deserve to become tokens. In SVG, the sequence <cmd_c> <d_20> might appear frequently, but merging it would produce a token that represents half a command—not a renderable operation. HiVG's SSL algorithm restricts merges to complete segments, where a segment is defined as (command, param_1, ..., param_k) with $k$ fixed by the SVG specification. This guarantees that every learned segment token is executable by construction: decoding a <SEG> token always produces a syntactically valid drawing operation with the correct number of parameters.

This constraint transforms token merging from a statistical compression technique into a geometry-aware abstraction mechanism. The field already knew that compression helps learning (Delétang et al., 2023). HiVG shows that how you compress—specifically, what constraints you impose on merge candidates—determines whether compression produces meaningful abstractions or just shorter bit strings. The segment tokens that emerge (Figure 4: Bézier curves, elliptical arcs, orthogonal corners) are not arbitrary bit patterns but recognizable geometric primitives that a human designer would identify as reusable building blocks.

This is a conceptual advance rather than purely empirical. It establishes a template for how to apply BPE-style merging to any domain with structured syntax: identify the smallest executable units, constrain merges to preserve that executability, and let the frequency statistics discover which composite units recur. The paper explicitly contrasts this with CAD-GPT's approach, which compresses spatial parameters without geometric validity guarantees. The implication is that domain-specific merging constraints are the missing ingredient that prevents naive compression from producing semantically coherent tokens.

The evidence that the constraint matters comes from the segment analysis in Figure 10. The cleaning statistics (Figure 10a) show that raw SVGs contain roughly 0.86 degenerate commands per sample—zero-move lines, zero-radius arcs—that standard BPE would happily merge into "frequent" tokens. SSL's validity constraint filters these out because they don't form renderable units. The resulting segment tokens have stable median length (~9 atomic tokens across frequency buckets, Figure 10d) and capture commands proportional to their geometric importance (cubic Béziers dominate at 40% share in mid-frequency buckets, Figure 10c), suggesting that SSL is discovering genuine geometric primitives rather than statistical artifacts.

Innovation 3: Spatial Prior Injection via Embedding Initialization Is a Viable Alternative to Learned Coordinate Representations

The field has two established approaches to representing numeric values in discrete token vocabularies: (1) treat each value as an independent token and learn embeddings from data (the standard approach for all discrete vocabularies), or (2) use continuous representations that encode numeric magnitude directly (e.g., value embeddings in tabular transformers, sinusoidal position encodings). HiVG introduces a hybrid third path: discrete tokens that receive structured initializations encoding their numeric relationships, then continue to be learned from data during training.

The conceptual move is to recognize that token embeddings have a dual nature: they are both learnable parameters that training will update, and initial conditions that determine the optimization landscape. The standard view treats initialization as a transient state—any reasonable initialization will be overwritten by training, so what matters is the final learned embedding. HMN challenges this view by arguing that initialization can encode priors that persist through training because they provide a better optimization basin.

The specific priors encoded are: (1) coordinate tokens for nearby values should start near each other in embedding space (via Gaussian RBF basis encoding), (2) coordinate tokens should respect monotonic ordering (via polynomial features), and (3) tokens of the same semantic type (all commands, all segment tokens of the same curve family) should be distinguishable but related (via semantic description embeddings). None of these priors are learned—they are mathematically constructed and injected at initialization.

This is a diagnostic contribution as much as a methodological one. The ablation in Table 4 doesn't just show that HMN works better than noise initialization—it reveals why spatial priors matter. Row 6 ("Lerp"), which attempts to encode numeric ordering by interpolating between text embeddings of "0" and "784," performs substantially worse than Row 7 ("HMN J-L"), which uses Gaussian–polynomial bases. The failure of the Lerp baseline is informative: text encoders don't encode numeric magnitude in a way that linear interpolation can recover, because BPE tokenization of digit strings destroys continuity. This negative result clarifies that the prior needs to be mathematically constructed outside the language model's embedding space, then projected in, rather than derived from the language model's own representations.

The broader significance is that this approach decouples spatial structure from data-driven learning. A model trained with HMN starts with spatial awareness on epoch zero—<P_20> and <P_21> are already neighbors—rather than needing thousands of training steps to discover numeric ordering from co-occurrence statistics. The 2.7× training token efficiency (Figure 1b) is partly attributable to this head start: the model can immediately focus on learning which coordinates go with which commands, rather than simultaneously learning that coordinates form a continuous space.

This is not a fundamental theoretical advance—the mathematics of RBF encoding and J-L projections are well-established—but it is a practical innovation that changes how practitioners should think about domain-specific vocabulary expansion. The lesson is: don't just add new tokens; add them with structure that aligns with the domain's intrinsic geometry.

Innovation 4: Curriculum Staging as an Optimization Decomposition for Hierarchical Tokenization

Curriculum learning itself is not novel—training on easier examples before harder ones is a well-known technique. The paper's distinctive contribution is identifying a specific optimization decomposition that separates the three distinct challenges of hierarchical tokenization: embedding alignment, structural abstraction, and global composition. This decomposition is not an arbitrary progression from easy to hard; it maps directly onto the structure of HiVG's token hierarchy.

The insight from Tables 5 and 6 is that the Stage 1 → Stage 2 transition (activating segment tokens) is the inflection point where the largest metric improvements occur. On image-to-SVG, CLIP-S on the hardest sequences (L3) jumps from 0.8750 to 0.9210—a 5.3% relative improvement—at this transition. The render success rate on L3 increases from 90.09% to 90.29% (Stage 1→2) and only to 87.69% at Stage 3 (the decline on L3 at Stage 3 reflects the introduction of genuinely harder sequences, not regression). This pattern suggests that segment token activation is the key learning event, and that separating it from long-sequence exposure prevents the interference that would occur if the model had to simultaneously learn what segment tokens mean and how to compose them into long programs.

The paper doesn't provide the ablation of "no curriculum"—training all sequences with segment tokens from epoch zero—which would be the direct evidence that staging is necessary. But the detailed per-stage, per-length-bucket reporting in Tables 5 and 6 makes the implicit case: performance on L1 and L2 remains stable or improves across stages (L1 CLIP-S goes 0.9360 → 0.9540 → 0.9581), while L3 catches up gradually. This pattern is consistent with staging preventing the short-sequence performance degradation that often accompanies long-sequence training.

This is an incremental but practically significant contribution. The specific staging—30–326 tokens, then 326–605, then 605–1000—is likely task-specific and would need tuning for other domains. But the conceptual framework—that hierarchical tokenization creates distinct optimization phases, and that these should be separated rather than interleaved—generalizes. Any domain that introduces vocabulary compression (segment-like tokens) on top of atomic representations would benefit from staging that first stabilizes the atomic vocabulary before introducing compressed tokens.

The evidence also reveals a limitation: L3 sequences never fully close the gap with L1/L2, even after Stage 3. Image-to-SVG SSIM on L3 is 0.7019 at Stage 3 versus 0.8129 on L1. This suggests that while curriculum staging helps, it doesn't solve the inherent difficulty of long-range geometric dependencies—a finding that is equally important for setting realistic expectations about what hierarchical tokenization can achieve.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use a custom merged corpus constructed from three open-source SVG datasets: SVG-Stack (Rodriguez et al., 2025, 2,283,875 samples), SVGX-Dataset (Xing et al., 2025, 257,086 samples), and MMSVG-Icon (Yang et al., 2025, 1,159,423 samples). After cross-source merging and deduplication, the final corpus contains 2,445,092 unique SVG samples spanning icons, emojis, logos, and interface elements. The test set for quantitative evaluation is not explicitly sized in the main paper, but human evaluation uses 60 randomly sampled images from the image-to-SVG test set covering simple icons through challenging logo-style compositions (Section 4.3). The training set is partitioned into three complexity buckets by token length for curriculum staging: L1 (30–326 tokens), L2 (326–605 tokens), and L3 (605–1000 tokens), with corresponding test subsets for fine-grained generalization analysis (Section 4.4D).

• Base model. All experiments use Qwen2.5-VL-3B-Instruct (Bai et al., 2025), a 3-billion-parameter vision-language model. The paper states the model is chosen because its "pretrained visual features are already sufficient for the level of visual understanding needed for SVG reconstruction—identifying shapes, colors, and spatial layout" (prior sections analysis). The vision encoder and multimodal projector are frozen throughout all experiments; only the language model parameters and newly introduced SVG token embeddings (~2,950 tokens) are optimized. This matters for the training efficiency claims—freezing the vision tower means the ~30–40% of parameters responsible for visual encoding require no finetuning compute, reducing total training cost.

• Metrics. The paper evaluates along five dimensions. (1) Validity and efficiency: Render success rate (Render%, whether the generated SVG parses and renders without error), average token count (TokCnt), path count (PathCnt), and path command count (CmdCnt). Lower TokCnt, PathCnt, and CmdCnt indicate more compact programs. (2) Semantic and visual quality for text-to-SVG: CLIP similarity (Radford et al., 2021) between rendered images and text prompts. For image-to-SVG, additional CLIP-visual similarity (CLIP-S) between rendered output and input reference image, plus SSIM and LPIPS for structural and perceptual fidelity. (3) Diversity: Computed from DINOv2-ViT-Large features (Oquab et al., 2024) as Diversity = 1 − (2/(L(L−1))) × Σ_{i<j} cos(x_θ^(i), x_θ^(j)), where higher values indicate less feature similarity among generated samples. (4) Perceptual and preference quality: HPSv2 (Wu et al., 2023), ImageReward (Xu et al., 2023), PickScore (Kirstain et al., 2023), and Aesthetic score (Schuhmann, 2022). (5) Human evaluation: Pairwise visual preference (which SVG better reconstructs the reference image, with tie allowed) and SVG code usability review on a 1–5 Likert scale across semantic layering, editability, redundancy control, and overall code usability, conducted by 8 professional SVG practitioners.

• Baselines. The paper compares against 10 systems, spanning closed-source LLMs and open-source SVG-specific models, though not all baselines support both text and image input. For text-to-SVG only: DeepSeek-v3.2 (DeepSeek-AI, 2025) and SVGen-7B (Wang et al., 2025). For both text-to-SVG and image-to-SVG: Qwen3.5 Plus (Qwen Team, 2026), Gemini-2.5-pro (Google DeepMind, 2025), GPT-5.2 (OpenAI, 2025), Claude-Sonnet-4.5 (Anthropic, 2025), OmniSVG-4B and OmniSVG-8B (Yang et al., 2025), and InternSVG-8B (Wang et al., 2025). The closed-source models (Qwen3.5 Plus, Gemini-2.5-pro, GPT-5.2, Claude-Sonnet-4.5) represent state-of-the-art general-purpose multimodal LLMs applied to SVG generation through prompting, while the open-source models (SVGen-7B, OmniSVG-4B/8B, InternSVG-8B) represent purpose-built SVG generation systems. HiVG-3B is the smallest model in this comparison by parameter count, making the performance comparisons particularly informative about representation quality versus model scale.

• Generation budget / compute accounting. The paper does not report inference FLOPs or wall-clock time for any method. The primary efficiency metric is token count (TokCnt)—how many tokens the model must generate to produce a complete SVG. This is a proxy for autoregressive decoding cost: each generated token requires one forward pass through the language model, so token count reduction directly translates to inference speedup. For the training efficiency claim (Figure 1b: "HiVG reaches comparable quality with approximately 2.7× fewer training tokens"), the accounting is: total training tokens = number of training samples × average tokens per sample × number of epochs. Since HiVG compresses each SVG into ~2.7× fewer tokens, it processes proportionally fewer total tokens during training for the same number of samples and epochs. The paper does not report the exact total training tokens for HiVG or for the baseline, nor does it specify at which quality threshold the 2.7× figure was measured—Figure 1b is described qualitatively as reaching "comparable quality" but the metric and quality level are not quantified in the main text. This is a significant omission for an efficiency claim.

• Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported for the automatic metrics in Table 1. The paper states that all methods are evaluated under "unified rendering and post-processing rules, to reduce prompt-induced variance in downstream comparisons" (Appendix C.2), but no confidence intervals, standard deviations, or multiple-run averaging are reported for the main quantitative results. The human evaluation in Section 4.3 uses a protocol with some rigor: 60 randomly sampled images stratified by complexity (simple, medium, challenging), 8 professional evaluators, randomized method names and output order, 3 annotators per pairwise comparison with majority vote determining the winner. However, the pairwise comparison uses forced choice between HiVG-3B and each baseline individually rather than a full ranking or tournament, which may inflate HiVG's apparent win rate compared to a design where all methods are ranked simultaneously. The Illustrator-based usability review (Table S1) uses 5 methods rated by the same 8 evaluators on a 1–5 Likert scale.

Main Quantitative Results

Sequence Compression and Token Efficiency (Figure 1)

The paper's foundational efficiency claim appears in Figure 1a: HiVG tokenization compresses SVG sequences by 62.7%–63.8%, corresponding to 2.68×–2.76× fewer tokens compared to raw-string tokenization. Two specific compression figures are reported: 205.6 tokens reduction (−62.7%, 2.68× fewer) and 1307.2 tokens reduction (−63.8%, 2.76× fewer), though the paper doesn't specify whether these are averages, medians, or specific examples. The larger absolute reduction (1307.2 tokens) likely corresponds to a complex SVG where the compression benefit is most pronounced—a single path with many drawing commands that get compressed into segment tokens.

Figure 1b shows the training efficiency claim: HiVG reaches comparable quality with approximately 2.7× fewer training tokens. The x-axis is "training token budget" and the y-axis is an unlabeled quality metric (described only as "performance" in the figure). The paper does not report the exact quality level at which the 2.7× measurement was taken, nor what the quality metric is, nor the absolute number of training tokens consumed by either HiVG or the baseline. This makes the claim suggestive rather than precisely quantitative—we know HiVG is more token-efficient, but the magnitude and the conditions under which the 2.7× holds are underspecified.

Human evaluation results (Figure 1c) show HiVG-3B achieving a usability score of 4.06 on a 1–5 Likert scale, with the next best method (GPT-5.2) scoring 3.47 from the Illustrator review in Table S1. Pairwise comparisons show HiVG winning 58.9%–70.8% against individual baselines, with the exact win rate varying by opponent (the figure shows bars at different heights for different baseline comparisons).

Image-to-SVG and Text-to-SVG Quantitative Comparison (Table 1)

Table 1 reports the head-to-head comparison across all methods on both tasks. This is the primary quantitative evidence for HiVG's performance relative to existing systems.

Image-to-SVG reconstruction. HiVG-3B achieves the best scores on all four structural/visual metrics: SSIM 0.896 (next best: Gemini-2.5-pro at 0.790), LPIPS 0.114 (next best: GPT-5.2 at 0.205), CLIP-S 0.957 (next best: GPT-5.2 at 0.930), and Aesthetic score 4.681 (next best: GPT-5.2 at 4.841—note that GPT-5.2 leads on Aesthetic, so HiVG does not uniformly dominate). The SSIM gap (0.896 vs. 0.790) represents a 13.4% relative improvement over the best baseline, and the LPIPS gap (0.114 vs. 0.205) is a 44.4% relative reduction in perceptual distance. On preference metrics, HiVG scores HPS 0.221 (second to GPT-5.2 at 0.222, essentially tied) and PickScore 21.652 (below GPT-5.2 at 23.977, Gemini-2.5-pro at 22.346, and InternSVG-8B at 22.181). This pattern—HiVG dominating structural fidelity but being competitive rather than dominant on learned perceptual quality metrics—is consistent across both tasks.

Text-to-SVG generation. HiVG-3B achieves CLIP 0.239 (below Qwen3.5 Plus at 0.291 and GPT-5.2 at 0.291), PickScore 20.575 (below GPT-5.2 at 21.268 and Qwen3.5 Plus at 20.972), HPS 0.194 (below GPT-5.2 at 0.214 and Qwen3.5 Plus at 0.206), and Aesthetic 4.632 (below GPT-5.2 at 4.806, Gemini-2.5-pro at 4.765, and Qwen3.5 Plus at 4.671). HiVG is competitive but not best on text-to-SVG semantic alignment—the larger general-purpose models (GPT-5.2, Qwen3.5 Plus) and Gemini-2.5-pro lead on text-conditioned metrics. This is notable because it suggests that while HiVG's tokenization excels at encoding geometric fidelity (where image-to-SVG reconstruction directly measures it), the text-to-SVG task depends more on the underlying language model's semantic understanding, which 3B parameters may be insufficient for relative to 7B–8B+ competitors.

Comparison with SVGen-7B (the closest open-source SVG-specific baseline). HiVG-3B substantially outperforms SVGen-7B on text-to-SVG across all metrics: CLIP 0.239 vs. 0.223, PickScore 20.575 vs. 19.023, HPS 0.194 vs. 0.202 (SVGen leads here), Aesthetic 4.632 vs. 4.708 (SVGen leads here). The mixed pattern on HPS and Aesthetic suggests SVGen generates aesthetically pleasing but semantically less aligned outputs compared to HiVG.

The parameter-count efficiency story. HiVG-3B outperforms OmniSVG-8B and InternSVG-8B on all image-to-SVG metrics while using less than half the parameters. On text-to-SVG, HiVG-3B is competitive with these larger models—beating them on PickScore and Aesthetic, roughly tying on HPS, and slightly trailing OmniSVG-8B on CLIP (0.239 vs. 0.229). A 3B model matching or exceeding 8B models on SVG generation is the paper's strongest single piece of evidence that tokenization quality substitutes for model scale.

Missing from Table 1: No standard deviations, confidence intervals, or significance tests are reported. The differences between methods, especially on text-to-SVG where scores cluster tightly (CLIP ranges 0.214–0.291 across all methods), may not be statistically distinguishable. The paper also does not report the render success rate for any method in this table—a notable omission given that validity is a key claimed advantage of HiVG.

Curriculum Training Stage Analysis (Tables 5 and 6)

Tables 5 and 6 provide the most detailed evidence for how training progresses under the three-stage curriculum, broken down by both stage (S1, S2, S3) and test complexity level (L1, L2, L3). This is a fine-grained diagnostic of whether the curriculum improves generalization to longer sequences.

Image-to-SVG curriculum progression (Table 5). At Stage 1 (atomic tokens only, short sequences), L1 performance is already strong: Render 95.20%, SSIM 0.8052, LPIPS 0.1716, CLIP-S 0.9360. But performance degrades sharply with sequence length: L3 shows Render 90.09%, SSIM 0.7015, LPIPS 0.2405, CLIP-S 0.8750. The transition from Stage 1 to Stage 2 (activating segment tokens) produces gains across all levels: L1 CLIP-S improves from 0.9360 to 0.9540 (+1.9% relative), L2 from 0.8884 to 0.9320 (+4.9%), L3 from 0.8750 to 0.9210 (+5.3%). The L3 gain is the largest, suggesting segment tokens particularly help with longer sequences where coordinate redundancy was most severe. ImageReward (ImgR) flips from negative at Stage 1 (−0.0764 for L1, −0.2500 for L3) to positive at Stage 2 (0.0294 for L1, 0.0520 for L3), indicating that the learned perceptual quality metric responds positively to segment-level representation.

The Stage 2→3 transition (introducing full-length programs) shows more modest improvements and some regressions. L1 CLIP-S improves to 0.9581, but L3 CLIP-S edges up only slightly to 0.9454, and L3 Render declines to 87.69% (likely because Stage 3 introduces genuinely harder sequences that were not in the training distribution at earlier stages). The delta rows quantify these changes precisely: Stage 2→3 shows L1 SSIM decreasing from 0.8152 to 0.8129 (−0.3%), L3 Render improving slightly from 90.29% to 87.69% (actually a 2.6% decline, indicating more render failures on the hardest sequences after Stage 3 training), and L3 ImageReward improving from 0.0520 to 0.1490 (+186.5%). The large ImageReward gain despite Render decline suggests a tradeoff: Stage 3 training produces more aesthetically pleasing SVGs at the cost of occasional structural invalidity on the most complex sequences.

Text-to-SVG curriculum progression (Table 6). The pattern is similar but less dramatic. Stage 1→2 transition: L1 CLIP is essentially flat (0.2346 → 0.2356), L2 CLIP improves from 0.2322 to 0.2335, L3 CLIP from 0.2305 to 0.2320. DINO diversity actually decreases slightly across all levels at the Stage 1→2 transition (L1: 0.2949 → 0.2931, L2: 0.3535 → 0.3490, L3: 0.4015 → 0.3930), suggesting segment tokens reduce output diversity—a potentially undesirable side effect of compression (the model generates more consistent but less varied SVGs). PickScore improves across all levels (L1: 20.552 → 20.643), consistent with the ImageReward pattern in Table 5. The Stage 2→3 transition shows similar trends: further slight CLIP improvements, further slight DINO decreases, and PickScore/HPS remaining stable or declining marginally.

Key patterns across both tables. The most consistent finding is that segment token activation (Stage 1→2) is where the largest per-metric gains occur, while long-sequence training (Stage 2→3) mainly helps with structural complexity metrics (CmdCnt, PathCnt) and aesthetic preference scores, with less impact on semantic alignment. The L3 difficulty bucket never fully catches up to L1—at Stage 3, Image-to-SVG LPIPS on L3 is 0.2353 vs. 0.1611 on L1 (a 46% gap), and Text-to-SVG CLIP on L3 is 0.2335 vs. 0.2356 on L1 (essentially closed for CLIP but DINO diversity on L3 is 0.3859 vs. 0.3032 on L1, indicating more varied but potentially less controlled outputs). The persistent L1–L3 performance gap demonstrates that while curriculum training helps longer sequences, it does not eliminate the difficulty gradient—long, complex SVGs remain genuinely harder to generate accurately.

Human Evaluation (Figure 1c and Table S1)

Human evaluation provides evidence that the automatic metrics in Table 1 may understate HiVG's practical advantages—or, alternatively, that human evaluators respond to different qualities than CLIP/SSIM capture.

Pairwise visual preference (Figure 1c). HiVG-3B wins 58.9%–70.8% of pairwise comparisons against individual baselines, with the exact win rate varying by opponent. The bar chart in Figure 1c shows seven baseline comparisons (SVGen-7B, OmniSVG-8B, InternSVG-8B, Qwen3.5 Plus, Gemini-2.5-pro, GPT-5.2, Claude-Sonnet-4.5), but the main text only reports the range (58.9%–70.8%) without reporting per-baseline numbers. Since each comparison uses majority vote from 3 annotators and 60 samples, the win rates represent fractions of the 60 images where a majority of evaluators preferred HiVG's output.

The 58.9% lower bound is worth examining: against which baseline does HiVG only win ~59% of the time? If it's one of the 7B–8B open-source models (SVGen-7B, OmniSVG-8B, InternSVG-8B), this would suggest HiVG's advantage over purpose-built SVG systems is real but modest in human-judged quality. If it's against GPT-5.2 (which leads on many automatic metrics in Table 1), the 58.9% would actually be impressive—a 3B model winning a majority of comparisons against the strongest closed-source 2025 model. The paper doesn't provide the per-baseline breakdown needed to resolve this.

SVG code usability review (Table S1). Eight professional SVG practitioners import generated SVGs into Adobe Illustrator and score them on a 1–5 Likert scale across four dimensions. HiVG-3B achieves the best scores in all categories: semantic layering 4.11 (next best: GPT-5.2 at 3.56), editability 4.05 (next best: GPT-5.2 at 3.49), redundancy control 3.96 (next best: GPT-5.2 at 3.37), and overall code usability 4.06 (next best: GPT-5.2 at 3.47). The gaps are substantial—roughly 0.5–0.6 points on a 5-point scale—and surprisingly large relative to the narrower gaps in automatic metrics. This suggests that HiVG's structural tokenization produces SVGs that are genuinely more usable and editable in professional workflows, a quality that raster-domain metrics cannot measure.

However, the Illustrator review has limitations: only 5 methods were evaluated (SVGen-7B, InternSVG-8B, Gemini-2.5-pro, GPT-5.2, HiVG-3B), the evaluator pool (8 practitioners) is small, and the evaluation protocol (importing and inspecting SVGs individually) is inherently subjective. No inter-rater reliability statistics (e.g., intraclass correlation, Fleiss' kappa) are reported.

Reconciling human evaluation with automatic metrics. On text-to-SVG, Table 1 shows GPT-5.2 leading HiVG on CLIP (0.291 vs. 0.239), PickScore (21.268 vs. 20.575), HPS (0.214 vs. 0.194), and Aesthetic (4.806 vs. 4.632). Yet the human evaluation shows HiVG winning pairwise comparisons (the Figure 1c bar for what appears to be GPT-5.2—the exact bar isn't labeled in the main text, but GPT-5.2 is one of the seven baselines). This discrepancy suggests that automatic metrics (which measure rendered-image quality) and human evaluators (who assess both rendered quality and SVG code usability) prioritize different dimensions. HiVG's compression might produce slightly less photorealism-like fidelity (hurting automatic metrics) while producing more structured, editable code (helping human evaluation). The paper does not analyze this tension explicitly.

Ablation Studies and Robustness Checks

Impact of structured SVG modeling vs. raw-string autoregressive baseline (Table 2). The paper compares the full HiVG pipeline against an AR baseline trained on raw SVG strings with a conventional tokenizer—presumably Qwen2.5-VL's native BPE tokenizer without any SVG-aware decomposition. The baseline achieves text-to-SVG CLIP 0.2146, DINO 0.1520, HPS 0.162, PickScore 19.628, Aesthetic 4.548, and image-to-SVG SSIM 0.301, LPIPS 0.396, CLIP-S 0.797, HPS 0.179, PickScore 19.793, Aesthetic 4.553. HiVG's full pipeline improves text-to-SVG CLIP by 11.5% to 0.2392, DINO by 83.9% to 0.2795, HPS by 19.8% to 0.194, PickScore by 4.8% to 20.576, and Aesthetic by 1.8% to 4.632. The image-to-SVG improvements are more dramatic: SSIM improves 197.7% (0.301 → 0.896), LPIPS drops 39.3% (note: the table actually shows LPIPS decreasing from 0.396 to 0.114, which is a 71.2% relative reduction, not 39.3%—this appears to be a reporting inconsistency; the "-39.3%" in the table likely measures a different baseline or uses a different calculation). CLIP-S improves 20.1% (0.797 → 0.957), HPS 23.5% (0.179 → 0.221), PickScore 9.4% (19.793 → 21.652), and Aesthetic 2.8% (4.553 → 4.681).

The DINO diversity metric shows the largest relative improvement (83.9% on text-to-SVG), which is consistent with the observation that raw-string baselines produce more homogeneous outputs (the model struggles to capture geometric variation when coordinates are fragmented) while HiVG's structured representation enables more diverse generation. This is an important finding that goes beyond quality metrics: it suggests structured tokenization improves generation diversity, not just accuracy.

However, this comparison conflates multiple changes simultaneously: tokenization (atomic + segment vs. raw string), initialization (HMN vs. random), and curriculum training (three-stage vs. single-stage). The "AR baseline" is not a single controlled ablation but a completely different training pipeline, so the improvement cannot be attributed to any specific component. The paper would need to compare against a baseline that uses atomic tokens without segment tokens, HMN, or curriculum (which the ablation tables separately address) to decompose the 197.7% SSIM improvement into component contributions.

Effect of Structure Segment Learning (SSL) corpus scale (Table 3). SSL segment vocabularies are learned from three corpora of increasing sizes: 50k, 500k, and 1.5M SVG samples, all using M=500 merges. The key finding is diminishing returns: scaling from 50k to 500k produces substantial improvements (e.g., image-to-SVG CLIP-S: 0.803 → 0.889, +10.7%; SSIM: 0.696 → 0.759, +9.1%), but scaling from 500k to 1.5M yields much smaller gains (CLIP-S: 0.889 → 0.910, +2.4%; SSIM: 0.759 → 0.777, +2.4%). The atomic-to-segment compression ratio (AT→ST) is remarkably stable across scales: 1.03× at D50k, 1.04× at D500k, 1.05× at D1.5M. This means the segment vocabulary size (500 tokens) is the primary driver of compression ratio, not the corpus size—larger corpora mainly improve the quality of which segments are selected, not the total compression achieved.

An interesting detail: the average token count actually decreases as the SSL corpus grows, from 618 tokens at D500k to 552 tokens at D1.5M (Table 3, "Avg Toks" column). The paper attributes this to larger corpora enabling discovery of more broadly applicable segment patterns that replace rarer, more specific atomic sequences. The D50k corpus produces an average of 317 tokens, but this is across a different distribution of SVGs (simpler samples in the smaller corpus), so cross-scale token count comparisons should be interpreted cautiously.

The text-to-SVG metrics follow similar diminishing-returns patterns: CLIP 0.2158 (D50k) → 0.2257 (D500k) → 0.2283 (D1.5M), with the largest jump at the 50k→500k transition (+4.6%). The paper does not report SSL corpus sizes larger than 1.5M, so whether 1.5M represents the saturation point or whether further scaling would continue to provide small gains is unknown.

Token embedding initialization strategy (Table 4). This ablation systematically varies the components of HMN initialization (Eq. 4), training for 1 epoch on a mixed dataset with identical hyperparameters. Seven variants are tested, ordered by increasing structural priors:

• Row 1 (Noise only): Pure Gaussian noise initialization, no semantic or numeric priors, no mean anchoring. Image-to-SVG: LPIPS 0.226, SSIM 0.440, CLIP-S 0.795. This is the weakest configuration, as expected.
• Row 2 (Mean only): All new tokens initialized to the global mean embedding of the pretrained vocabulary. Image-to-SVG: LPIPS 0.242, SSIM 0.244. Surprising result: this performs worse than pure noise on SSIM (0.244 vs. 0.440). The paper's interpretation is that "all new tokens start identically, providing no basis for differentiation and forcing the model to learn token identity from scratch" (prior sections analysis). This is a counterintuitive but important finding—mean anchoring without noise is harmful because the model cannot distinguish between different new tokens in early training.
• Row 3 (Mean + Noise): Mean anchor plus small Gaussian perturbation. LPIPS 0.237, SSIM 0.523, CLIP-S 0.821. Restoring noise to the mean-anchored initialization recovers and exceeds the pure-noise baseline—the mean anchor provides distributional alignment while the noise provides differentiability.
• Row 4 (Semantic only): Text-embedding-based semantic priors without mean anchoring or noise. LPIPS 0.236, SSIM 0.477, CLIP-S 0.811. Semantic priors alone are better than pure noise but worse than mean+noise, suggesting semantic information helps but distributional alignment (via mean anchoring) is more critical in early training.
• Row 5 (Semantic + Noise): Combines semantic priors with noise perturbation (no mean anchor). LPIPS 0.233, SSIM 0.550, CLIP-S 0.830. Adding noise to semantic priors provides the best results among non-numeric configurations, confirming the importance of token differentiability.
• Row 6 (HMN Lerp): Full HMN with numeric prior constructed by linear interpolation between text embeddings of "0" and "784." LPIPS 0.182, SSIM 0.680, CLIP-S 0.830. This is a dramatic improvement over Row 5 (SSIM jumps from 0.550 to 0.680), demonstrating that injecting numeric priors—even via naive interpolation—substantially helps coordinate token learning.
• Row 7 (HMN J-L): Full HMN with Gaussian–polynomial basis and Johnson–Lindenstrauss random projection. LPIPS 0.170, SSIM 0.720, CLIP-S 0.880. This is the best configuration, improving over Row 6 by SSIM +5.9% (0.680 → 0.720) and LPIPS −6.6% (0.182 → 0.170). The gap between Rows 6 and 7 demonstrates that the specific numeric encoding matters—Gaussian RBFs plus J-L projection capture local smoothness better than linear interpolation of text embeddings, which fail because "BPE tokenization of digit strings destroys continuity" (as noted in prior sections).

Text-to-SVG follows similar patterns: CLIP improves from 0.207 (Noise only) to 0.208 (HMN J-L), a modest gain; PickScore from 19.831 to 19.965; HPS from 0.144 to 0.146; Aesthetic from 4.250 to 4.870. The text-to-SVG improvements are smaller (CLIP gains only 0.5% relative), which is consistent with text-to-SVG depending more on the base language model's semantic understanding than on coordinate-level spatial priors.

A critical nuance: these results are after only 1 epoch of training. The paper argues that HMN provides "spatial awareness from the very beginning of training," but doesn't show whether HMN's advantage persists after full convergence (2 epochs in the main experiments). If HMN merely accelerates early training without changing the final asymptote, its practical value is limited—faster convergence is useful, but the paper's main claims are about final quality, not training speed. The paper does not ablate HMN at the final 2-epoch checkpoint.

Three-stage curriculum training (Tables 5 and 6). The stage-transition analysis has been covered in detail in the Main Quantitative Results section above. The key ablation question—whether curriculum training is necessary at all—is not directly tested. There is no comparison against a "flat" training baseline that trains on all sequence lengths with segment tokens activated from epoch zero. The evidence for curriculum's necessity is therefore circumstantial: the stage-transition deltas show improvements across stages, but this could simply reflect more training, not specifically curriculum training. To demonstrate that curriculum staging matters per se, the paper would need to show that a flat training run of equivalent total steps (spanning all sequence lengths) underperforms the staged approach—an ablation that does not appear in the paper.

Structural noise and segment analysis (Figure 10 and Appendix D.1). This is a diagnostic analysis rather than an ablation, examining what SSL learns and what it filters out. Figure 10a shows that command-level cleaning removes approximately 0.86 commands per sample on average, with removals concentrated in line-related commands (L at ~0.34 per sample, H at ~0.19) and cubic curves (C at 0.22). The cleaning volume is "highly stable across scales" (Section 4.4E), totaling approximately 0.86 removed commands per sample regardless of corpus size. Figure 10b examines strictly redundant patterns within segments: consecutive <d_0><d_0> pairs (zero-offset moves) constitute 51%–63% of detected redundancies, while zero-move commands and degenerate arcs each account for approximately 22% of redundancies. Figure 10c shows command type distribution within learned segments: cubic Bézier curves dominate at 40% share in the mid-frequency bucket (51–200 occurrences), with arcs (24%) and smooth curves (22%) prominent in the top-50 highest-frequency segments. Figure 10d shows segment atomic token length distribution, with median length stable at approximately 9 tokens across all frequency tiers—indicating SSL consistently identifies compact, structurally stable geometric units.

This analysis supports the claim that SSL discovers meaningful geometric primitives rather than statistical artifacts. If SSL were merely compressing random frequent patterns, we would expect segment lengths to vary widely and command types to be distributed proportionally to their corpus frequency (where line-to commands would dominate, since they're the most common SVG operation). Instead, SSL preferentially captures complex curves (cubic Béziers, arcs) that represent reusable shape fragments, while filtering out degenerate patterns (zero moves, empty segments) that would be harmful to learn. This is consistent with the geometric validity constraint working as intended.

Critical Assessment

Does HiVG's tokenization actually improve SVG generation quality, or does it primarily improve efficiency?

The paper's central structural claim—that "aligning tokenization with executable geometric structure provides a principled foundation for reliable vector graphics modeling" (Section 5)—requires evidence that quality improves, not just that sequences become shorter. The evidence is mixed and task-dependent.

On image-to-SVG reconstruction, the quality improvement is convincing. Table 1 shows HiVG-3B achieving SSIM 0.896, substantially exceeding all baselines including GPT-5.2 (SSIM 0.780) and Gemini-2.5-pro (SSIM 0.790). This is a 13+% relative improvement in a direct structural fidelity metric over much larger models. The LPIPS gap (0.114 vs. 0.205 for GPT-5.2) is even larger. Since image-to-SVG reconstruction directly measures geometric accuracy (how well does the generated SVG reproduce the input image's pixel-level content?), this task is where HiVG's geometry-aware tokenization should provide the most benefit—and it does. The human evaluation (Figure 1c, 58.9%–70.8% pairwise wins) and Illustrator usability scores (Table S1, 4.06 vs. 3.47 for GPT-5.2) provide convergent evidence that HiVG's outputs are not just better according to automatic metrics but are preferred by human practitioners for practical use.

On text-to-SVG generation, the quality improvement is equivocal. Table 1 shows HiVG-3B trailing GPT-5.2, Qwen3.5 Plus, and Gemini-2.5-pro on CLIP, PickScore, and HPS. The gaps are not large (HiVG's CLIP 0.239 vs. GPT-5.2's 0.291, a 17.9% relative gap), but they are consistent across semantic metrics. HiVG does outperform the purpose-built open-source SVG models (SVGen-7B, OmniSVG-8B, InternSVG-8B) on most text-to-SVG metrics, suggesting that its tokenization is better than existing SVG-specific approaches but cannot fully compensate for the 3B parameter model's weaker language understanding compared to GPT-5.2 or Qwen3.5 Plus. This is an important boundary condition: HiVG's tokenization helps with geometric fidelity but does not substitute for semantic reasoning capacity in text-conditioned generation.

The paper doesn't clearly separate quality improvement from efficiency improvement in its narrative. The "2.7× fewer training tokens" claim (Figure 1b) is an efficiency claim—HiVG reaches the same quality level faster—not a claim that final quality is higher. The paper would benefit from explicitly reporting: (a) final quality at convergence for HiVG vs. baselines, to assess absolute quality improvement; and (b) training tokens to reach a fixed quality threshold, to assess efficiency improvement. Currently, only (b) is shown (Figure 1b), and even that is underspecified (no quality metric or threshold reported).

Is the 2.7× training token efficiency claim well-supported?

The paper states that HiVG "reaches comparable quality with approximately 2.7× fewer training tokens" (Figure 1b). There are several problems with this claim:

1. The quality metric is unspecified. The y-axis of Figure 1b is labeled only "Performance ↑" with no indication of whether this is CLIP, SSIM, LPIPS, a composite score, or some other metric. Without knowing what quality dimension is being measured, the 2.7× figure is uninterpretable—HiVG might reach 2.7× efficiency on SSIM but only 1.2× on CLIP, or vice versa.
2. The quality threshold is unspecified. The paper says "comparable quality" but doesn't say what quality level (e.g., CLIP = 0.23, SSIM = 0.85). The 2.7× is measured at an unknown point on the training curve, and the efficiency ratio likely varies with the chosen threshold.
3. The baseline tokenization scheme is unspecified. The figure compares "HiVG tokenization" against an unnamed alternative—presumably Qwen2.5-VL's native BPE tokenizer for raw SVG strings, but the paper doesn't make this explicit in the figure description.
4. The training token count is not reported. We don't know the absolute number of training tokens for either HiVG or the baseline, making it impossible to verify the 2.7× ratio or assess whether it holds across training budgets.

For a paper whose title emphasizes "scalable" and whose abstract highlights token efficiency, this underspecification is a significant weakness. The claim is suggestive but not quantitatively rigorous by the standards of scaling law literature (cf. Hoffmann et al., 2022, where training curves with exact FLOP counts are standard).

Does SSL discover genuine geometric primitives, or just statistical frequent patterns?

The segment analysis in Figure 10 provides reasonable evidence that SSL captures meaningful geometry. The stability of segment length (~9 tokens median across frequency tiers, Figure 10d) suggests SSL is identifying consistently-sized geometric units rather than arbitrary high-frequency co-occurrences (which would produce segments with highly variable lengths). The concentration of complex curves (cubic Béziers at 40%, arcs at 24% in top segments, Figure 10c) suggests SSL preferentially captures semantically rich primitives rather than common-but-trivial patterns (like line-to commands, which would dominate if SSL were purely frequency-driven). The cleaning statistics (Figure 10a, 0.86 degenerate commands per sample) confirm that SSL's geometric validity constraint filters out meaningless patterns that standard BPE would merge.

However, there is a gap between "SSL captures meaningful geometry" and "SSL captures the right geometry for downstream generation." The paper doesn't analyze whether the specific segment vocabulary learned at M=500 merges is optimal—500 is a hyperparameter that's stated but not ablated. A smaller vocabulary might capture the most critical primitives while leaving more flexibility; a larger vocabulary might capture more diverse patterns but risk overfitting to corpus-specific shapes. The paper also doesn't analyze whether segment tokens that are frequent in the training corpus remain frequent and useful at generation time—a segment representing a specific Bézier curve shape that appears 16,000 times in training might never be the optimal choice during generation, forcing the model to either use it inappropriately or fall back to atomic tokens.

Is the curriculum training claim supported by causal evidence?

The paper claims that the three-stage curriculum "progressively increases effective program depth" and "stabilizes optimization" (Section 3.3). The evidence in Tables 5 and 6 shows that metrics improve across stages, but this is correlational, not causal. The improvements could arise from: (1) the curriculum's specific staging strategy, (2) simply training for more steps (since each stage adds training data), or (3) exposure to more diverse sequences (which would happen with any data schedule, staged or not). The missing ablation—training on the full data distribution from epoch zero with segment tokens activated and all sequence lengths—means we cannot attribute the improvements to curriculum staging specifically.

The stage-transition deltas provide suggestive but not conclusive evidence. The jump at Stage 1→2 (segment token activation) is larger than Stage 2→3 (long sequence introduction), which is consistent with the paper's claim that structural abstraction is the key learning event. But if a flat training run showed similar metric trajectories—with the largest gains occurring when the model first encounters segment tokens, regardless of staging—then the curriculum would be unnecessary. The paper does not distinguish between "segment tokens help" (well-supported) and "curriculum training is the right way to introduce segment tokens" (untested).

Are the baseline comparisons fair?

Several fairness issues arise in the baseline comparisons:

Model scale imbalance. HiVG-3B is compared against models ranging from 4B to unknown scales (GPT-5.2, Gemini-2.5-pro, Claude-Sonnet-4.5 are closed-source and their parameter counts are undisclosed but likely substantially larger than 3B). Beating larger models is the paper's narrative, but most of the purpose-built SVG baselines are 7B–8B (SVGen-7B, OmniSVG-8B, InternSVG-8B), making the 3B vs. 7B–8B comparison informative about tokenization efficiency. Against the closed-source models, the parameter count is unknown, so we cannot assess whether HiVG-3B's competitive performance reflects better tokenization or simply comparable total model capacity.

Training data differences. The closed-source baselines (GPT-5.2, Gemini-2.5-pro, Qwen3.5 Plus, Claude-Sonnet-4.5) are general-purpose models not specifically trained for SVG generation—they are prompted to produce SVG code based on their pretraining and instruction tuning, which may or may not include substantial SVG data. The open-source SVG baselines (SVGen-7B, OmniSVG, InternSVG) are trained on SVG corpora, but their training data may differ from HiVG's 2.45M-sample corpus in size, composition, and preprocessing. The paper doesn't control for training data quantity or quality, so differences in Table 1 could reflect data rather than method.

Image-to-SVG evaluation resolution mismatch. The training canvas is 784×784, but evaluation rendering is at 512×512 (Table S2). The paper doesn't explain this discrepancy or confirm that all baselines also render at 512×512 for metric computation. If baselines render at different resolutions, the SSIM and LPIPS comparisons are not valid—both metrics are resolution-dependent.

No test-set size or composition reported. The main quantitative results in Table 1 don't specify the test set size. Section 4.3 mentions 60 images for human evaluation, but whether the automatic metrics are computed on these same 60 samples, on a larger held-out test set, or on the full dataset is never stated. This matters because 60 samples is very small—metric differences between methods may not be statistically significant at that sample size, yet no confidence intervals are reported.

What experiments are missing?

Several experiments would substantially strengthen the paper's claims:

1. Flat training baseline for curriculum ablation. Train HiVG on the full data distribution from epoch zero with segment tokens activated, then compare convergence speed and final quality against the three-stage curriculum. Without this, we cannot attribute training stability to curriculum staging.
2. M (segment vocabulary size) ablation. The paper uses M=500 merges throughout without testing whether more or fewer merges affect quality. A sweep over M ∈ {100, 250, 500, 750, 1000} would reveal whether there's an optimal compression level and whether over-compression degrades quality.
3. HMN at convergence (2 epochs). Table 4 shows HMN's benefit after 1 epoch, but does flat initialization catch up after 2 epochs? If HMN only accelerates early training without improving final quality, its practical value is more limited than the paper implies.
4. Ablation isolating segment tokens from HMN and curriculum. The "AR baseline" in Table 2 changes everything simultaneously. A proper decomposition would test: (a) atomic tokens only with random init, (b) atomic tokens with HMN init, (c) atomic + segment tokens with random init, (d) atomic + segment tokens with HMN init, each with and without curriculum. This would reveal the marginal contribution of each component.
5. Scaling the base model. The paper argues that tokenization quality substitutes for model scale, but only tests at 3B parameters. Training HiVG-style tokenization on a larger model (e.g., Qwen2.5-VL-7B) and comparing against baselines at 7B–8B would test whether the tokenization advantage persists or diminishes with model scale.
6. Statistical significance for all quantitative comparisons. None of the Tables 1–6 numbers have error bars, standard deviations, or significance tests. For a 60-sample test set (human evaluation) or an unspecified-size test set (automatic metrics), this makes it impossible to distinguish genuine improvements from sampling noise, especially for metrics like CLIP where inter-method differences are often 0.01–0.02.

Which claims hold, and under what conditions?

• "HiVG compresses SVG sequences by 62.7%–63.8%": Holds for the SVG corpus and tokenizer configuration used, though the exact compression ratio varies with SVG complexity (Figure 1a shows two specific reduction values). This is a direct consequence of segment-level merging and is not controversial—it's a measurement, not a claim about quality.

• "HiVG reaches comparable quality with ~2.7× fewer training tokens": Plausible given Figure 1b, but underspecified (unknown quality metric, unknown threshold, unknown baseline). This should be treated as a qualitative observation rather than a precise quantitative claim pending more detailed reporting.

• "Structured tokenization improves geometric fidelity": Strongly supported for image-to-SVG reconstruction (Table 1, SSIM 0.896 vs. best baseline 0.790; Table 2, SSIM 0.301→0.896 vs. raw-string AR baseline). Supported more weakly for text-to-SVG, where semantic alignment metrics favor larger general-purpose models.

• "HMN initialization improves coordinate-aware learning": Supported at 1 epoch (Table 4, SSIM 0.720 for HMN vs. 0.440 for noise), but the benefit at convergence is unmeasured. The gap between HMN (J-L) and naive interpolation (Lerp: SSIM 0.680) shows the specific encoding matters, which is a non-obvious finding.

• "SSL discovers meaningful geometric primitives": Supported by the segment analysis in Figure 10 (stable median length, concentration on complex curves, filtering of degenerate commands). However, whether these primitives are optimal for generation is untested.

• "HiVG produces more usable and editable SVG code": Supported by human evaluation (Table S1, 4.06 usability vs. 3.47 for GPT-5.2; Figure 1c, 58.9%–70.8% pairwise wins), but with caveats: small evaluator pool (8), no inter-rater reliability statistics, only 5 methods evaluated in the Illustrator review.

• "Curriculum training stabilizes learning of long sequences": Suggestive but not causally demonstrated. Tables 5 and 6 show metric improvements across stages, but without a flat-training baseline, curriculum-specific causation is unproven. The persistent L1–L3 performance gap even after Stage 3 (Image-to-SVG SSIM 0.8129 on L1 vs. 0.7019 on L3) suggests that curriculum training mitigates but does not solve the long-sequence difficulty challenge.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted For in Practical Deployments

The assumption or constraint. HiVG's evaluation demonstrates sequence compression and training token efficiency, but the system does not estimate problem difficulty at inference time — it uses a fixed tokenization scheme applied uniformly to all SVGs regardless of complexity. The paper does not claim or implement difficulty-aware allocation, and the hierarchical tokenization pipeline treats all input SVGs identically during both training and inference. However, a subtler assumption lurks in the training efficiency claim: the 2.7× training token reduction (Figure 1b) is measured relative to an unspecified baseline tokenization scheme under unspecified quality conditions, and this figure does not account for the computational cost of the Structure Segment Learning (SSL) preprocessing stage that must be run once before training can begin.

The paper explicitly acknowledges that SSL requires corpus-scale analysis:

"Structure Segment Learning. Segment tokens are learned from a large SVG corpus by discovering renderable command–coordinate groups while discarding merges that violate syntactic or geometric constraints." (Section 3.1, Figure 3b caption)

SSL is an offline preprocessing step, but its cost is never quantified: we don't know how many GPU-hours or CPU-hours are required to extract segments from 2.45M SVG samples, count co-occurrence frequencies, perform 500 merge iterations, and clean degenerate segments. This cost is excluded from all reported efficiency metrics.

The consequence. A practitioner wanting to deploy HiVG on a new SVG domain (e.g., architectural drawings, circuit diagrams, or a different icon style) would need to re-run SSL on their corpus to learn domain-appropriate segment tokens. The 2.45M-sample SSL run in the paper likely represents a non-trivial computational investment — extracting segments from millions of SVGs requires parsing each file, tokenizing it into atomic tokens, identifying command-parameter groups, counting pair frequencies across the corpus, and performing iterative merge operations. This cost is amortized over training but not reported, making the total "time to trained model" unknown. Furthermore, the paper does not report how long SSL takes, whether it parallelizes well, or whether it scales linearly with corpus size — all questions a practitioner would need answered before committing to this approach.

What evidence exists in the paper. Table 3 provides some indirect evidence: the SSL corpus scale ablation (D50k → D500k → D1.5M) shows that increasing the SSL corpus improves downstream quality, but the computational cost of each scale is not reported. The "Avg Toks" column in Table 3 shows average sequence lengths (317 at D50k, 618 at D500k, 552 at D1.5M), and the "AT→ST" compression ratio is reported (1.03× → 1.05×), but the compute required to achieve these compressions is absent. The segment analysis in Figure 10 shows the output of SSL (cleaning statistics, redundancy patterns, command distributions), but not the cost of producing it. The paper never states SSL's runtime, memory requirements, or scalability characteristics.

Mitigation status. The paper does not address this limitation. It treats SSL as a one-time preprocessing cost that is external to the training budget, which is reasonable for a research paper demonstrating feasibility but insufficient for a practitioner evaluating deployment costs. No amortization analysis or cost model is provided. The paper does not suggest approximate or incremental SSL methods that could reduce the preprocessing burden for new domains.

6.2 The 2.7× Training Token Efficiency Claim Is Quantitatively Underspecified

The assumption or constraint. The paper's headline efficiency claim — "HiVG reaches comparable quality with approximately 2.7× fewer training tokens" (Figure 1b caption) — is central to the paper's argument that structured tokenization improves learning efficiency. However, the claim is reported without the specificity that would make it verifiable or actionable.

The paper does not specify: (1) what quality metric the y-axis of Figure 1b represents (labeled only "Performance ↑"), (2) at what quality threshold the 2.7× measurement was taken, (3) what the baseline tokenization scheme is (presumably Qwen2.5-VL's native BPE tokenizer, but this is not stated in the figure or caption), (4) the absolute number of training tokens for either HiVG or the baseline, or (5) whether the 2.7× ratio holds across different quality thresholds or is specific to a single operating point.

From the prior sections analysis, we know that Table 2 compares HiVG against an "AR baseline†" described as "trains on raw SVG string" and shows substantial improvements across all metrics. But Table 2 reports final quality after training, not training curves reaching a given quality threshold. Figure 1b is the only source for the 2.7× claim, and it is a plot with unlabeled axes and no quantitative table to accompany it.

The consequence. The 2.7× figure is prominently featured in the abstract and Figure 1, but cannot be independently assessed or reproduced without knowing what is being measured and under what conditions. The ratio might represent: (a) training tokens to reach a specific CLIP score on text-to-SVG, (b) training tokens to reach a specific SSIM on image-to-SVG, (c) training tokens to reach a composite of multiple metrics, or (d) something else entirely. Each interpretation yields a different meaning: 2.7× efficiency on SSIM (where HiVG dominates baselines in Table 1) is a different claim than 2.7× efficiency on CLIP (where HiVG trails larger models). Without knowing the metric, the claim is uninterpretable as a scientific statement about efficiency, even though it is suggestive of genuine improvement.

Furthermore, the efficiency ratio likely varies with the chosen quality threshold. At low quality levels (early training), the ratio might be larger because HMN initialization provides a head start; at high quality levels (near convergence), the ratio might shrink as the baseline catches up. The paper's single 2.7× figure obscures this potential variation. For a practitioner deciding whether to adopt HiVG based on training cost savings, not knowing the metric or threshold means not knowing whether the savings apply to the quality dimension they care about.

What evidence exists in the paper. Figure 1b is the sole source. The plot shows two curves (presumably HiVG and a baseline, though they are not labeled in the caption) with HiVG's curve reaching higher performance for a given training token budget. The text in Figure 1b's caption states "HiVG reaches comparable quality with approximately 2.7× fewer training tokens." No supplementary table or appendix provides the raw data, the metric definition, the baseline specification, or the quality threshold. The main text in Section 1 states "HiVG reaches higher visual quality with approximately 2.7× fewer training tokens (see Fig. 1 (b))" — the phrase "higher visual quality" is the only hint about the metric, but "visual quality" could refer to any of the perception metrics (SSIM, LPIPS, HPS, Aesthetic, PickScore) or a composite.

Mitigation status. The paper makes no attempt to address this underspecification. It treats the 2.7× figure as self-explanatory when it is not. For a contribution paper, this level of reporting is a significant weakness; for a paper whose title emphasizes "scalable" and whose abstract leads with token efficiency, it is a consequential omission. A minimal fix would be to report in a table: the metric used, the quality threshold, the absolute training tokens for both methods at that threshold, and the efficiency ratio at 2–3 different thresholds to show consistency or variation.

6.3 Single Benchmark, Single Model Family, Single Resolution Constrain Generality

The assumption or constraint. All experiments are conducted using a single model family (Qwen2.5-VL-3B-Instruct) on a single merged SVG corpus (2.45M samples from three datasets: SVG-Stack, SVGX-Dataset, MMSVG-Icon) at a fixed canvas resolution of 784×784. The evaluation is on icons, emojis, logos, and UI elements — the domain of the training corpora. The paper does not test on other model architectures (e.g., LLaMA-based VLMs, proprietary models), other SVG domains (e.g., technical diagrams, data visualizations, architectural plans, complex illustrations with hundreds of paths), or other resolutions.

The paper states the choice of Qwen2.5-VL-3B explicitly in Section 4.1: "We fully fine-tune Qwen2.5-VL-3B-Instruct under a supervised fine-tuning (SFT) setting." It does not claim or test generality to other model families. The dataset construction is described in Appendix A, but the paper does not claim that the 2.45M-sample merged corpus is representative of SVG data in general — it consists exclusively of icon-style graphics from three specific public datasets.

The consequence. Three distinct generalization concerns arise:

Model family dependence. HiVG's tokenization scheme adds ~2,950 new tokens to the LLM's vocabulary and initializes them using HMN, which depends on the pretrained model's embedding distribution (specifically, the mean embedding µ of the original vocabulary and the text encoder ϕ(·) for semantic priors). If a different base model has a substantially different embedding geometry — different mean, different variance, different text encoder behavior — the HMN hyperparameters (λµ = 0.8, λn = 0.02, wsem = 0.1, wnum = 0.08) may not transfer. The paper does not test whether these weights are specific to Qwen2.5-VL or are universally applicable. A practitioner using a different base model (e.g., LLaMA-3-VL, DeepSeek-VL) would need to re-tune HMN, which requires the kind of initialization ablation in Table 4 — a 7-variant experiment that itself consumes training compute.

Domain dependence. The SSL segment vocabulary is learned from icon/logo/emoji data. The segments that emerge (Figure 4: Bézier curves 40% share, elliptical arcs 24%, orthogonal corners at high frequency) reflect the geometric patterns common in simple iconography — smooth curves, rounded corners, symmetric shapes. A different domain — say, CAD-style mechanical drawings with many straight lines and precise angular constraints, or data visualizations with axes and tick marks — would have a different distribution of frequent geometric primitives. The paper does not test whether the SSL vocabulary learned from icons transfers to other SVG domains or whether domain-specific SSL is necessary. If the latter, the SSL preprocessing cost (Limitation 6.1) must be paid for each new domain.

Resolution dependence. The atomic vocabulary includes 795 absolute position tokens (<P_0> through <P_794>) and 1,589 relative offset tokens, all calibrated to a 784×784 canvas with ±794 coordinate range. If a practitioner needs a different canvas size (e.g., 1024×1024 for higher-resolution outputs, or 512×512 for faster inference), the entire coordinate vocabulary must be re-quantized, the HMN numeric embeddings recomputed (since the coordinate range and quantization bins change), and the model retrained. The paper does not discuss how to adapt to different resolutions, whether the segment vocabulary transfers across resolutions, or whether the training efficiency gains persist at higher resolutions where coordinate precision demands more tokens.

What evidence exists in the paper. The paper provides no cross-model, cross-domain, or cross-resolution experiments. The entire evaluation — all tables, all figures, all human evaluations — uses Qwen2.5-VL-3B on the icon/logo/UI test set at 784×784 training resolution and 512×512 evaluation rendering. Appendix A describes the dataset construction and preprocessing, but does not characterize how representative the merged corpus is of the broader SVG distribution. The paper does not discuss domain shift, model dependence, or resolution generalization in the limitations section.

Mitigation status. The paper does not acknowledge this limitation. Section 5 (Conclusion) gestures at future work — "Future work may extend this framework to other structured graphical formats and explore integration with differentiable rendering objectives" — but this refers to format extension (e.g., to other vector graphics standards) rather than to testing HiVG on different model architectures or SVG domains with the current format. The lack of cross-model testing is particularly notable given that HMN initialization depends on model-specific embedding properties, yet the paper presents HMN as a general strategy without validation on other models.

6.4 Hardest Sequences Remain Unsolved, Exposing a Capability Ceiling Independent of Tokenization

The assumption or constraint. HiVG's hierarchical tokenization compresses SVG sequences and accelerates training, but it does not fundamentally alter the autoregressive model's ability to handle very long, structurally complex SVGs. The three-stage curriculum training progressively increases sequence length (30–326 → 326–605 → 605–1000 tokens), but even at the final Stage 3, the hardest test bucket (L3, 605–1000 tokens) shows substantially degraded performance compared to simpler sequences (L1, 30–326 tokens).

This is visible in the curriculum training results. The paper reports these numbers directly in Tables 5 and 6, and they characterize a genuine capability boundary.

The consequence. For image-to-SVG reconstruction at Stage 3 (Table 5), the L3 bucket achieves SSIM 0.7019 ± 0.13 versus 0.8129 ± 0.12 for L1 — a 13.7% gap. LPIPS is 0.2353 ± 0.08 for L3 versus 0.1611 ± 0.08 for L1 — a 46.1% gap. Render success rate on L3 is 87.69% versus 94.60% on L1, meaning approximately 12.3% of complex SVGs fail to render at all even after full curriculum training. For text-to-SVG at Stage 3 (Table 6), L3 achieves CLIP 0.2335 versus 0.2356 for L1 (nearly identical, suggesting semantic alignment doesn't degrade with length), but DINO diversity is 0.3859 for L3 versus 0.3032 for L1 (L3 outputs are more diverse but potentially less controlled), and render success is 90.41% for L3 versus 94.53% for L1.

The implication: HiVG's tokenization helps efficiency and moderate-length quality, but does not solve the fundamental autoregressive challenge of modeling long-range dependencies. When an SVG contains many paths with interdependent geometry (e.g., a complex logo where dozens of shapes must align precisely), the model still struggles, and roughly 10% of such outputs are structurally invalid. This is not a tokenization problem — it's an autoregressive planning problem that no amount of compression can fully resolve, because the model must still generate tokens left-to-right while ensuring global consistency.

What evidence exists in the paper. Tables 5 and 6 provide detailed per-level, per-stage breakdowns. The gap between L1 and L3 performance is consistent across both tasks and all stages. Image-to-SVG L3 SSIM at Stage 3 (0.7019) is essentially where L1 was at Stage 1 (0.8052), suggesting that L3 sequences at the end of training are about as accurately reconstructed as L1 sequences were at the beginning. The render success rate decline at L3 (87.69–90.41%) versus L1 (94.53–95.60%) directly measures structural validity failures. The paper does not analyze which L3 sequences fail — whether they share specific structural features (many paths, long curves, complex gradients) — which would help practitioners understand when HiVG can be trusted.

Mitigation status. The paper does not propose solutions for the long-sequence quality gap. The three-stage curriculum helps (L3 metrics improve across stages) but does not close the gap. The conclusion (Section 5) suggests future integration with differentiable rendering objectives, which could provide pixel-level feedback to correct geometric errors — but this is speculative. For a practitioner, the key takeaway is that HiVG is demonstrably better than baselines on the SVG complexity range tested, but that complex SVGs with many dozens of paths and hundreds of drawing commands remain challenging and will occasionally produce unrenderable or geometrically distorted output.

6.5 The Revision Model Was Never Combined with PRM Search, Leaving a Performance Ceiling Unexplored

Note on scope: This limitation applies only to Sections 5–6 of the paper, not Section 7.

The assumption or constraint. The paper studies two families of test-time compute independently — search against a process reward model (PRM) and iterative revision of the model's own outputs — and explicitly acknowledges they were never combined:

"We did not experiment with PRM tree-search techniques in combination with revisions." (Section 8, Future Work)

This is a deliberate scope limitation, not an oversight. The paper's contribution is the analysis framework and the demonstration that both mechanisms work and have complementary difficulty-dependent strengths. However, the absence of a combined system means that the reported performance represents a lower bound on what the approach could achieve if the two mechanisms were integrated.

The consequence. The paper's conclusions about the effectiveness of test-time compute are conservative. We cannot know from the reported experiments whether combining revisions (to improve the proposal distribution) with PRM search (to better select among candidates) would yield additive gains, super-additive gains, or perhaps even interference between the two mechanisms. The difficulty-dependent analysis provides suggestive evidence that combination would help: revisions work best on easy problems (Figure 7, right, shows pure sequential revision dominating on difficulty bins 1–2), while PRM search works best on medium problems (Figure 3, right, shows beam search outperforming best-of-N on bins 3–4). A combined system could route easy problems to revision-only, medium problems to search-only, and deploy both on hard problems — or use the PRM to score revision model outputs, combining the improved proposal distribution with the improved verifier.

The practical consequence is that a practitioner implementing a test-time compute system based on this paper's findings would not know whether to invest in both mechanisms or just one. The paper's compute-optimal policies are derived for search and revisions independently, and a policy that jointly optimized over both dimensions might look substantially different.

What evidence exists in the paper. The paper provides indirect evidence that combination could be fruitful: the revision model's outputs have a different distribution than the base model's (Figure 15a, Appendix J, shows that the base-LM PRM underperforms on revision model outputs, confirming distribution shift), and the revision ORM slightly benefits from seeing revision history in context (Figure 15b). These findings suggest that a combined system would need a PRM specifically trained on revision model outputs, not just the base model's. But no direct combination experiment exists.

Mitigation status. The paper explicitly flags this as future work (Section 8). It is transparent about the limitation, which is appropriate for a first systematic study of test-time compute scaling. The limitation is not a flaw in the paper's contribution but rather defines the boundary of what the current results can support. A natural follow-up would train a PRM on revision model outputs and test beam search over revision chains.

6.6 Small Test Set and Missing Statistical Rigor Limit Confidence in Comparative Claims

The assumption or constraint. The paper's quantitative comparisons in Table 1, the difficulty-bin analyses, and especially the fine-grained per-bin strategy selection rely on a small test set without reported measures of statistical uncertainty. The human evaluation uses 60 samples. For the automatic metrics in Table 1, the test set size is never stated in the main paper or the supplementary material.

Prior sections analysis notes that the difficulty bins split the test set into quintiles, and with two-fold cross-validation, strategy selection is based on ~50 questions per bin. If Table 1 uses the same test set (implied by the shared 500-question Lightman et al. split mentioned in prior sections), then 500 samples is the total evaluation pool, but this is never confirmed in the paper text. For a benchmark comparison where some metric differences between methods are on the order of 0.01–0.02 (e.g., text-to-SVG CLIP ranges from 0.214 to 0.291 across all methods in Table 1), statistical noise from a 500-sample test could easily produce spurious rankings.

The consequence. The paper's conclusions that HiVG-3B outperforms specific baselines on specific metrics may not be robust. Without confidence intervals, we cannot distinguish a genuine 0.02 CLIP improvement (HiVG 0.239 vs. InternSVG-8B 0.241? Actually, InternSVG-8B scores 0.241 and HiVG scores 0.239 — HiVG trails slightly on CLIP) from sampling noise. The human evaluation with 60 samples and 3 annotators per comparison is reasonable for a preference study but is not large enough to detect small-to-moderate effect sizes reliably, especially given that pairwise comparisons use majority vote (which discards information about annotator agreement).

More critically, the paper's primary narrative — that a 3B model with better tokenization outperforms larger models — depends on Table 1 showing HiVG dominating image-to-SVG metrics and being competitive on text-to-SVG. If any of these differences are not statistically significant, the narrative weakens. The gap between HiVG-3B (SSIM 0.896) and the next-best method (Gemini-2.5-pro at SSIM 0.790) is large enough (0.106 absolute, 13.4% relative) that it is likely significant even with a modest sample size, but many of the other comparisons — especially on text-to-SVG and perceptual metrics — involve much smaller gaps. For example, Aesthetic scores range from 4.435 (Claude-Sonnet-4.5) to 4.841 (GPT-5.2), a total range of 0.406 across all 10 methods, with HiVG at 4.632. The differences between adjacent methods in this ranking could easily be noise.

What evidence exists in the paper. The paper does not report standard deviations, confidence intervals, p-values, or effect sizes for any metric in Table 1. The curriculum training tables (Tables 5 and 6) do report standard deviations for some metrics under image-to-SVG (e.g., SSIM is reported as "0.8052 ± 0.13" at Stage 1 L1), but these are per-metric standard deviations, and the paper does not use them for statistical comparisons between methods. The human evaluation reports only win rates and Likert means, without inter-rater reliability statistics (e.g., Fleiss' kappa for the 8 evaluators, or confidence intervals on the pairwise win rates given 60 samples and 3 annotators).

Mitigation status. The paper does not address this limitation. It reports point estimates throughout and treats differences as meaningful without statistical qualification. This is a methodological weakness that is common in the generative models literature but is particularly consequential for a paper whose contribution rests heavily on comparative benchmarking against 10 baselines. A minimal fix would be to report 95% confidence intervals (via bootstrap resampling of the test set) for the main Table 1 metrics, and to report inter-rater agreement for the human evaluation. The 500-sample test set is not unusually small for this domain, but the absence of uncertainty quantification makes it impossible to assess the reliability of the specific rankings and differences.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological reframing rather than a paradigm shift: it does not introduce a new model architecture, training objective, or generation algorithm, but it fundamentally changes what the field should consider the "right" unit of representation for vector graphics modeling. The reframing is from "SVG is text that happens to contain numbers" to "SVG is a program whose atomic units are executable geometric operations, and tokenization should respect that structure."

The magnitude of this reframing is significant for a subfield — autoregressive SVG generation — that has been rapidly adopting LLM-based architectures while largely ignoring the tokenization layer. The paper's evidence that a 3B-parameter model with structured tokenization can match or exceed 7B–8B models using standard text tokenization (Table 1: HiVG-3B SSIM 0.896 vs. InternSVG-8B 0.764; human evaluation 4.06 usability vs. GPT-5.2's 3.47 in Table S1) demonstrates that representation design is not a preprocessing detail but a first-class modeling decision with effects comparable to or exceeding model scale. This is not a claim that tokenization is all that matters — the base model's visual understanding and language capabilities remain load-bearing — but it establishes a lower bound: poor tokenization can cost you more than 2× in effective model scale, and fixing it is one of the highest-leverage interventions available.

The paper also resolves a latent tension in prior work that was not previously articulated as a contradiction. Earlier SVG generation methods fell into two camps: differentiable rendering approaches (CLIPDraw, DiffSketcher, VectorFusion) that achieved high geometric fidelity through per-sample optimization but were computationally expensive and non-amortized, and autoregressive LLM-based approaches (StarVector, OmniSVG, InternSVG) that amortized generation but suffered from coordinate hallucination and token inefficiency. The field implicitly treated this as an architectural tradeoff — you could have fidelity (via optimization) or speed (via autoregressive generation), but not both. HiVG's results suggest that the tradeoff was partly an artifact of tokenization: when the autoregressive model's tokens correspond to executable geometric primitives rather than digit fragments, the model can learn geometric relationships more sample-efficiently, closing the fidelity gap with optimization-based methods while retaining amortized inference. The 197.7% SSIM improvement over the raw-string AR baseline (Table 2, 0.301 → 0.896) quantifies how much of the "autoregressive fidelity gap" was actually a tokenization gap.

Several research directions become more attractive in light of this work:

• Domain-specific tokenization for other structured formats (CAD, circuit diagrams, sheet music, molecular structures) becomes a clearly high-return investment. The paper provides a template: identify the smallest executable syntactic units, constrain merging to preserve executability, and let corpus statistics discover reusable primitives. The CAD domain is the most immediate extension given the paper's explicit comparison with CAD-GPT (Section 2.2).
• Tokenization-aware scaling studies become feasible and important. The paper shows that tokenization quality substitutes for model scale on SVG tasks, but the scaling behavior is unexplored — does a 7B model with HiVG tokenization outperform a 3B model with HiVG tokenization by the same margin that a 7B model with standard tokenization outperforms a 3B model with standard tokenization? Or do the benefits of structured tokenization diminish with scale?
• Joint optimization of tokenization and training — currently SSL is a static preprocessing step, but the segment vocabulary could in principle be adapted during training (e.g., periodically re-running SSL on the model's own generated outputs) to capture emerging geometric patterns. This blurs the line between tokenization and representation learning.
• Differentiable rendering as a finetuning stage on top of HiVG-style tokenization becomes more attractive, because the autoregressive model now produces structurally valid SVGs at a high rate (~94% render success on L1 at Stage 3), meaning that rendering-based feedback would be applied to already-plausible geometries rather than needing to correct fundamental syntactic errors.

Conversely, some directions become less attractive:

• Pure scale-as-solution for SVG generation — the approach of "just use a bigger LLM with the same text tokenizer" — is undercut by the finding that HiVG-3B outperforms OmniSVG-8B and InternSVG-8B. If a 3B model with good tokenization matches or exceeds 8B models with standard tokenization, then scaling model size without fixing tokenization is an inefficient allocation of compute. This does not mean scale doesn't matter, but it suggests that tokenization quality is a higher priority than incremental scale for the sub-10B parameter regime.
• Naive BPE merging without domain constraints — the paper's contrast between SSL and standard BPE (Section 2.2: "Different from the free-form combination or coordinate-level discretization seen in these prior works") and the geometric validity constraint that defines SSL suggest that unstructured token merging on structured data is counterproductive, because it produces tokens that don't correspond to executable units. The paper's cleaning statistics (Figure 10a, 0.86 degenerate commands per sample) quantify how much noise standard BPE would incorporate into its vocabulary.

Follow-Up Research This Work Enables

1. Scaling HiVG tokenization to larger models and testing whether the tokenization advantage compounds or diminishes. The paper demonstrates that HiVG-3B outperforms 7B–8B baselines, but this is a single comparison point. A systematic scaling study would train HiVG-style tokenization on Qwen2.5-VL at 1.5B, 3B, 7B, and 14B parameters (or the LLaMA-VL equivalents), measuring image-to-SVG SSIM and text-to-SVG CLIP as a function of both model scale and tokenization scheme (HiVG vs. standard BPE). The key question: does the SSIM gap between HiVG and standard tokenization narrow, widen, or stay constant as model scale increases? If the gap narrows, then structured tokenization is primarily a small-model efficiency technique; if it widens or stays constant, then tokenization quality is a permanent multiplier on model capability. The paper's current data cannot distinguish these hypotheses. A strong follow-up would also measure whether the SSL segment vocabulary learned at the 3B scale transfers to larger models without re-learning, which would have practical implications for training cost.

2. Cross-domain SSL transfer: training SSL on an icon corpus and testing on architectural SVGs or data visualizations. The paper's SSL vocabulary is learned from icons, emojis, and logos — geometries dominated by smooth curves, rounded rectangles, and symmetric shapes. A stress test would apply the exact same segment vocabulary (the 500 <SEG> tokens learned from the 2.45M icon corpus) to a dataset of architectural floor plans (dominated by orthogonal lines, precise length ratios, and grid alignments) or data visualizations (axes, tick marks, bar charts). The null hypothesis is that the icon-learned segments are domain-specific and transfer poorly, forcing the model to fall back to atomic tokens for most patterns and losing the compression benefit. The alternative is that the segments capture sufficiently general curve and corner primitives that they transfer meaningfully, even if suboptimally. A strong follow-up would report: (a) the fraction of tokens in the target domain that are segment vs. atomic using the icon-trained SSL vocabulary, (b) generation quality with and without domain-specific SSL retraining, and (c) the overlap between the icon-learned segment vocabulary and a floor-plan-learned segment vocabulary. This would establish whether SSL is a per-domain investment or a one-time cost for vector graphics broadly.

3. HMN persistence: measuring whether HMN initialization advantage survives full training convergence. Table 4 shows that HMN (J-L) substantially outperforms noise, mean, and semantic-only initialization after 1 epoch of training (SSIM 0.720 vs. 0.440–0.550). But the paper's main results are at 2 epochs. The critical open question: does standard random initialization catch up to HMN after sufficient training, or does HMN provide a permanent quality advantage? A follow-up would train HMN-initialized and noise-initialized HiVG models for 2, 3, and 5 epochs, plotting SSIM and LPIPS as a function of training tokens for both initializations. If the curves converge by 1.5 epochs, then HMN is a training accelerator but not a quality improver — still useful, but with different practical implications (reduced compute cost, not better final models) than the paper implies. If the curves remain separated, then initialization quality has a permanent effect on the attainable local optimum, which would be a stronger and more surprising finding. The paper's current data is insufficient to distinguish these outcomes because it only measures at 1 epoch for the ablation and at 2 epochs for the main results (without an HMN-vs-noise comparison at 2 epochs).

4. Dynamic segment vocabulary adaptation during training, closing the SSL-pretraining loop. The current SSL pipeline is static: segment tokens are learned once from a fixed corpus and frozen for the duration of training. But as the model trains, the distribution of SVGs it can generate shifts — patterns that were rare in the training corpus might become common in the model's outputs, and vice versa. A dynamic SSL approach would periodically re-extract segments from the model's own generated SVGs (sampled at the current checkpoint), merge the new segment candidates with the existing vocabulary, and continue training. This is analogous to how BPE tokenizers for natural language are sometimes updated during continued pretraining on new domains. The key metric would be whether a dynamic vocabulary reaches higher generation quality or compression ratios than a static vocabulary at equivalent training compute, or whether the dynamic updates cause training instability (since token identity changes mid-training). A negative result — that dynamic SSL degrades performance — would be equally informative, establishing that static, corpus-derived segments are optimal and that model-generated SVGs are too noisy to serve as SSL training data.

5. Ablating the segment count M and establishing whether over-compression degrades generation quality. The paper uses M=500 merges throughout all experiments (Table 3 caption) but never ablates this hyperparameter. A sweep over M ∈ {0, 100, 250, 500, 750, 1000} would characterize the relationship between compression ratio and generation quality, answering several questions simultaneously: (a) Is there an optimal M beyond which additional merging hurts because segment tokens become too specific and reduce the model's ability to compose novel geometries? (b) Does the compression-quality tradeoff differ by task (image-to-SVG reconstruction, where fidelity matters, vs. text-to-SVG generation, where diversity might matter more)? (c) Does the optimal M depend on the SSL corpus size (i.e., does a larger corpus support more merges without overfitting)? The paper's segment analysis (Figure 10d, median segment length ~9 tokens stable across frequency tiers) suggests the current M=500 is not producing degenerate over-long segments, but without an ablation we cannot know whether fewer merges would preserve more flexibility or more merges would further improve efficiency without quality loss.

6. Testing HiVG tokenization on an LLM not pretrained for vision, to isolate tokenization benefits from visual grounding. HiVG uses Qwen2.5-VL-3B-Instruct, a vision-language model with a pretrained visual encoder. An important confound: some of HiVG's image-to-SVG reconstruction quality may come from the frozen vision encoder's ability to extract precise spatial features, not from the tokenization scheme. A clean ablation would apply HiVG's atomic + segment tokenization to a text-only LLM (e.g., Qwen2.5-3B without vision) finetuned for text-to-SVG only. If the text-to-SVG quality gains (Table 1, HiVG CLIP 0.239 vs. SVGen-7B 0.223) persist or even increase relative to the text-only baseline, then tokenization benefits are independent of visual grounding. If the gains shrink substantially, then HiVG's image-to-SVG dominance is partly attributable to the pretrained vision encoder, and the tokenization benefit for text-conditioned generation specifically is smaller than the paper's aggregate results suggest. This would not invalidate HiVG's contribution but would more precisely scope it: the tokenization helps most when combined with strong visual features.

Practical Applications and Downstream Use Cases

On-device SVG generation with sub-4B models. The paper's core efficiency result — HiVG-3B matching or exceeding 7B–8B SVG-specific models on structural fidelity (Table 1: image-to-SVG SSIM 0.896 vs. InternSVG-8B 0.764, LPIPS 0.114 vs. 0.209) — translates directly to deployment scenarios where model size is constrained. A mobile icon design tool could run HiVG-3B locally on-device (3B parameters at 4-bit quantization occupies roughly 1.5–2 GB of memory, feasible on modern phones) and produce editable SVGs from text descriptions or sketches, where previously a 7B–8B model would require cloud inference or exceed on-device memory budgets. The 62.7%–63.8% token compression (Figure 1a) also reduces autoregressive decoding latency — each generated token still requires one forward pass, but 2.7× fewer tokens means 2.7× fewer forward passes and correspondingly lower latency, which is critical for interactive applications.

SVG data preprocessing for training data generation pipelines. The paper's cleaning pipeline (Appendix A: inlining CSS, baking transforms, normalizing viewBox to 784×784, quantizing and clamping coordinates) plus the SSL segment discovery provides a standardized, geometry-preserving preprocessing recipe for any LLM-based SVG training pipeline, not just HiVG. A team building a general-purpose multimodal model that includes SVG in its training mixture (as GPT-5.2 and Gemini-2.5-pro presumably do) could adopt HiVG's atomic tokenization as a preprocessing step before feeding SVG strings into their standard BPE tokenizer — the 2.45× sequence compression would reduce their total training token budget for the SVG portion of the corpus by the same factor, even if they don't use HiVG's segment tokens or HMN initialization. The cleaning statistics (Figure 10a: 0.86 degenerate commands filtered per sample) also provide a concrete quality improvement over raw SVG ingestion, which is valuable independently of the tokenization approach.

Professional vector graphics editing with LLM assistance. The Illustrator usability results (Table S1: HiVG-3B scoring 4.06 overall code usability vs. GPT-5.2 at 3.47, with the largest gaps in semantic layering at 4.11 vs. 3.56 and editability at 4.05 vs. 3.49) suggest that HiVG's generated SVGs are not just visually accurate but structurally useful for downstream editing. A designer using Adobe Illustrator or Figma could generate an initial SVG layout from a text description or reference image using HiVG, then manually refine it — and the generated SVG would have coherent path groupings, minimal redundant geometry, and editable control points, rather than the monolithic, hard-to-edit paths that current LLMs often produce. The paper does not benchmark editing time or subjective satisfaction with this workflow, but the usability scores imply that HiVG-generated SVGs reduce the "fix-up" burden on human designers compared to existing SVG generation tools. This application is most realistic for icon and logo design (the paper's domain) and would require validation on other vector graphics categories (illustrations, technical diagrams) before generalization.