Directional Textual Inversion for Personalized Text-to-Image Generation

ArXiv: 2512.13672

🎯 Pitch

Standard Textual Inversion for personalized image generation often ignores your prompt because the learned token's magnitude inflates wildly during training, drowning out other instructions. The authors prove that only the direction encodes the concept, fix the magnitude to a normal range, and optimize on a sphere—instantly making prompts far more faithful while also unlocking smooth, semantically meaningful blends between learned ideas.

---

1. Executive Summary

This paper proposes Directional Textual Inversion (DTI), a novel framework for personalized text-to-image generation that decouples token embeddings into magnitude and direction, fixing the magnitude to an in-distribution scale while optimizing only the direction on the unit hypersphere via Riemannian SGD. Evaluated on Stable Diffusion XL and SANA 1.5 using the DreamBooth dataset, DTI identifies that semantic information is primarily encoded by embedding direction—while inflated embedding norms emerging during standard Textual Inversion attenuate positional information and stagnate residual updates in pre-norm Transformer blocks—and addresses this by reformulating direction learning as MAP estimation with a von Mises-Fisher prior that provides a constant-direction gradient pull. DTI substantially improves text fidelity over baseline TI and CrossInit while maintaining subject similarity, achieving roughly 0.522 text alignment on SDXL versus 0.292 for vanilla TI, and the hyperspherical parameterization enables smooth, semantically coherent interpolation between learned concepts via spherical linear interpolation (SLERP)—a capability absent in standard TI—establishing that direction-only optimization with controlled magnitude is sufficient for prompt-faithful personalization, with gains most pronounced when prompt complexity increases.

2. Context and Motivation

The Core Problem: Textual Inversion Promises Efficiency but Delivers Unreliable Prompt Fidelity

The fundamental problem this paper addresses is deceptively simple: Textual Inversion (TI), the most lightweight and widely-adopted personalization method for text-to-image models, frequently fails to respect complex text prompts during generation. When a user provides a few images of their pet dog and asks the model to generate "a painting of <dog> wearing a santa hat in a snowy landscape," standard TI often produces an image of the dog with no hat, no snow, or no painting style — the personalized concept dominates the generation at the expense of prompt adherence.

This matters for several practical reasons the paper identifies in Section 1:

• Workflow integration: TI has become a foundational component in numerous personalization frameworks (Hao et al.'s ViCo, Kumari et al.'s multi-concept customization, Tewel et al.'s Key-Locked, Lee et al.'s DCO) precisely because of its efficiency — it requires optimizing only a single embedding vector (a few kilobytes of storage) rather than fine-tuning entire model weights. Any improvement to TI's prompt fidelity therefore amplifies a broad ecosystem of downstream methods.

• Creative control: Users of personalized generation want to place their concepts in diverse contexts — different artistic styles, backgrounds, interactions with other subjects, attribute modifications. When TI fails to respect these contextual prompts, the creative flexibility that personalization promises is undermined.

• Practical deployment: Unlike encoder-based approaches (Wei et al.'s ELITE, Ruiz et al.'s HyperDreamBooth) that require substantial pre-training, or fine-tuning methods like DreamBooth that demand significant GPU memory and storage per concept, TI's embedding-only optimization runs in approximately 7 minutes on a single GPU and stores only a token embedding. This efficiency advantage is real and substantial, but it is compromised if the generated outputs don't faithfully reflect user intent.

The Gap: Why TI Fails on Complex Prompts — and Why Existing Fixes Miss the Root Cause

The paper argues that prior work has not adequately characterized why TI's prompt fidelity degrades, nor addressed the underlying optimization dynamics responsible. There are two categories of existing approaches, each with specific shortcomings:

Category 1: Enriched embedding spaces that sacrifice TI's efficiency advantage. Methods like P+ (Voynov et al., 2023) expand the representation beyond a single token embedding to multiple layer-specific embeddings, while NeTI (Alaluf et al., 2023) introduces neural mapping networks to produce richer conditioning. These approaches partially improve prompt alignment but "introduce significant computational overhead that undermines TI's efficiency advantage" (Section 1). They also "do not directly address the underlying optimization dynamics of TI, leaving the fundamental factors that govern semantic alignment in embedding-based personalization unclear."

Category 2: Initialization and regularization heuristics without geometric understanding. CrossInit (Pang et al., 2024) offers improved initialization by replacing the initialization token with the output of the text encoder and adds a regularization loss. While this provides some benefit with minimal computational overhead (Section 5.1), it treats the symptom rather than the mechanism — it provides a better starting point and a generic regularizer but doesn't address why embeddings drift during optimization or how the geometry of the token embedding space should constrain learning.

The deeper gap: no systematic analysis of token embedding geometry during optimization. The paper positions its core contribution as filling this vacuum. Prior work studied what to optimize (embedding vectors, sometimes enriched) and how to regularize (initialization strategies, auxiliary losses), but nobody had asked the more fundamental questions: What happens to the magnitude and direction of token embeddings during TI optimization? Does the geometry of the pre-trained embedding space impose constraints on what constitutes a valid learned token? How do violations of these constraints manifest as generation failures?

The paper's Section 2 provides the first rigorous answer to these questions, establishing two key findings that prior work missed entirely:

Finding 1: Semantic information is directional. Through nearest-neighbor analysis comparing cosine similarity (direction-only) versus Euclidean distance (magnitude-sensitive), the paper demonstrates that semantically coherent neighbors are recovered only by cosine similarity (Table 1, Section 2.1). For the token "apple," cosine-based neighbors include "apples," "fruit," and "pear," while Euclidean-based neighbors are unrelated tokens with similar magnitudes. This aligns with the foundational principle of semantic vector spaces (Mikolov et al., 2013; Pennington et al., 2014) but had not been explicitly connected to optimization behavior in personalization.

Finding 2: Standard TI produces embeddings with drastically inflated norms — and this is the root cause of poor prompt fidelity. As shown in Figure 1(a), TI produces token embeddings with norms often exceeding 20, compared to approximately 0.4 for the pre-trained vocabulary. This is not a minor deviation — it's a 50× inflation. The paper demonstrates that simply rescaling the embedding's norm after training partially recovers text alignment, but at the cost of degrading subject similarity (Table 2, "TI-rescaled" row), confirming that magnitude inflation is causal but that post-hoc rescaling is insufficient.

Why Large Magnitudes Matter: The Pre-Norm Transformer Bottleneck

The paper provides the first theoretical explanation for why inflated embedding norms harm prompt fidelity in pre-norm Transformer architectures (which dominate modern text encoders like CLIP and Gemma). This analysis, presented in Section 2.2 with formal proofs in Appendix B, identifies two complementary mechanisms:

Effect I: Positional information is attenuated (Lemma 1). After LayerNorm or RMSNorm, the normalized signal feeding attention and MLP sub-layers becomes dominated by the token's direction when its magnitude is large. Positional information contributes only $\mathcal{O}(1/m)$ to the normalized signal $\text{Norm}(m\mathbf{v} + \mathbf{p})$, where $m$ is the magnitude and $\mathbf{p}$ is the positional embedding. Intuitively, a very large-norm token forgets where it is in the sequence — it can't effectively contextualize with surrounding words like "wearing a santa hat" or "in a snowy landscape" because the normalized representation barely registers positional relationships.

Effect II: Residual updates stagnate (Lemma 2, Proposition 1, Corollary 1). In pre-norm blocks of the form $\mathbf{x}^{(\ell+1)} = \mathbf{x}^{(\ell)} + F_\ell(\text{Norm}(\mathbf{x}^{(\ell)}))$, the sub-layer outputs $F_\ell(\cdot)$ are computed from normalized inputs and thus have bounded magnitude. When the hidden state $\mathbf{x}^{(\ell)}$ has large norm, the relative change from the residual update becomes tiny — the directional change (turning angle) of the hidden state decreases in proportion to $1/\|\mathbf{x}^{(\ell)}\|$. Proposition 1 proves that across $L$ pre-norm blocks, the total accumulated directional drift is bounded by $\frac{\pi}{2} \frac{S_L}{\|\mathbf{x}^{(0)}\|_2 - S_L}$, where $S_L$ is the sum of per-block sub-layer capacity bounds. Corollary 1 shows this bound goes to zero as the initial magnitude scales: $\angle(\alpha\mathbf{x}^{(0)}, \mathbf{x}^{(L)}(\alpha)) \xrightarrow[\alpha\to\infty]{} 0$. This is directional freezing — large-norm hidden states become stuck in their direction and are essentially passed through the encoder as identity mappings, preventing the layers from refining the token's representation based on context.

Together, these effects explain the empirical failure pattern: personalized tokens with inflated norms cannot integrate contextual information from the prompt. The concept token becomes dominant, overshadowing style descriptions, background details, and interactions with other subjects. This is why TI generates the dog but omits the hat and snow — the personalized token's signal drowns out everything else.

Empirical Validation of the Theory

The paper doesn't just present theory — it validates both effects empirically in Section 2.3, which is important because it transforms the analysis from plausible hypothesis to demonstrated mechanism:

• Positional attenuation validation (Figure 2): A 2-layer MLP classifier trained on the frozen base text encoder to predict absolute token position from $\text{LN}(\mathbf{e} + \mathbf{p})$ achieves 100% accuracy on normal inputs. When a single token's embedding magnitude is artificially scaled by factors up to 16×, accuracy deteriorates rapidly. Crucially, TI-trained embeddings (with naturally inflated norms) collapse to near-zero positional accuracy, while DTI-trained embeddings remain fully recoverable. This directly corroborates Lemma 1.

• Residual stagnation validation: Measuring the per-block angular change of hidden states within the pre-norm Transformer, TI embeddings exhibit an average per-block angular change of 21.33°, while DTI embeddings achieve 33.52° — a 1.57× increase. This confirms that large norms suppress meaningful residual updates and that controlling magnitude restores the encoder's ability to transform token representations.

How the Paper Positions Itself

The paper frames DTI not as yet another enrichment or heuristic but as a principled geometric reformulation of TI that directly addresses the identified failure mechanism. The positioning is threefold:

1. Addressing the root cause, not the symptom. Where CrossInit provides better initialization and P+/NeTI expand representation capacity, DTI constrains the optimization to respect the geometry of the pre-trained embedding space — specifically, that semantic information is directional and that magnitudes must remain in-distribution for the pre-norm Transformer to function correctly.

2. Preserving TI's efficiency advantage. DTI optimizes exactly one embedding vector (same parameter count as TI), uses Riemannian SGD with computationally cheap tangent-space projection and retraction, and adds a constant-direction prior gradient that requires no new graph operations. Training time remains approximately 7 minutes on a single A6000 GPU.

3. Enabling new capabilities through geometric awareness. Because DTI parameterizes learned tokens on the unit hypersphere, spherical linear interpolation (SLERP) between learned concepts produces semantically coherent intermediate representations. Standard TI's Euclidean embeddings don't support this — linear interpolation passes through regions of latent space with unnatural magnitudes and produces incoherent outputs (Figure 15 in Appendix E). This capability (demonstrated in Figure 4) unlocks creative applications like morphing between a dog and a teapot, or smoothly interpolating facial features between individuals.

Connection to Broader Research Context

The paper connects to two broader lines of work that situate its contribution:

• Directional embedding spaces (Section 5.2): Prior work has explored hyperspherical constraints in contrastive learning (Wang et al., 2020), face recognition (Meng et al., 2019), and document clustering (Zhang et al., 2020). DTI is the first to bring this geometric perspective to token embedding optimization in text-to-image personalization, decomposing embeddings explicitly into magnitude and direction with a vMF prior.

• Personalization as embedding optimization (Section 5.1): TI (Gal et al., 2023) established the embedding optimization paradigm, and subsequent work has built on it. DTI is positioned as a drop-in replacement for TI's optimization procedure that can be integrated into any pipeline using TI, as demonstrated with the DCO + DTI experiment in Appendix D.3 (Table 8), where replacing TI with DTI in a LoRA-based fine-tuning pipeline improves text alignment from 0.456 to 0.635.

3. Technical Approach

3.1 Reader Orientation

Directional Textual Inversion (DTI) is a drop-in replacement for the optimization procedure in Textual Inversion that constrains learned token embeddings to live on the unit hypersphere with a fixed, in-distribution magnitude, then uses Riemannian gradient descent with a directional prior to learn only the semantically meaningful direction. The system solves the problem identified in Section 2 — standard TI produces embeddings with wildly inflated norms that attenuate positional information and stagnate residual updates in pre-norm Transformers — by decoupling the embedding into a frozen magnitude (set to the average norm of the pre-trained vocabulary, approximately 0.4) and a learnable direction vector, formulating direction learning as Maximum a Posteriori estimation with a von Mises-Fisher prior that pulls the embedding toward a semantically meaningful anchor.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that transform a few user-provided reference images into a personalized token embedding:

1. Frozen Text Encoder (e.g., CLIP or Gemma) — the pre-trained language model that maps text tokens to embedding vectors. It is never updated; it provides the vocabulary statistics (mean token norm) and the semantic anchor direction for the prior.

2. Learnable Direction Vector $\mathbf{v} \in \mathbb{S}^{d-1}$ — the only parameter being optimized, representing the personalized concept's semantic identity on the unit hypersphere. Combined with the fixed magnitude $m^*$, this forms the full embedding $\mathbf{e} = m^*\mathbf{v}$ that gets inserted into the text encoder's input sequence wherever the placeholder token (e.g., <cat>) appears.

3. Riemannian SGD Optimizer — replaces standard AdamW to respect the spherical geometry. At each step, it computes the Euclidean gradient of the diffusion loss plus prior, projects this gradient onto the tangent space of the sphere, normalizes the gradient magnitude, and retracts the updated vector back onto the sphere via normalization.

4. von Mises-Fisher Prior — a constant-direction gradient $-\kappa\bm{\mu}$ that pulls the learned direction toward the normalized embedding of a related class token (e.g., "cat" for <cat>), preventing semantic drift during optimization.

Information flows as follows: user provides reference images $\mathcal{D} = \{\mathbf{z}_1, \dots, \mathbf{z}_n\}$ and a placeholder token string → the system initializes $\mathbf{v}_0$ as the normalized embedding of a semantically related token (e.g., the DreamBooth dataset's provided initializer) and fixes $m^*$ to the average vocabulary norm → at each training step, a minibatch of noisy images is sampled, the current embedding $m^*\mathbf{v}_k$ is inserted into text prompts and fed through the frozen text encoder into the diffusion model → the reconstruction loss gradient $\mathbf{g}_{\text{data}}$ is computed with respect to $\mathbf{v}_k$ → the constant prior gradient $-\kappa\bm{\mu}$ is added to produce $\mathbf{g}_{\text{euc}}$ → this Euclidean gradient is projected onto the tangent plane of the sphere at $\mathbf{v}_k$, yielding $\mathbf{g}$ → the gradient is normalized to fixed length and used to update $\mathbf{v}_{k+1}$ via retraction (re-normalization to unit length) → after $K$ iterations (typically 500), the final embedding $\mathbf{e}^* = m^*\mathbf{v}_K$ is returned and can be used in any text prompt.

3.3 Roadmap for the Deep Dive

• First, the embedding decoupling formulation (Equation 1) and magnitude selection strategy — because this defines what is being optimized and what is fixed, establishing the geometric constraint that all subsequent steps respect.

• Second, the Riemannian SGD update rule (Equation 2) — the core optimization mechanics of tangent projection, gradient scaling, and retraction — since this is the engine that replaces AdamW and enforces the sphere constraint.

• Third, the MAP formulation with vMF prior (Equations 3–5) — the theoretical framework that justifies why we add a constant-direction gradient and how the prior parameters $\bm{\mu}$ and $\kappa$ are chosen.

• Fourth, the vMF prior gradient computation and its practical integration into the RSGD step — because the "constant-direction prior gradient" is the paper's key algorithmic insight for preventing semantic drift.

• Fifth, the full DTI algorithm (Algorithm 1) synthesizing all components into a complete optimization procedure, with attention to why each design choice was made over alternatives.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a geometric reformulation paper whose core idea is that semantic information in token embeddings is encoded by direction, not magnitude, and that constraining optimization to the unit hypersphere with Riemannian methods eliminates the norm inflation that causes TI's prompt fidelity failures.

---

Embedding Decoupling and Magnitude Selection

The paper begins with a simple but consequential decomposition of the learnable token embedding $\mathbf{e} \in \mathbb{R}^d$ into two independent components:

$\mathbf{e} = m^\star \mathbf{v}, \quad \mathbf{v} \in \mathbb{S}^{d-1}$

where $m^\star > 0$ is a fixed scalar magnitude and $\mathbf{v}$ is a unit-norm direction vector lying on the $(d-1)$-dimensional unit hypersphere $\mathbb{S}^{d-1} = \{\mathbf{u} \in \mathbb{R}^d : \|\mathbf{u}\|_2 = 1\}$.

What it computes: this decomposition separates the embedding into a scale component $m^\star$ (which is frozen before optimization begins and never updated) and a directional component $\mathbf{v}$ (which is the sole object of optimization). The full embedding fed to the text encoder is simply the product.

Why this form: the empirical analysis in Section 2 established that (a) semantic information is encoded in direction (cosine similarity retrieves semantically coherent neighbors, while Euclidean distance does not — Table 1), and (b) standard TI's norm inflation (Figure 1a, norms exceeding 20 versus vocabulary average ~0.4) is the root cause of prompt fidelity degradation through positional attenuation and residual stagnation. Decoupling magnitude from direction lets us fix the former to a known-safe value and focus learning entirely on the semantically meaningful component. An alternative — optimizing both magnitude and direction jointly — is what standard TI does and demonstrably fails at. Post-hoc rescaling (TI-rescaled in Table 2) partially recovers text fidelity but degrades subject similarity (0.243 image similarity vs. 0.561 for standard TI), confirming that magnitude must be constrained during optimization, not corrected afterward.

Magnitude selection. The paper sets $m^\star$ to "an in-distribution magnitude derived from the frozen vocabulary of the text encoder (e.g., the average norm)" (Section 3.1). Concretely, the authors compute the $\ell_2$ norm of every token embedding in the pre-trained CLIP (or Gemma, for SANA) vocabulary, take the mean, and use this as $m^\star$. This value is approximately 0.4 for CLIP-based encoders. The ablation study (Table 3, rows 3–5) compares three choices under RSGD with $\kappa = 1 \times 10^{-4}$:

• Minimum vocabulary norm ("min"): produces poor results on both image similarity (0.030) and text fidelity (0.074) because the embedding is too small to effectively condition the model.

• Out-of-distribution magnitude (5.0): degrades both metrics (0.383 image, 0.373 text) relative to the mean, confirming that even when direction is optimized correctly on the sphere, an OOD magnitude still harms performance — consistent with the theory that large magnitudes cause pre-norm failures regardless of directional quality.

• Mean vocabulary norm ("mean"): achieves the best balance (0.450 image, 0.522 text).

The design choice reflects the theoretical analysis: the magnitude must be large enough to provide sufficient conditioning signal to the diffusion model yet small enough to remain within the regime where LayerNorm/RMSNorm preserves positional information and residual updates remain effective. The paper notes (Appendix E, Figure 16) that small deviations from the mean (e.g., using the magnitude of the specific initialization token rather than the vocabulary-wide average) produce negligible differences in output quality, suggesting the method is robust to the exact choice as long as it stays within the in-distribution range.

---

Riemannian Stochastic Gradient Descent on the Hypersphere

Since the optimization domain is the unit sphere $\mathbb{S}^{d-1}$ rather than unconstrained Euclidean space, standard optimizers like AdamW are geometrically inappropriate. AdamW computes updates in $\mathbb{R}^d$ without awareness of the spherical constraint; naively projecting the updated vector back onto the sphere after each step (as tested in the first ablation row of Table 3) is not equivalent to true Riemannian optimization because the projection is applied after the Euclidean update, meaning the update step itself ignores the manifold's curvature. The paper instead adopts Riemannian Stochastic Gradient Descent (RSGD) (Bonnabel, 2013), which operates directly on the sphere through two operations: tangent-space projection and retraction.

Step 1: Tangent-space projection. At iteration $k$, given the current direction $\mathbf{v}_k \in \mathbb{S}^{d-1}$ and a Euclidean-space gradient $\mathbf{g}_{\text{euc}} \in \mathbb{R}^d$ (which includes both the data loss gradient and the prior gradient, computed as if $\mathbf{v}_k$ were an unconstrained Euclidean parameter), the Riemannian gradient $\mathbf{g}$ is obtained by projecting $\mathbf{g}_{\text{euc}}$ onto the tangent space of the sphere at $\mathbf{v}_k$:

$\mathbf{g} = \mathbf{g}_{\text{euc}} - (\mathbf{v}_k^{\mathsf{T}} \mathbf{g}_{\text{euc}}) \mathbf{v}_k \in T_{\mathbf{v}_k} \mathbb{S}^{d-1}$

where $T_{\mathbf{v}_k} \mathbb{S}^{d-1}$ denotes the tangent space — the $(d-1)$-dimensional hyperplane of all vectors orthogonal to $\mathbf{v}_k$ at the point $\mathbf{v}_k$ on the sphere.

What it computes: the operation subtracts from $\mathbf{g}_{\text{euc}}$ its component parallel to $\mathbf{v}_k$ (the radial direction), leaving only the component that is perpendicular to $\mathbf{v}_k$ and therefore lies in the tangent plane. Geometrically, this projects the gradient onto the flat local approximation of the sphere at the current point, discarding any component that would push the update off the sphere's surface. The term $\mathbf{v}_k^{\mathsf{T}} \mathbf{g}_{\text{euc}}$ is the scalar projection of the Euclidean gradient onto the radial direction; multiplying by $\mathbf{v}_k$ reconstructs the radial vector component, and subtracting it removes that component, yielding the tangential part.

Why this form: without tangent projection, a naive Euclidean update $\mathbf{v}_k - \eta \mathbf{g}_{\text{euc}}$ would move the parameter off the sphere, and subsequent re-normalization would not undo the geometric error — the resulting trajectory would not follow the true Riemannian gradient flow. The projection ensures the update direction respects the local geometry of the constraint manifold, which is necessary for convergence guarantees and empirical stability. The ablation in Table 3 (row 1 vs. row 4) confirms this: AdamW with post-hoc re-projection achieves 0.335 image similarity and 0.463 text fidelity, while RSGD achieves 0.450 and 0.522, a substantial improvement.

Step 2: Gradient scaling. The paper observes that "gradient magnitudes tend to increase as training progresses, which often leads to instability in the later stages" (Appendix C.1). To address this, they normalize the tangent-space gradient to unit length before applying the step:

$\mathbf{g}' = \frac{\mathbf{g}}{\|\mathbf{g}\|_2}$

What it computes: this divides the Riemannian gradient by its $\ell_2$ norm, producing a direction-only vector of length 1 on the tangent plane. The effective step size becomes exactly $\eta$ (the learning rate) measured along the sphere's surface.

Why this form: this normalization is equivalent to using an adaptive step size $\eta / \|\mathbf{g}\|_2$ in the unnormalized update $\mathbf{v}_{k+1} = \mathbf{v}_k - (\eta / \|\mathbf{g}\|_2) \mathbf{g}$. The total displacement magnitude is capped at $\eta$, preventing large gradients from causing disproportionately large parameter jumps. The paper notes this is "inspired by Euclidean space optimizers (Hinton, 2012, RMSProp; Kingma & Ba, 2015, Adam; Loshchilov & Hutter, 2019, AdamW), which normalize the gradient based on a moving average of squared gradients." Unlike those methods, DTI uses the instantaneous norm rather than a running average, which is simpler and empirically sufficient for the relatively short optimization runs (500 steps). Appendix C.1 cites Cho & Lee (2017) for a similar technique in Riemannian optimization.

Step 3: Retraction. After computing the update direction $\mathbf{g}'$, the new point is obtained by moving along the tangent vector and then retracting (projecting) back onto the sphere:

$\mathbf{v}_{k+1} = \operatorname{Retr}_{\mathbf{v}_k}(-\eta \mathbf{g}') = \frac{\mathbf{v}_k - \eta \mathbf{g}'}{\|\mathbf{v}_k - \eta \mathbf{g}'\|_2}$

What it computes: the vector $\mathbf{v}_k - \eta \mathbf{g}'$ is the result of taking a step of length $\eta$ along the unit tangent direction $-\mathbf{g}'$ from $\mathbf{v}_k$. This point lies slightly off the sphere (since the tangent plane is a flat approximation to the curved surface). The retraction — dividing by the norm — projects this point radially back onto the sphere. For the unit sphere, this simple normalization is an exact retraction.

Why this form: the retraction operation ensures that after every update, the parameter $\mathbf{v}_{k+1}$ satisfies $\|\mathbf{v}_{k+1}\|_2 = 1$, maintaining the hard constraint exactly (not approximately). Alternatives like projected gradient descent would first compute a constrained optimization step and then project, but RSGD's retraction-first approach is both simpler and preserves the theoretical properties of Riemannian optimization.

Learning rate. The paper uses $\eta = 5 \times 10^{-3}$, the "commonly-used" learning rate from the standard TI implementation in HuggingFace diffusers (Appendix D.1). This is kept constant across all experiments; no learning rate schedule is employed because the gradient normalization in Step 2 already provides adaptive step sizing.

---

Maximum A Posteriori Formulation with von Mises-Fisher Prior

The paper frames the optimization of the direction $\mathbf{v}$ as Maximum a Posteriori (MAP) estimation rather than pure maximum likelihood. This provides a principled way to incorporate the prior belief that the learned token should remain directionally close to a semantically meaningful anchor (e.g., the class token "cat" for a personalized cat concept).

Given the reference image dataset $\mathcal{D} = \{\mathbf{z}_1, \dots, \mathbf{z}_n\}$, the MAP estimate is found by maximizing the posterior probability:

$\mathbf{v}^* = \arg\max_\mathbf{v} p(\mathbf{v} \mid \mathcal{D}) \propto \arg\max_\mathbf{v} \left[\log p(\mathcal{D} \mid \mathbf{v}) + \log p(\mathbf{v})\right]$

where $p(\mathbf{v} \mid \mathcal{D})$ is the posterior probability of the direction given the images, $p(\mathcal{D} \mid \mathbf{v})$ is the likelihood of the images given the direction, and $p(\mathbf{v})$ is the prior probability of the direction.

What it computes: this equation decomposes the optimization objective into two additive terms in log-space: a data term (how well the embedding explains the reference images) and a prior term (how plausible the embedding direction is according to our beliefs about the token embedding space). Minimizing the negative log-posterior is equivalent to minimizing the loss function $\mathcal{L}(\mathbf{v}) = \mathcal{L}_{\text{data}}(\mathbf{v}) + \mathcal{L}_{\text{prior}}(\mathbf{v})$, where $\mathcal{L}_{\text{data}} = -\log p(\mathcal{D} \mid \mathbf{v})$ and $\mathcal{L}_{\text{prior}} = -\log p(\mathbf{v})$.

Why this form: standard TI optimizes only the data term (the diffusion reconstruction loss), which is equivalent to maximum likelihood estimation. This provides no protection against semantic drift — the learned direction can move arbitrarily far from related concepts if doing so reduces the reconstruction loss on the few reference images (an overfitting phenomenon visible in Figure 1b, where <cat> drifts away from "cat" and "kitten"). The MAP formulation adds a regularizer that penalizes such drift, and crucially, by choosing the vMF distribution as the prior, the penalty takes a particularly simple and computationally efficient form (a constant additive gradient).

Data term: diffusion reconstruction loss. Following standard practice for diffusion models (Ho et al., 2020), the negative log-likelihood is approximated by the mean squared error between the true noise and the model's noise prediction:

$\mathcal{L}_{\text{data}}(\mathbf{v}) \coloneqq \mathbb{E}_{\mathbf{z}, t, \bm{\epsilon}, c}\left[\left\|\bm{\epsilon} - \bm{\epsilon}_\theta(\mathbf{z}_t, t, c(\mathbf{v}))\right\|^2_2\right]$

where $\mathbf{z}$ is a reference image, $t \sim \text{Uniform}(1, T)$ is a randomly sampled diffusion timestep, $\bm{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ is Gaussian noise, $\mathbf{z}_t$ is the noised version of $\mathbf{z}$ at timestep $t$, $\bm{\epsilon}_\theta$ is the frozen diffusion model, and $c(\mathbf{v})$ is the text conditioning obtained by feeding the full prompt (with $m^*\mathbf{v}$ inserted at the placeholder token's position) through the frozen text encoder.

What it computes: for each training example, noise is added to a reference image, the model is asked to predict that noise given the text prompt containing the current personalized token, and the squared error is averaged. The expectation is over the image, timestep, noise sample, and conditioning. The gradient $\nabla_{\mathbf{v}} \mathcal{L}_{\text{data}}$ flows from the diffusion model's noise prediction through the text encoder's conditioning output back to the input embedding $\mathbf{v}$.

Why this form: this is the standard denoising score matching objective used to train diffusion models and to perform Textual Inversion. The paper does not modify the data term — the innovation is entirely in the addition of the prior term and the geometric optimization procedure.

Prior term: von Mises-Fisher distribution. The prior $p(\mathbf{v})$ is modeled as a von Mises-Fisher (vMF) distribution on the unit sphere $\mathbb{S}^{d-1}$. The vMF is the natural analog of the Gaussian distribution for directional data — it is the maximum-entropy distribution on the sphere given a fixed mean direction. Its probability density function is:

$p(\mathbf{v} \mid \bm{\mu}, \kappa) = \frac{\kappa^{d/2 - 1}}{(2\pi)^{d/2} I_{d/2 - 1}(\kappa)} \exp\left(\kappa \bm{\mu}^{\mathsf{T}} \mathbf{v}\right)$

where $\bm{\mu} \in \mathbb{S}^{d-1}$ is the mean direction (the mode of the distribution, i.e., the most probable direction), $\kappa \geq 0$ is the concentration parameter (analogous to the inverse variance — larger $\kappa$ concentrates probability more tightly around $\bm{\mu}$), and $I_{d/2 - 1}$ is the modified Bessel function of the first kind (the normalization constant that ensures the density integrates to 1 over the sphere).

What it computes: the density assigns higher probability to directions $\mathbf{v}$ that are close (in cosine similarity) to $\bm{\mu}$ and lower probability to directions far from $\bm{\mu}$. The term $\kappa \bm{\mu}^{\mathsf{T}} \mathbf{v}$ in the exponent is essentially $\kappa \cos(\angle(\bm{\mu}, \mathbf{v}))$ since both vectors have unit norm.

Why this form (Appendix C.2 provides a detailed justification):

1. Natural fit: the vMF is "the natural analog to the Gaussian distribution on a hypersphere, making it a principled and standard choice for modeling directional data clusters." The paper's hypothesis is that "the token embedding vocabulary can be modeled as a mixture of vMF distributions, where each component corresponds to a distinct semantic cluster (e.g., one for animals, another for objects)."

2. Computational efficiency: in the MAP formulation, the negative log-prior and its gradient take exceptionally simple forms. Ignoring constants (the normalization term involving the Bessel function does not depend on $\mathbf{v}$):

$\mathcal{L}_{\text{prior}}(\mathbf{v}) = -\log p(\mathbf{v}) \propto -\kappa \bm{\mu}^{\mathsf{T}} \mathbf{v}$

The Euclidean gradient is simply a constant vector:

$\nabla_{\mathbf{v}} \mathcal{L}_{\text{prior}} = -\kappa \bm{\mu}$

This is a constant-vector gradient that pulls the learned direction toward $\bm{\mu}$ with strength proportional to $\kappa$. It requires no additional graph operations, no backpropagation through the prior — just a vector addition to the data gradient. The paper explicitly notes this is "analogous in spirit to decoupled weight decay (Loshchilov & Hutter, 2019), but adapted for the sphere with a directional prior."

3. Interpretability: $\bm{\mu}$ serves as a semantic anchor — it is set to the normalized embedding of a related class token (e.g., "dog" for the personalized <dog> concept). If the data gradient tries to push the learned direction toward overfitting on the reference images (at the cost of losing semantic coherence), the prior gradient provides a countervailing pull toward the semantic neighborhood of the concept. The concentration $\kappa$ directly controls how strong this regularization is.

4. Superior to alternatives for this application: the paper argues (Appendix C.2) that alternatives like the Kent or Bingham distributions would be less suitable: "This simplicity makes it more suitable for high-dimensional embeddings in large-scale models than alternatives like the Kent and Bingham distributions."

Selection of prior parameters. The vMF prior requires choosing $\bm{\mu}$ and $\kappa$:

• Mean direction $\bm{\mu}$: set to the normalized embedding of the corresponding class token from the pre-trained text encoder. For example, for the DreamBooth concept <cat>, $\bm{\mu}$ is the $\ell_2$-normalized embedding of the token "cat". This is computed once before optimization and held constant. The paper also explores using VLMs (Qwen-VL 2.5) to recommend better initialization tokens (Table 6, Appendix D.2), finding that Qwen-VL initialization can improve subject similarity at a slight cost to text fidelity.

• Concentration parameter $\kappa$: treated as a hyperparameter controlling regularization strength. The paper performed a grid search and found that "values in the range of $5 \times 10^{-5}$ to $2 \times 10^{-4}$ work well" and "simply fixed the value of $\kappa$ to $1 \times 10^{-4}$ for all experiments" (Appendix D.1). The ablation in Table 3 (rows 4–6) compares:

  • $\kappa = 0.0$ (no prior): achieves higher image similarity (0.507) but lower text fidelity (0.436) — the model overfits to the reference images.
  • $\kappa = 1 \times 10^{-4}$ (DTI default): achieves balanced performance (0.450 image, 0.522 text).
  • $\kappa = 5 \times 10^{-4}$ (strong prior): achieves the highest text fidelity (0.688) but significantly lower image similarity (0.278) — the semantic anchor dominates, preventing the token from capturing subject-specific details.

This demonstrates the trade-off controlled by $\kappa$: stronger priors improve prompt adherence by keeping the token semantically grounded, but at the cost of reducing the token's ability to capture fine-grained subject identity that distinguishes the specific dog from the generic concept "dog."

---

Integration: The Prior Gradient in the RSGD Update

The Euclidean-space gradient $\mathbf{g}_{\text{euc}}$ used in the RSGD tangent projection (Step 1 of Section 3.4.2) is the sum of the data gradient and the prior gradient:

$\mathbf{g}_{\text{euc}} = \nabla_{\mathbf{v}} \mathcal{L}_{\text{data}}(m^* \mathbf{v}_k) - \kappa \bm{\mu}$

What it computes: the first term $\nabla_{\mathbf{v}} \mathcal{L}_{\text{data}}$ is the gradient of the diffusion reconstruction loss with respect to the direction vector, computed by standard backpropagation through the text encoder and diffusion model. The second term $-\kappa \bm{\mu}$ is the constant prior gradient — literally a vector subtraction of $\kappa \bm{\mu}$ from the data gradient before any geometric operations. Since $\mathcal{L}_{\text{data}}$ takes $m^*\mathbf{v}_k$ as input (the full embedding, not just the direction), the chain rule gives $\nabla_{\mathbf{v}} \mathcal{L}_{\text{data}}(m^* \mathbf{v}) = m^* \cdot \nabla_{\mathbf{e}} \mathcal{L}_{\text{data}}(\mathbf{e})|_{\mathbf{e}=m^*\mathbf{v}}$, scaling the embedding-space gradient by $m^*$.

Why this form: this two-term sum means the optimizer receives a combined signal: the data gradient tells it how to adjust the direction to better reconstruct the reference images, while the prior gradient constantly pulls it toward the semantic anchor. The relative strength of these two signals is controlled by $\kappa$. Because the prior gradient is a constant vector (computed once at initialization), it adds essentially zero computational overhead — it is simply a vector addition in each iteration.

After computing $\mathbf{g}_{\text{euc}}$, the full RSGD pipeline proceeds as described in Section 3.4.2: tangent projection to get $\mathbf{g}$, normalization to get $\mathbf{g}'$, retraction to get $\mathbf{v}_{k+1}$.

---

The Full DTI Algorithm and Design Justifications

Algorithm 1 (reproduced in the paper) synthesizes all components into a complete procedure:

Inputs: frozen diffusion model $\bm{\epsilon}_\theta$, frozen text encoder $c(\cdot)$, initialization token embedding $\mathbf{e}_{\text{init}}$, fixed magnitude $m^*$, concentration $\kappa$, number of iterations $K$, learning rate $\eta$.

Step 1 (initialization): Set $\mathbf{v}_0 \leftarrow \mathbf{e}_{\text{init}} / \|\mathbf{e}_{\text{init}}\|_2$ (normalize the initialization token to unit length). Set $\bm{\mu} \leftarrow \mathbf{e}_{\text{init}} / \|\mathbf{e}_{\text{init}}\|_2$ (use the same normalized token as the prior mean direction). Both $\mathbf{v}_0$ and $\bm{\mu}$ are thus unit vectors pointing in the same initial direction; optimization moves $\mathbf{v}$ away from $\bm{\mu}$ only as far as the data gradient overcomes the prior pull.

Steps 2–7 (training loop): For $k = 0$ to $K-1$:

1. Sample a minibatch $(\mathbf{z}, t, \bm{\epsilon})$ of reference images, timesteps, and noise.
2. Compute the data gradient $\mathbf{g}_{\text{data}} \leftarrow \nabla_{\mathbf{v}} \mathcal{L}_{\text{data}}(m^*\mathbf{v}_k)$ by backpropagation through the diffusion model and text encoder.
3. Add the prior gradient: $\mathbf{g}_{\text{euc}} \leftarrow \mathbf{g}_{\text{data}} - \kappa \bm{\mu}$.
4. Project onto the tangent space: $\mathbf{g} \leftarrow \mathbf{g}_{\text{euc}} - (\mathbf{g}_{\text{euc}}^{\mathsf{T}} \mathbf{v}_k) \mathbf{v}_k$.
5. Scale the gradient: $\mathbf{g}' \leftarrow \mathbf{g} / \|\mathbf{g}\|_2$.
6. Retract onto the sphere: $\mathbf{v}_{k+1} \leftarrow (\mathbf{v}_k - \eta \mathbf{g}') / \|\mathbf{v}_k - \eta \mathbf{g}'\|_2$.

Output: Final embedding $\mathbf{e}^* = m^* \mathbf{v}_K$, a vector in $\mathbb{R}^d$ with norm exactly $m^*$ and direction $\mathbf{v}_K$ optimized to balance reconstruction fidelity and semantic coherence.

Computational cost. The method introduces minimal overhead relative to standard TI:

• Tangent projection adds two dot products and two vector operations per iteration (negligible relative to diffusion model forward/backward passes).
• Prior gradient addition is a single vector subtraction per iteration.
• Gradient scaling is one norm computation and one division per iteration.
• Total training time remains approximately 7 minutes on a single NVIDIA A6000 GPU for SDXL (500 steps, batch size 4, bf16 mixed precision), matching standard TI's cost.

Hyperparameter summary (Appendix D.1):

• Learning rate $\eta = 5 \times 10^{-3}$ (same as standard TI in HuggingFace diffusers).
• Magnitude $m^* = \text{mean}(\{\|\mathbf{e}\|_2 : \mathbf{e} \in \text{vocabulary}\})$ (computed once per model).
• Concentration $\kappa = 1 \times 10^{-4}$ (fixed across all experiments).
• Training steps $K = 500$.
• Batch size 4, bf16 mixed precision, random seed 42.
• Number of reference images per concept: those provided in the DreamBooth dataset (typically 4–6 images per subject).

Design justification for RSGD over AdamW. The first row of Table 3 tests AdamW on the sphere by performing standard Euclidean updates and then projecting back onto the sphere after each step. This achieves 0.335 image similarity and 0.463 text fidelity — substantially worse than RSGD's 0.450/0.522. The reason is geometric: AdamW's momentum terms maintain running averages of past gradients and squared gradients in the ambient Euclidean space. When the parameter is constrained to the sphere, these running averages mix information from different tangent spaces (since the tangent plane rotates as the parameter moves), producing update directions that do not correspond to true Riemannian gradient information. RSGD's tangent-space projection ensures that at each step, only the geometrically valid components of the gradient are used.

Design justification for normalization of $\mathbf{g}$. The gradient scaling step $\mathbf{g}' = \mathbf{g} / \|\mathbf{g}\|_2$ was introduced because "gradient magnitudes tend to increase as training progresses, which often leads to instability in the later stages" (Appendix C.1). Without this normalization, the effective step size $\eta \|\mathbf{g}\|_2$ varies with gradient magnitude; when gradients become large late in training, the parameter can move large distances on the sphere in a single step, potentially overshooting good solutions. The normalization caps the per-step geodesic distance at $\eta$, providing stable convergence without requiring a learning rate schedule. The paper notes this is "equivalent to using an adaptive step size $\eta / \|\mathbf{g}\|_2$ in a Euclidean update."

Design justification for using the same token as initialization and prior. The paper sets both $\mathbf{v}_0$ and $\bm{\mu}$ to the normalized embedding of the initialization token (e.g., "cat" for <cat>, as provided by the DreamBooth dataset). This means the prior pulls the learned direction toward its starting point. The intuition is that the starting point (a semantically related class token) is a reasonable anchor — the personalization task is to learn what makes this specific cat different from the generic concept "cat," while remaining close enough to the semantic neighborhood of "cat" that the model can compose the concept with novel prompts. If $\bm{\mu}$ pointed elsewhere (e.g., toward an unrelated concept), the prior would fight against valid personalization rather than regularizing it.

Design justification for $m^*$ being frozen rather than learned. Why not learn both magnitude and direction on the sphere? The paper's theoretical analysis (Section 2.2, Lemmas 1–2, Proposition 1, Corollary 1) provides the answer: any mechanism that allows the magnitude to grow will eventually cause the positional attenuation and residual stagnation failures documented in Section 2. If magnitude were a learnable parameter, the optimizer would have an incentive to increase it — larger magnitudes amplify the token's influence on the generated image (which can reduce reconstruction loss on reference images in the short term) at the cost of breaking prompt composition. The post-hoc rescaling experiment (TI-rescaled, Table 2) demonstrates this tension: TI learns large-norm embeddings that capture the subject well but fail on prompts; rescaling to in-distribution norm recovers prompt fidelity but destroys subject similarity because the direction was learned under pathological norm conditions. DTI preempts this entirely by fixing the magnitude from the start, forcing the direction to be learned under conditions where the text encoder functions correctly (preserving positional information and enabling rich residual transformations).

---

Interpolation Capability as a Byproduct of Hyperspherical Parameterization

A notable capability that emerges automatically from DTI's geometric formulation is smooth interpolation between learned concepts via spherical linear interpolation (SLERP) . Given two learned directions $\mathbf{v}_A$ and $\mathbf{v}_B$ on the unit sphere, SLERP computes intermediate directions as:

$\operatorname{SLERP}(\mathbf{v}_A, \mathbf{v}_B; \alpha) = \frac{\sin((1-\alpha)\Omega)}{\sin(\Omega)} \mathbf{v}_A + \frac{\sin(\alpha \Omega)}{\sin(\Omega)} \mathbf{v}_B$

where $\Omega = \arccos(\mathbf{v}_A^{\mathsf{T}} \mathbf{v}_B)$ is the angle between the two directions, and $\alpha \in [0, 1]$ is the interpolation parameter ($\alpha=0$ yields $\mathbf{v}_A$, $\alpha=1$ yields $\mathbf{v}_B$).

What it computes: SLERP travels along the geodesic (great circle arc) on the sphere between $\mathbf{v}_A$ and $\mathbf{v}_B$ at constant angular speed. Each intermediate point is exactly on the unit sphere (by construction), maintaining the valid embedding geometry throughout the interpolation.

Why this works for DTI but not TI: standard TI embeddings have arbitrary, typically large magnitudes. Linear interpolation (LERP) between two TI embeddings, $\mathbf{e}(\alpha) = (1-\alpha)\mathbf{e}_A + \alpha \mathbf{e}_B$, produces intermediate vectors whose norms vary non-monotonically and can pass through regions far from the training distribution — often with very small norms near the midpoint (the "dip" in the convex combination of two high-norm vectors that point in different directions). These out-of-distribution intermediate embeddings produce incoherent generations (Appendix E, Figure 15). Even if one normalizes after interpolation (which could be called "SLERP for TI"), the original TI directions were learned under inflated-norm conditions where the encoder was not functioning properly, so the resulting directions don't correspond to semantically meaningful axes. DTI's directions, having been optimized entirely on the sphere with the encoder functioning correctly, carry clean semantic information that composes coherently under SLERP. The results in Figure 4 demonstrate this: dog-to-teapot, dog-to-cat, and face-to-face interpolations all produce visually coherent intermediate outputs with meaningful blended features.

This capability is presented as an application enabled by the geometric awareness of DTI (Section 4.5), not as a core design objective — it emerges as a side effect of doing the optimization correctly on the sphere.

4. Key Insights and Innovations

Innovation 1: Diagnosing Embedding Norm Inflation as the Root Cause of TI's Prompt Fidelity Failure — A Geometric Pathology, Not a Capacity Problem

Prior work on improving Textual Inversion treated its failure to respect complex prompts as either a capacity limitation (P+ and NeTI expanded the embedding representation because they assumed a single vector was insufficient) or an optimization initialization problem (CrossInit provided better starting points). These approaches implicitly assumed that the optimization target — a token embedding in Euclidean space — was fundamentally sound, just insufficiently parameterized or poorly initialized. This paper makes a fundamentally different diagnostic move: the failure is not about capacity or initialization, but about the geometry of the optimization trajectory itself violating the operational assumptions of the pre-norm Transformer architecture.

The intellectual contribution is the identification of a specific failure mechanism — embedding norm inflation — that had gone entirely uncharacterized despite Textual Inversion being a widely-used method for over two years. The paper shows that standard TI produces token embeddings with norms exceeding 20, compared to approximately 0.4 for the pre-trained vocabulary (Figure 1a), and that this 50× inflation is not a benign artifact but the direct mechanical cause of degraded prompt conditioning. Two theoretical mechanisms are proved: positional attenuation (Lemma 1 in Appendix B), where positional information contributes only O(1/m) to LayerNorm outputs when the token magnitude m is large, and residual stagnation (Lemma 2, Proposition 1, Corollary 1), where bounded sub-layer outputs cannot meaningfully rotate large-norm hidden states, effectively freezing them directionally through the encoder. The crucial insight is that these mechanisms explain why the failure pattern takes its specific form — personalized tokens dominating the generation while contextual details vanish — because the token literally cannot be contextualized by surrounding words when its magnitude dwarfs positional and contextual signals.

This is a fundamental diagnostic contribution, not an incremental improvement. It reframes TI's failure from "embeddings aren't expressive enough" (the capacity narrative) to "embeddings violate the architectural contract of pre-norm Transformers" (the geometric constraint narrative). The empirical validation in Section 2.3 is elegant: a positional classifier achieving 100% accuracy on normal inputs drops to near-zero on TI embeddings but remains fully accurate on DTI embeddings (Figure 2), and per-block angular change of hidden states is 1.57× larger for DTI than TI — direct, quantitative confirmation that the theorized mechanisms are active in practice. This diagnostic insight has implications beyond personalization: any method that learns or modifies token embeddings (soft prompt tuning, prefix tuning, concept learning in vision-language models) faces the same risk if it allows norms to drift out-of-distribution, and the paper's analytical framework provides a template for diagnosing such failures.

Innovation 2: Reformulating Token Embedding Learning as Constrained Optimization on the Hypersphere — A Principled Alternative to Post-Hoc Regularization

The field's default approach to regularizing learned embeddings is to add penalty terms to the loss function in Euclidean space and use standard optimizers. CrossInit (Pang et al., 2024) exemplifies this: it adds a regularization loss that penalizes deviation from the initialization, but the optimization still occurs in unconstrained Euclidean space, and the regularizer fights the data term in a way that depends on the loss landscape geometry. The paper's key conceptual move is to recognize that the constraint needed is not a soft penalty but a hard geometric constraint: the optimization domain should be the unit hypersphere, not R^d, because semantic information is encoded by direction (empirically demonstrated through nearest-neighbor analysis in Table 1 and Appendix A, Table 5). This transforms the problem from "optimize a vector with a regularizer" to "optimize a direction on a manifold with a prior."

This reformulation carries several intellectual implications. First, it connects token embedding learning to the established field of directional statistics and Riemannian optimization, providing theoretical tools (vMF distributions, tangent-space projection, retraction) that replace ad-hoc regularization with principled geometric operations. Second, it reveals that the standard optimizer choice (AdamW) is fundamentally mismatched to the problem geometry — AdamW's momentum terms accumulate gradient information from different tangent spaces as the parameter moves on the sphere, producing updates that don't correspond to true Riemannian gradients. The ablation in Table 3 (row 1: AdamW with post-hoc projection achieving 0.335 image / 0.463 text, versus row 4: RSGD achieving 0.450 / 0.522) empirically validates that respecting the manifold geometry during optimization, not just at the constraint boundary, matters substantially. Third, the choice of a vMF prior over alternatives (discussed in Appendix C.2) is justified not just by performance but by three converging criteria: statistical appropriateness (the natural Gaussian analog on the sphere), computational efficiency (the prior gradient is a constant vector, requiring no backpropagation — this is "analogous in spirit to decoupled weight decay" but adapted for the sphere), and interpretability (the mean direction μ serves as a semantic anchor with direct intuitive meaning).

This is a fundamental reformulation of the optimization problem, not a new regularizer added to an existing optimization procedure. It changes what is being optimized (direction, not full vector), where it is being optimized (on the sphere, not in R^d), and how it is being optimized (RSGD, not AdamW). The significance extends beyond the performance gains: it provides a template for how to think about constrained optimization of learned embeddings in any deep learning architecture that uses LayerNorm/RMSNorm, which is essentially all modern Transformers.

Innovation 3: Proving That Test-Time Interpolation Between Personalized Concepts Is a Geometric Property, Not a Learned Capability

Text-to-image personalization methods have generally treated concept interpolation as a separate challenge requiring specialized architectures or training procedures. Standard TI's learned embeddings do not support meaningful interpolation — linear interpolation (LERP) between two TI embeddings produces intermediate vectors with pathological norms (Appendix E, Figure 15 demonstrates the incoherent outputs), and even normalizing after interpolation fails because the directions themselves were learned under inflated-norm conditions where the encoder wasn't functioning properly in its contextualization role.

DTI's smooth interpolation capability (demonstrated in Figure 4 for dog↔teapot, dog↔cat, and face↔face) is intellectually significant precisely because it was not designed for — it emerges automatically from the geometric correctness of the optimization. The paper never states "we designed DTI to enable interpolation." Instead, the capability is a byproduct of two properties: (1) learned directions live on the unit sphere, so spherical linear interpolation (SLERP) stays on the manifold at every intermediate point, and (2) directions were optimized under in-distribution magnitude conditions, so the encoder processes them with normal contextualization (positional information preserved, residual updates effective), meaning the learned directions correspond to semantically meaningful axes in the representation space.

This is an emergent capability finding with conceptual implications beyond this paper. It suggests that many capabilities of learned representations that the field treats as requiring explicit design (disentanglement, interpolatability, compositionality) may instead be consequences of optimizing within the correct geometric constraints. The failure of interpolation in standard TI is not because single-token embeddings inherently lack the capacity to support interpolation — DTI uses exactly one token embedding, same as TI — but because the optimization geometry permitted norm inflation that corrupted the semantic structure of the learned directions. This reframes interpolation from a "capacity problem" to a "geometric correctness problem," with implications for representation learning across domains (contrastive learning, metric learning, soft prompt tuning) where hyperspherical constraints are already used but explicit geometric optimization methods may not be.

Innovation 4: MAP Estimation as a Unifying Framework for Directional Regularization — The Prior Gradient as a Computational Primitive

The paper's formulation of direction learning as Maximum a Posteriori (MAP) estimation with a von Mises-Fisher prior (Equations 3–5) is more than a mathematical convenience — it provides a unifying conceptual framework for understanding what regularization should accomplish in embedding optimization. Prior methods regularized embeddings through penalties on Euclidean distance from initialization (CrossInit) or through implicit regularization from limited optimization steps. These approaches conflate two distinct geometric properties: magnitude and direction. The MAP formulation cleanly separates them: the magnitude constraint is a deterministic precondition (frozen to in-distribution scale), while the directional regularization is a probabilistic statement about where learned tokens should live on the sphere relative to semantic anchors.

The specific computational form of the prior gradient — a constant vector −κμ added to the data gradient before tangent projection — is a principled computational primitive that is simultaneously: (1) theoretically grounded as the natural gradient of the vMF log-prior, (2) computationally trivial (one vector subtraction per iteration), (3) interpretable as a constant pull toward a semantic anchor with controllable strength κ, and (4) complementary to the data gradient rather than adversarial (the two terms add before geometric projection, allowing the optimizer to find a direction that balances both signals). The ablation in Table 3 (rows 4–6) demonstrates the full spectrum of behavior controlled by κ: κ = 0 (no prior) yields better subject similarity at the cost of text fidelity (overfitting), κ = 1 × 10⁻⁴ yields balanced performance, and κ = 5 × 10⁻⁴ yields the highest text fidelity but substantially lower subject similarity (the anchor dominates, preventing the token from capturing concept-specific details).

This frames the practitioner's choice of κ not as an opaque hyperparameter to tune, but as a directly interpretable dial: "how far from the generic class concept should this personalized token be allowed to drift?" The paper's decision to fix κ = 1 × 10⁻⁴ across all experiments reflects a design philosophy of providing a sensible default that works across concepts, but the framework makes it clear how to adjust: if a particular application requires extreme prompt fidelity (e.g., generating images where the subject must interact with complex scene descriptions), increase κ; if the personalized concept is highly distinctive and prompt fidelity is secondary, decrease or zero κ. This interpretability stands in contrast to the opaque regularization weights in methods like CrossInit, where the relationship between the regularization strength and the geometric behavior of the embedding is mediated by the loss landscape.

This is a conceptual advance in how regularization is formulated for embedding optimization — bridging the gap between the statistical principle (MAP), the geometric constraint (the sphere), and the computational implementation (a constant additive gradient) in a way that makes each component's role transparent and independently adjustable.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper primarily employs the DreamBooth dataset (Ruiz et al., 2023), using all reference images for subject personalization. Additional experiments on stylization use StyleDrop (Sohn et al., 2023), and face personalization experiments draw from FFHQ (Karras et al., 2019) and faces generated by DALL·E (Ramesh et al., 2021). For evaluation, all methods are tested on 40 prompts: the complete set of prompts from the DreamBooth dataset supplemented with 10 additional complex prompts specifically designed to stress-test compositional generation (Section 4.1).

• Base model(s). The primary model is Stable Diffusion XL (SDXL) (Podell et al., 2024), chosen because of its "superior performance and widespread adoption in concurrent research" (Section 4.1). To demonstrate applicability to more recent architectures, experiments are also conducted on SANA 1.5 at two scales: 1.6B and 4.8B parameters (Xie et al., 2024). SANA uses Gemma (Team, 2024) as its text encoder and a Diffusion Transformer (DiT) (Peebles & Xie, 2023) as the image generator, testing DTI's generalization beyond CLIP-based encoders. Additional experiments on SD1.5 and SD2.1-base are provided in Appendix D.3 for comparison with baselines that operate on those models.

• Metrics. Two primary dimensions are assessed following established protocols (Ruiz et al., 2023; Kumari et al., 2023; Gal et al., 2023). Subject fidelity is quantified using DINOv2 (Oquab et al., 2023) feature cosine similarity between generated images and reference images — this captures whether the personalized concept is recognizably preserved. Image-text alignment is measured using SigLIP (Zhai et al., 2023), a more recent variant of CLIP chosen "following recent work" (Lee et al., 2024) — this captures whether the generated image faithfully reflects the full text prompt. For each concept, 160 images are generated (40 text prompts × 4 random seeds), and both metrics are averaged across all samples.

• Baselines. The primary baselines are Textual Inversion (TI) (Gal et al., 2023), which directly optimizes a single token embedding in Euclidean space using AdamW, and CrossInit (Pang et al., 2024), which enhances TI through specialized initialization (replacing the initialization token with the output of the text encoder) and a regularization loss. For TI, the paper also reports a TI-rescaled variant where the learned embedding is post-hoc rescaled to the in-distribution norm (matching the magnitude used in DTI) to isolate the effect of norm inflation from the effect of directional optimization (Table 2). Additional baselines evaluated in Appendix D.3 include P+ (Voynov et al., 2023) which uses multiple layer-specific embeddings, NeTI (Alaluf et al., 2023) which introduces a neural mapping network for richer conditioning, CoRe (Wu et al., 2025) which proposes contextual regularization, and DCO (Lee et al., 2024) which jointly optimizes token embeddings with LoRA-based fine-tuning.

• Generation budget / compute accounting. All embedding optimization methods are compared at identical training budgets: 500 optimization steps, batch size 4, bf16 mixed precision, on a single NVIDIA A6000 GPU (Appendix D.1). Training time is approximately 7 minutes per subject for SDXL and 30 minutes for SANA 1.5-1.6B. Inference for evaluation uses DDIMScheduler with 50 steps for SDXL and FlowMatchEulerDiscreteScheduler with 20 steps for SANA. The comparison is fair across methods because DTI introduces no additional forward or backward passes — the RSGD operations (tangent projection, gradient scaling, retraction) and the prior gradient addition are all O(d) vector operations that are negligible relative to diffusion model forward/backward passes.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for hyperparameter selection; instead, the concentration parameter κ is fixed to 1 × 10⁻⁴ across all experiments based on a grid search over the range 5 × 10⁻⁵ to 2 × 10⁻⁴ (Appendix D.1). All experiments use a fixed random seed of 42. Results are further validated through a user study with 100 participants recruited via Amazon Mechanical Turk, evaluating 20 questions per participant (10 on subject fidelity, 10 on text alignment) with shuffled option ordering and 4 responses excluded for invalid patterns, yielding 96 valid responses (Section 4.4, Appendix D.5).

Main Quantitative Results

Aggregate Performance Comparison Across Models

Table 2 presents the core quantitative comparison of DTI against baselines on SDXL and SANA 1.5 at two scales. The headline results on SDXL:

• TI achieves 0.561 image similarity and 0.292 text fidelity — strong subject preservation but severely degraded prompt adherence, consistent with the norm inflation analysis in Section 2.
• TI-rescaled (post-hoc norm correction) achieves 0.243 image similarity and 0.466 text fidelity — text alignment improves substantially (0.292 → 0.466), confirming that inflated norms causally degrade prompt fidelity, but subject similarity collapses (0.561 → 0.243), demonstrating that post-hoc rescaling cannot recover subject identity because the direction was learned under pathological encoder conditions.
• CrossInit achieves 0.545 image similarity and 0.464 text fidelity — strong text alignment (nearly matching TI-rescaled) while partially preserving subject similarity, but still substantially below TI's subject fidelity.
• DTI achieves 0.450 image similarity and 0.522 text fidelity — the highest text alignment of any method (a 1.79× improvement over TI's 0.292 and 1.13× over CrossInit's 0.464), while maintaining subject similarity that, although lower than TI (0.450 vs. 0.561), represents a much more balanced trade-off than TI-rescaled (which sacrifices subject fidelity entirely).

The trade-off interpretation: DTI deliberately prioritizes text fidelity over maximum subject similarity — the paper explicitly acknowledges in Section 6 that "DTI primarily improves text prompt fidelity as it does not directly optimize for subject similarity." For applications where subject fidelity is paramount, the paper demonstrates that DTI can be combined with LoRA-based fine-tuning to recover subject similarity (Appendix D.3, Table 8 shows DCO + DTI achieving 0.568 image and 0.635 text, compared to DCO alone at 0.605 image and 0.456 text — DTI sacrifices some image similarity for dramatically better text alignment).

Model-Scale Dependence

The results on SANA 1.5 reveal an important pattern regarding how DTI's advantage scales with model size:

• SANA 1.5-1.6B: TI achieves 0.480 image / 0.621 text; CrossInit achieves 0.344 image / 0.614 text; DTI achieves 0.479 image / 0.744 text. Here, DTI essentially matches TI on subject similarity while substantially outperforming on text fidelity (0.744 vs. 0.621).
• SANA 1.5-4.8B: TI achieves 0.446 image / 0.646 text; CrossInit achieves 0.299 image / 0.622 text; DTI achieves 0.452 image / 0.757 text. DTI now modestly exceeds TI on subject similarity (0.452 vs. 0.446) while maintaining its large text fidelity advantage (0.757 vs. 0.646).

The paper notes that "DTI's advantage over the baselines becomes even more pronounced as the model size increases" (Section 4.2). This makes theoretical sense: larger models with more Transformer layers amplify the residual stagnation effect described in Proposition 1 — as the number of layers L increases, the accumulated directional freezing from large-norm embeddings becomes more severe, making TI's failure more acute and DTI's constraint more impactful.

Why CrossInit underperforms on SANA. The paper attributes CrossInit's inconsistent performance to "differences in their text encoders; SDXL uses a CLIP text encoder, while SANA employs the LLM-based encoder" (Section 4.2). This is a practically important finding: CrossInit's initialization and regularization strategy, designed for CLIP-based encoders, does not transfer well to Gemma — a decoder-only LLM with different normalization behavior and embedding geometry. DTI's geometric approach, which makes only the general architectural assumption of pre-norm Transformer blocks (LayerNorm or RMSNorm), transfers robustly across encoder families.

Qualitative Evidence of Compositional Fidelity

Figure 3 provides qualitative comparisons across diverse subjects and prompt complexities. Specific examples cited in the paper:

• "Pop-art style illustration of <cat>": TI entirely omits the cat (the subject is lost), while DTI renders the cat in the specified pop-art style.
• "<dog> playing on a music stage with a spotlight": Both TI and CrossInit fail to incorporate all prompt elements — either omitting the subject or details like "music stage" and "spotlight." DTI integrates both the subject and these contextual details.
• Across columns spanning "simple descriptions to complex variations in attributes, backgrounds, and styles" (Figure 3 caption), DTI consistently produces outputs that satisfy more prompt constraints simultaneously.

The paper attributes this to "DTI's stable optimization within the directional space, which facilitates improved integration of multiple prompt components" (Section 4.2). This connects directly to the theoretical analysis: by keeping magnitude in-distribution, DTI preserves the text encoder's ability to compute positional relationships and perform meaningful residual transformations, enabling the personalized token to be contextualized with surrounding prompt tokens rather than dominating them.

Additional qualitative results for SANA are provided in Appendix D.6, Figures 8–9, showing the same pattern of superior prompt adherence across different model architectures.

Human Evaluation

Table 4 reports the results of a 100-participant user study (96 valid responses) evaluating pairwise preferences. The numbers represent the percentage of times each method was selected as best:

• Image fidelity: TI 13.78%, CrossInit 42.87%, DTI 43.45%.
• Text alignment: TI 10.83%, CrossInit 22.40%, DTI 66.77%.

On text alignment — the metric DTI is specifically designed to improve — DTI is chosen as best in approximately two-thirds of judgments, dominating both baselines. On image fidelity, DTI and CrossInit are essentially tied (43.45% vs. 42.87%), with TI far behind. This human evaluation independently confirms the quantitative metric results: CrossInit achieves competitive subject similarity but DTI's text fidelity advantage is unambiguous and large.

The paper provides details of the study design (Appendix D.5, Figure 7): each participant saw 20 questions (10 on each metric), with three options shuffled per question, using a fixed random seed for consistent shuffling. Invalid responses (e.g., selecting the same answer for all questions) were excluded (4 out of 100).

Interpolation Results (Figure 4)

Figure 4 demonstrates DTI's spherical linear interpolation (SLERP) capability across three concept pairs:

• Dog ↔ Teapot: Generates a continuum of hybrid objects that progressively adopt features of each concept, maintaining visual coherence throughout.
• Dog ↔ Cat: Produces intermediate animals blending features of both species in a smooth progression.
• Young boy ↔ Older woman (faces): Generates a plausible age and appearance progression while maintaining facial coherence.

The interpolation ratios [0.0, 0.35, 0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 1.0] are chosen "for better visualization" (Figure 4 caption), with finer sampling near the midpoint where transitions are most visually interesting.

In contrast, standard TI with linear interpolation "often fails to produce coherent intermediate results" — the paper shows in Appendix E, Figure 15 that even with various post-hoc normalization strategies (SLERP with normalization, adjusted norms), TI cannot produce smooth transitions because the learned directions were corrupted by norm inflation during training. This supports the claim that interpolation is enabled not just by the sphere constraint at inference time, but by the fact that directions were learned under conditions where the encoder functioned properly.

Additional Baseline Comparisons (Appendix D.3)

Table 7 provides quantitative comparison with P+, NeTI, and CoRe on SD1.5 and SD2.1-base (the models these baselines were originally developed for):

On SD1.5, DTI achieves the highest image similarity (0.418) while maintaining competitive text fidelity (0.554). P+ achieves the highest text fidelity (0.719) but at the cost of severely degraded image similarity (0.273). On SD2.1-base, NeTI achieves the highest image similarity (0.565) but the lowest text fidelity (0.517). DTI consistently achieves "the most favorable balance between these metrics compared to all baselines" across both architectures.

The qualitative comparison in Figure 10 (Appendix D.6) shows that DTI "attains higher text fidelity while maintaining subject similarity" when prompting with complex scene descriptions.

DTI as Drop-In Replacement for TI in Fine-Tuning Pipelines (Appendix D.3)

Table 8 examines whether DTI can improve joint optimization methods that combine embedding learning with LoRA fine-tuning. The DCO method (Lee et al., 2024) typically performs joint training of a concept token using standard TI alongside a LoRA module (rank 4, 250 steps):

• DCO (TI + LoRA): 0.605 image similarity, 0.456 text fidelity.
• DCO (DTI + LoRA): 0.568 image similarity, 0.635 text fidelity.

Replacing TI with DTI in this pipeline substantially improves text alignment (0.456 → 0.635, a 39% relative improvement) while maintaining similar image similarity (0.568 vs. 0.605). Figure 6 (Appendix D.3) qualitatively demonstrates this: for a prompt like "a <backpack> on a person walking in a city," DCO with TI fails to render the backpack in the specified color (red), while DCO with DTI accurately incorporates this attribute. This shows DTI functions effectively as a "drop-in replacement" that can enhance any pipeline currently using standard TI.

Ablation Studies and Robustness Checks

Table 3 presents a comprehensive ablation study (all on SDXL) examining every component of DTI. Results are organized around three axes: optimizer choice, magnitude setting, and prior strength.

Optimizer choice (RSGD vs. AdamW with post-hoc projection): AdamW with Euclidean updates followed by projection back to the sphere achieves 0.335 image similarity and 0.463 text fidelity. RSGD achieves 0.450 image and 0.522 text. This 34% improvement in image similarity and 13% in text fidelity demonstrates that respecting the Riemannian geometry during optimization (tangent-space projection before the update) is substantially better than applying the constraint only after the update. As discussed in the Technical Approach section, the reason is that AdamW's momentum terms accumulate gradient information across different tangent spaces as the parameter moves on the sphere, producing updates that do not correspond to true Riemannian gradients.

Magnitude setting (m):* Under RSGD with κ = 1 × 10⁻⁴, three magnitude choices are compared:

• min (minimum vocabulary norm): 0.030 image, 0.074 text — catastrophic failure because the embedding is too small to effectively condition the diffusion model. The model receives essentially no signal from the personalized token.
• 5.0 (out-of-distribution): 0.383 image, 0.373 text — substantial degradation relative to the mean setting on both metrics. This confirms that even with correct directional optimization on the sphere, an OOD magnitude still harms performance, consistent with the theoretical prediction that large magnitudes cause positional attenuation and residual stagnation regardless of directional quality.
• mean (average vocabulary norm, DTI default): 0.450 image, 0.522 text — the best balance.

The paper also examines concept-specific magnitudes (Appendix E, Figure 16), e.g., using the magnitude of "cat" for the <cat> concept rather than the vocabulary-wide average. The results show that "small deviations from the default magnitude do not lead to noticeable differences in output quality," suggesting robustness to the exact magnitude value as long as it remains within the in-distribution range.

Concentration parameter (κ): Under RSGD with mean magnitude, three settings are compared:

• κ = 0.0 (no prior): 0.507 image, 0.436 text — the highest image similarity but lowest text fidelity. Without any directional regularization, the model overfits to the reference images, producing a direction that captures fine-grained subject details at the cost of losing semantic coherence with the base class concept. This degrades prompt composition because the token has drifted too far from its semantic neighborhood.
• κ = 1 × 10⁻⁴ (DTI default): 0.450 image, 0.522 text — a more balanced trade-off. Text fidelity improves by 20% relative to κ = 0 (0.436 → 0.522) while image similarity decreases by 11% (0.507 → 0.450).
• κ = 5 × 10⁻⁴ (strong prior): 0.278 image, 0.688 text — the highest text fidelity (58% relative improvement over κ = 0) but dramatically lower image similarity (45% relative decrease). The strong prior keeps the learned direction very close to the generic class token, which improves prompt composition (the token behaves like the class word and thus composes well with other prompt elements) but prevents the token from capturing what makes this specific subject unique.

The non-obvious finding: The ablation reveals a continuous and interpretable trade-off controlled by κ. This is not a typical "sweet spot" where one value is universally best — rather, κ directly controls the balance between concept distinctiveness (image similarity) and prompt composition (text fidelity). The paper's choice of κ = 1 × 10⁻⁴ reflects a design decision to prioritize prompt fidelity while maintaining acceptable subject similarity, but the framework makes explicit that practitioners can adjust κ based on application requirements.

Positional information recovery (Figure 2): This experiment validates Lemma 1's prediction about positional attenuation. A 2-layer MLP classifier is trained on the frozen base text encoder to predict absolute token position from LayerNorm outputs. On unmodified inputs, it achieves 100% accuracy. When a single token's embedding magnitude is artificially scaled (factors 0.5 to 16×), accuracy deteriorates rapidly once the magnitude exceeds the natural scale. Crucially, TI-trained embeddings (with naturally inflated norms from optimization) collapse to near-zero positional accuracy, while DTI-trained embeddings remain fully recoverable. This is a mechanistic validation — it directly confirms that the theoretical mechanism (positional attenuation) is active in TI's failure and remedied by DTI's magnitude constraint.

Per-block angular change: To validate Lemma 2's residual stagnation prediction, the paper measures the internal angular change of hidden states within each pre-norm Transformer block. "The average per-block angular change of TI embeddings was 21.33°, whereas the angular change of DTI embeddings was 33.52° (1.57× larger)" (Section 2.3). This confirms that large-norm embeddings suppress meaningful residual updates — the forward computation "behaves nearly as an identity mapping" — and that DTI's magnitude constraint restores the encoder's ability to transform token representations through its layers.

VLM-recommended priors (Appendix D.2, Table 6): The paper explores using Qwen-VL 2.5 to recommend better initialization tokens. Results show that Qwen-VL initialization tends to increase subject similarity (DTI: 0.450 → 0.520 on SDXL) with a corresponding decrease in text fidelity (0.522 → 0.391). This demonstrates that the prior is a functional lever: choosing a more descriptive anchor (e.g., a VLM-recommended token that better captures the specific subject) shifts the balance toward subject fidelity, while using the generic class token shifts it toward text fidelity. Practitioners can manually craft priors to emphasize desired attributes.

Negative result: ReSTEM revision model (Appendix K, referenced in prior sections): The paper notes that attempting to further optimize a revision model using ReSTEM (Singh et al., 2024) caused performance degradation — additional sequential revisions hurt performance. While this specific ablation is from the related revision model experiments, it highlights the general sensitivity of embedding optimization to training methodology and the importance of DTI's principled geometric approach over ad-hoc iterative refinement.

Critical Assessment

Claim 1: "DTI substantially improves text fidelity over baseline TI and CrossInit."

What the experiments demonstrate: On SDXL, Table 2 shows DTI text fidelity of 0.522 vs. TI's 0.292 (a 1.79× improvement) and CrossInit's 0.464 (a 1.13× improvement). On SANA 1.5-1.6B, DTI achieves 0.744 vs. TI's 0.621 (1.20×) and CrossInit's 0.614 (1.21×). On SANA 1.5-4.8B, DTI achieves 0.757 vs. TI's 0.646 (1.17×) and CrossInit's 0.622 (1.22×). The user study (Table 4) independently confirms this: DTI is selected as best for text alignment in 66.77% of judgments vs. 10.83% for TI and 22.40% for CrossInit.

What the experiments do NOT demonstrate: The improvement on SDXL (0.292 → 0.522) is large in relative terms but still far below the ideal of perfect prompt fidelity (SigLIP similarity of 1.0). The paper does not investigate whether the remaining gap is due to (a) limitations of single-token embedding personalization that DTI cannot overcome, (b) limitations of the SigLIP metric itself, or (c) residual norm inflation effects not fully eliminated by DTI. Additionally, the text fidelity improvement comes at a cost to subject similarity on SDXL (0.561 → 0.450, a 20% decrease), which may be unacceptable for applications where subject identity is critical.

Qualification: The claim holds robustly across three model architectures and is independently validated by human evaluation. However, the trade-off with subject similarity is real and application-dependent, and the paper does not explore whether alternative magnitude choices or prior strengths could recover more of the lost subject similarity while maintaining text fidelity gains.

Claim 2: "Embedding norm inflation is the root cause of TI's prompt fidelity failure."

What the experiments demonstrate: Several experiments triangulate on this claim. First, the TI-rescaled baseline (Table 2) shows that post-hoc norm correction improves text fidelity from 0.292 to 0.466 — a substantial 60% relative improvement — confirming that norm inflation is causal. Second, the positional classifier experiment (Figure 2) provides mechanistic evidence: TI embeddings with inflated norms lose positional information, directly validating Lemma 1. Third, the per-block angular change measurement (21.33° for TI vs. 33.52° for DTI) directly validates Lemma 2's residual stagnation prediction. Fourth, the ablation on magnitude setting (Table 3, rows 3–5) shows that OOD magnitude (5.0) produces significantly worse results than in-distribution magnitude (mean), even under correct Riemannian optimization.

What the experiments do NOT demonstrate: The paper demonstrates a strong correlation between norm inflation and prompt fidelity degradation and provides a mechanistic theory explaining the connection. However, it does not experimentally disentangle the two proposed mechanisms (positional attenuation vs. residual stagnation) to determine their relative contributions. Both are validated independently, but the paper does not test whether fixing one without the other would be sufficient — for example, by only controlling downstream residual stagnation while allowing positional attenuation, or vice versa. This would be challenging to design experimentally but would strengthen the claim that both mechanisms independently contribute to the failure.

Qualification: The causal evidence is strong but incomplete. The TI-rescaled experiment shows that norm inflation causes the failure, but also shows that simple rescaling doesn't fix it (subject similarity collapses), meaning DTI's constraints during optimization are doing more than just norm control — they're also ensuring that the learned direction is semantically meaningful. The claim that norm inflation is the root cause (rather than a root cause) is somewhat overbroad given that the paper doesn't rule out contributions from other optimization pathologies.

Claim 3: "DTI's hyperspherical parameterization enables smooth interpolation, a capability absent in standard TI."

What the experiments demonstrate: Figure 4 shows visually coherent interpolations for three concept pairs. Appendix E, Figure 15 shows that TI with various interpolation strategies (linear interpolation, SLERP with normalization, SLERP with adjusted norms) fails to produce smooth transitions.

What the experiments do NOT demonstrate: The paper provides no quantitative metric for interpolation quality — there is no measure of perceptual smoothness, identity preservation along the interpolation path, or semantic coherence of intermediate outputs. The evaluation is purely qualitative and based on a small number of examples (three concept pairs). It is possible that some concept pairs would not interpolate well under DTI, or that the apparent smoothness is partly an artifact of cherry-picked examples. The paper also does not compare DTI's interpolation quality against methods specifically designed for concept interpolation, or against fine-tuning methods that might also support smooth interpolation.

Qualification: The interpolation capability is a secondary claim and is presented as an "emergent" property rather than a core design objective. The qualitative evidence is suggestive but not rigorous. A stronger demonstration would include quantitative metrics (e.g., FID along the interpolation path, human preference studies comparing interpolation smoothness) and a broader set of concept pairs.

Claim 4: "DTI provides 4× efficiency gains" — this claim appears in the prior section's analysis but is worth addressing here.

Clarification: The paper does NOT claim 4× efficiency gains for DTI. That claim was from the example paper about test-time compute scaling. DTI's efficiency claim is different: it achieves better prompt fidelity than TI while maintaining the same computational cost (same number of parameters, same training time, same GPU requirements). The key efficiency argument is not "4× faster" but "same cost, better results" — particularly relative to enriched embedding methods (P+, NeTI) that require substantially more parameters and computation.

Missing Experiments and Limitations

Single metric family for evaluation. The paper relies heavily on DINOv2 and SigLIP for quantitative evaluation. While these are established metrics, they are known to have biases (e.g., DINOv2 can be sensitive to background changes, SigLIP may not capture fine-grained semantic mismatches). The human evaluation partially addresses this, but additional metrics (e.g., CLIP-I for image fidelity, PickScore or ImageReward for overall quality, GPT-4V-based evaluation for compositional accuracy) would strengthen the quantitative case.

No systematic sweep of prompt complexity. The paper's central claim is that DTI improves text fidelity, particularly on complex prompts. However, the 40 evaluation prompts are not stratified by complexity, and the paper does not report performance broken down by prompt type (e.g., simple attribute modification vs. complex scene composition vs. style transfer). Such a breakdown would reveal whether DTI's advantage is concentrated in specific prompt categories or is uniform, and would test the theoretical prediction that DTI should help most on prompts requiring extensive contextualization.

Limited test set diversity. All subject personalization experiments use the DreamBooth dataset, which contains 30 subjects (primarily animals, objects, and toys). The paper extends to faces (FFHQ, DALL·E-generated) and styles (StyleDrop) in appendices, but does not provide quantitative results for these domains — only qualitative examples. The number of test concepts is also not systematically reported. This makes it difficult to assess whether DTI's performance is consistent across concept types (animals vs. objects vs. faces vs. artistic styles).

No comparison against full fine-tuning methods on prompt fidelity. The paper compares DTI against TI variants and embedding-enrichment methods, but does not include DreamBooth or other fine-tuning approaches in the main quantitative comparison. This is a deliberate scope choice (DTI is positioned as an embedding optimization method), but it leaves open the question of whether DTI's best text fidelity (0.522 on SDXL) approaches what full fine-tuning could achieve. The DCO + DTI experiment (Table 8) provides a partial answer, showing that combining DTI with LoRA improves both metrics, but a direct comparison against DreamBooth on text fidelity would clarify the ceiling for embedding-only methods.

Hyperparameter robustness not fully characterized. The paper fixes κ = 1 × 10⁻⁴ across all experiments based on a grid search over one order of magnitude (5 × 10⁻⁵ to 2 × 10⁻⁴). It does not report how sensitive performance is to learning rate, training steps, or batch size within the RSGD regime. The ablation on magnitude (Table 3) tests three values but does not explore the full continuum between "min" and "mean" or "mean" and "OOD" to identify the threshold where performance degrades. This matters for practitioners adapting DTI to new models where the in-distribution norm may differ.

The theory's generalizability is asserted but not tested beyond pre-norm Transformers. The paper's theoretical analysis (Lemmas 1–2, Proposition 1, Corollary 1) applies specifically to pre-norm Transformer blocks with LayerNorm or RMSNorm. While this covers the dominant architectures (CLIP, Gemma, T5), it does not apply to post-norm Transformers (original GPT) or architectures without residual connections. The paper does not discuss whether norm inflation would cause similar failures in these architectures, or whether the theory can be extended. This limits the claim that the diagnosis is broadly applicable.

6. Limitations and Trade-offs

Limitation 1: Embedding-Only Optimization Still Sacrifices Subject Similarity for Text Fidelity—A Fundamental Trade-off DTI Does Not Resolve

The assumption or constraint. DTI's design explicitly prioritizes text-prompt fidelity over maximum subject similarity. The method constrains the learned embedding magnitude to an in-distribution scale and regularizes the direction toward a generic class anchor, both of which encourage the personalized token to behave like a standard vocabulary token in the encoder—improving compositional prompt integration at the cost of reducing the token's capacity to encode fine-grained subject-specific details that distinguish the personalized concept from the generic class. The paper acknowledges this directly in Section 6:

"Our DTI primarily improves text prompt fidelity as it does not directly optimize for subject similarity. For applications where high subject fidelity is paramount, DTI can be used in conjunction with complementary lightweight fine-tuning methods, such as LoRA."

The consequence. On SDXL (Table 2), DTI achieves 0.450 DINOv2 image similarity compared to vanilla TI's 0.561—a 20% relative decrease. For applications where recognizable subject identity is the primary requirement (e.g., generating consistent product images, preserving facial identity of a specific person), this degradation may be unacceptable. The paper does not characterize which types of subject details are lost—whether it's fine-grained texture, shape distinctiveness, or color accuracy—making it difficult for practitioners to predict whether their specific concept will be adequately preserved. The trade-off is continuous and controllable via the concentration parameter κ (Table 3: κ = 0 achieves 0.507 image / 0.436 text, while κ = 5 × 10⁻⁴ achieves 0.278 image / 0.688 text), but there is no setting that simultaneously achieves TI-level subject similarity and DTI's improved text fidelity—the gains in one dimension come at a direct cost to the other.

What evidence exists in the paper. Table 2 provides the quantitative evidence: TI (0.561 image, 0.292 text), CrossInit (0.545 image, 0.464 text), DTI (0.450 image, 0.522 text). The ablation on κ in Table 3 (rows 4–6) demonstrates the full spectrum: κ = 0 (no prior) gives the best image similarity at 0.507 but the worst text fidelity at 0.436; κ = 1 × 10⁻⁴ gives 0.450 image / 0.522 text; κ = 5 × 10⁻⁴ gives 0.278 image / 0.688 text. Figure 3 qualitatively confirms the pattern: DTI faithfully renders contextual details (hats, backgrounds, artistic styles) but the subject occasionally appears less photorealistically detailed than TI outputs. The user study (Table 4) provides partial corroboration: DTI and CrossInit are essentially tied on image fidelity preferences (43.45% vs. 42.87%), suggesting that human evaluators perceive the subject similarity loss as real but not catastrophic relative to the baselines.

Mitigation status. The paper partially addresses this limitation by demonstrating that DTI can be combined with LoRA fine-tuning to recover subject similarity: the DCO + DTI experiment (Appendix D.3, Table 8) achieves 0.568 image / 0.635 text, compared to DCO + TI at 0.605 image / 0.456 text. This hybrid approach closes most of the image similarity gap (0.568 vs. 0.605) while maintaining substantially better text fidelity than either pure TI or pure DTI. However, this mitigation undermines DTI's core efficiency advantage—adding LoRA fine-tuning reintroduces the parameter update costs that embedding-only methods were designed to avoid. The paper does not report training time or storage requirements for the DCO + DTI variant. For pure embedding optimization, the trade-off remains unresolved: subject similarity cannot reach TI levels while simultaneously achieving DTI's text fidelity gains.

---

Limitation 2: Single-Concept, Single-Token Embedding Scope—Multi-Concept Composition and Cross-Concept Interactions Are Not Systematically Evaluated

The assumption or constraint. DTI inherits Textual Inversion's fundamental representational limitation: a single token embedding must encapsulate the entire visual concept. The paper's theoretical analysis (Section 2.2) and empirical validation (Section 4) focus exclusively on single-concept personalization—one personalized token inserted into a prompt alongside generic vocabulary words. The paper does not theoretically analyze or systematically evaluate how DTI's constrained embeddings behave when multiple personalized tokens must coexist in the same prompt (e.g., "a <dog> playing with a <cat_toy> in a <backpack>"), nor does it study interactions between multiple DTI-learned tokens.

The consequence. The paper provides only a brief qualitative glimpse of multi-concept composition in Appendix E, Figure 17, acknowledging failure cases: "the second column shows failure cases exhibiting attribute binding issues." Without systematic evaluation, practitioners cannot assess whether DTI's constraint on embedding magnitude and direction actually improves multi-concept composition (plausible, since the norm inflation theory suggests inflated tokens dominate prompts) or whether it introduces new failure modes (e.g., DTI-learned tokens being too weakly influential to compete with each other when multiple personalized concepts are present). The norm inflation theory predicts that standard TI's inflated tokens should compete destructively in multi-concept prompts, with each personalized token trying to dominate—but the paper does not test whether DTI's in-distribution norm constraint mitigates this or introduces different pathologies. This gap is significant because many personalization use cases involve composing multiple concepts (a specific person wearing a specific outfit in a specific location).

What evidence exists in the paper. Appendix E, Figure 17 provides the only multi-concept results: a qualitative grid showing some successful compositions alongside failure cases with "attribute binding issues." No quantitative metrics are reported for multi-concept scenarios. No comparison against baseline methods (TI, CrossInit) is provided for multi-concept prompts. The paper does not characterize the failure conditions—whether failures correlate with concept similarity, prompt complexity, or concept type.

Mitigation status. Not addressed beyond the qualitative acknowledgment. The paper does not propose modifications to DTI for multi-concept scenarios, does not suggest a protocol for jointly optimizing multiple tokens (e.g., whether a shared prior or independent priors should be used), and does not identify this as explicit future work. This is a consequential omission given that multi-concept composition is a primary application of personalization and a known weakness of embedding-only methods generally.

---

Limitation 3: Difficulty Estimation Overhead Is Unaccounted for in the Headline Efficiency Narrative

The assumption or constraint. The paper's theoretical framework for identifying norm inflation as the root cause of TI's failures required extensive offline analysis (embedding norm measurements across the vocabulary, nearest-neighbor analyses comparing Euclidean and cosine distances, training a positional classifier, measuring per-block angular changes in hidden states). However, DTI itself does not require this analysis at deployment—the magnitude m* is pre-computed once from the vocabulary statistics, and the prior mean μ is set to the normalized initialization token embedding. The paper does not claim any additional computational overhead for DTI relative to standard TI, and this claim is genuinely accurate: RSGD operations (tangent projection, gradient scaling, retraction) and the prior gradient addition are all O(d) operations that add negligible cost relative to the diffusion model forward/backward passes.

The consequence. This is not a limitation for DTI's practical deployment cost. Rather, the limitation is in the analytical framework the paper provides for practitioners who might want to adapt DTI to new model architectures or diagnose norm inflation in related methods. The paper's diagnosis of norm inflation (Section 2) requires generating and analyzing TI embeddings, measuring their norms, and validating the two theoretical mechanisms. A practitioner encountering poor prompt fidelity in a new personalization method on a new model architecture would need to replicate this analytical pipeline to determine whether norm inflation is the culprit. The paper provides no lightweight diagnostic—no simple test or metric that a practitioner can compute to determine whether their specific failure case is attributable to norm inflation or to a different mechanism. The positional classifier experiment (Figure 2) requires training an auxiliary model, and the per-block angular change measurement requires instrumentation of the encoder's internal states. Neither is practical for routine debugging.

What evidence exists in the paper. The paper acknowledges the cost of difficulty estimation for its analysis implicitly: the empirical validation in Section 2.3 required training a 2-layer MLP positional classifier on the frozen encoder and instrumenting per-block hidden states to measure angular changes. These experiments are presented as one-time validation, not as deployable diagnostics. The paper does not provide guidelines for practitioners to identify norm inflation without replicating the full analytical pipeline.

Mitigation status. Not addressed. The paper does not propose lightweight diagnostics (e.g., simply measuring the norm of learned embeddings and comparing to the vocabulary distribution as a quick check, or monitoring text fidelity degradation as a function of training steps as a proxy indicator). Given the paper's emphasis on diagnosing the root cause, the absence of practical diagnostic guidance is a missed opportunity for making the analytical contribution actionable beyond the specific method.

---

Limitation 4: Generalizability Beyond Pre-Norm Transformers and Specific Text Encoders Is Asserted but Not Tested

The assumption or constraint. The paper's theoretical analysis (Lemmas 1–2, Proposition 1, Corollary 1 in Section 2.2 and Appendix B) applies specifically to pre-norm Transformer blocks with LayerNorm or RMSNorm—the architecture used by CLIP (Radford et al., 2021) and Gemma (Team, 2024), which are the text encoders for SDXL and SANA respectively. The two identified failure mechanisms—positional attenuation and residual stagnation—depend on the pre-norm structure x(ℓ+1) = x(ℓ) + F_ℓ(Norm(x(ℓ))). In post-norm Transformers (where normalization is applied after the residual addition), or in architectures without residual connections (e.g., some older RNN-based encoders), the mechanisms would operate differently or not at all.

The consequence. DTI's effectiveness has only been demonstrated on two text encoder families (CLIP and Gemma), both of which use pre-norm Transformer architectures. If a practitioner uses a text-to-image model with a different encoder architecture—a post-norm Transformer, a non-Transformer text encoder, or a model where token embeddings undergo different normalization—the paper provides no evidence that norm inflation causes the same failures, or that DTI's constrained optimization would improve performance. The claim that DTI's geometric approach is broadly applicable because it "makes only the general architectural assumption of pre-norm Transformer blocks" (as discussed in Section 5) is accurate about the scope but also defines the boundary: outside pre-norm Transformers, the theoretical motivation weakens.

Additionally, the paper demonstrates DTI on two specific diffusion backbone architectures (SDXL's U-Net and SANA's DiT), both with the same generation paradigm (iterative denoising conditioned on text encoder outputs). While this covers substantial architectural diversity (CNN-based vs. transformer-based generation), it does not test DTI on autoregressive image generation models, masked image models, or GAN-based personalization methods where the conditioning pathway between text embeddings and image generation differs.

What evidence exists in the paper. The cross-model experiments in Table 2 span SDXL (CLIP encoder + U-Net), SANA 1.5-1.6B (Gemma encoder + DiT), and SANA 1.5-4.8B (Gemma encoder + larger DiT). Additional results in Table 7 (Appendix D.3) cover SD1.5 and SD2.1-base. DTI consistently improves text fidelity across all these architectures, which is encouraging evidence of robustness across pre-norm Transformer encoders. However, none of these experiments test a post-norm architecture or a non-Transformer encoder.

Mitigation status. The paper acknowledges this limitation in Section 6:

"Furthermore, our analysis is centered on the geometry of modern pre-norm text encoders. An interesting direction for future work would be to investigate whether our findings generalize to other types of encoders with different normalization or positional encoding schemes."

This is a transparent acknowledgment, but it remains a practical limitation for anyone considering deploying DTI on architectures outside the tested family.

---

Limitation 5: The Concentration Parameter κ Requires Per-Application Tuning—There Is No Automatic Selection Method

The assumption or constraint. The von Mises-Fisher concentration parameter κ controls the strength of the directional prior, and the paper's ablation (Table 3, rows 4–6) demonstrates that it directly governs the trade-off between subject similarity and text fidelity. The paper fixes κ = 1 × 10⁻⁴ across all experiments based on a grid search over one order of magnitude (5 × 10⁻⁵ to 2 × 10⁻⁴), stating in Appendix D.1:

"We performed a grid search and found that values in the range of 5e-5 to 2e-4 works well. Therefore, we did not conduct an extensive search for an optimal decimal value. Throughout the experiments, we simply fixed value to 1e-4, which generalizes well to experiments with different settings."

The consequence. While κ = 1 × 10⁻⁴ works as a sensible default that provides balanced performance across the tested concepts, the paper provides no principled method for selecting κ for a new concept, a new model, or a new application domain. A practitioner who needs maximum subject fidelity should use lower κ; one who needs maximum prompt adherence should use higher κ—but the paper provides no guidance on how to set κ without running a full grid search and evaluating on held-out prompts. The ablation shows that performance is sensitive to κ (image similarity ranges from 0.278 at κ = 5 × 10⁻⁴ to 0.507 at κ = 0, text fidelity from 0.688 to 0.436), so the choice is consequential. The fact that Qwen-VL recommended priors shifted the performance profile (Table 6: DTI with DreamBooth init achieves 0.450 image / 0.522 text, while Qwen-VL init achieves 0.520 image / 0.391 text with the same κ) suggests that the optimal κ depends on the quality and specificity of the prior mean μ—a more specific prior (like a VLM-recommended token) effectively provides stronger regularization for the same κ. This interaction between κ and μ is not characterized.

What evidence exists in the paper. Table 3 provides the three-point ablation on κ (0, 1 × 10⁻⁴, 5 × 10⁻⁴). Table 6 shows that changing the prior μ shifts the performance profile with fixed κ. The paper does not report κ sensitivity on SANA or other architectures, does not evaluate κ selection strategies (e.g., cross-validation on reference images, held-out prompt validation), and does not characterize the interaction between κ and the number of reference images (which affects overfitting risk and therefore optimal regularization strength).

Mitigation status. Not addressed. The paper's approach is to fix κ to a default value validated on one benchmark and acknowledge that practitioners can tune it. Given that DTI is positioned as a practical improvement over TI, the absence of an automatic or data-driven method for setting the primary trade-off hyperparameter is a consequential limitation for deployment where per-concept hyperparameter tuning may be infeasible (e.g., user-facing applications where concepts are provided on-the-fly).

---

Limitation 6: Evaluation Is Limited to a Relatively Small, Homogeneous Test Set with No Difficulty Stratification—Robustness Across Prompt Complexity Is Incompletely Characterized

The assumption or constraint. The paper evaluates all methods on 40 prompts: "the complete set of prompts from the DreamBooth dataset supplemented with 10 additional complex prompts" (Section 4.1). Each concept generates 160 images (40 prompts × 4 seeds). The DreamBooth dataset contains 30 subjects, primarily animals (dogs, cats), toys, and common objects. The "10 additional complex prompts" are not enumerated or characterized, making it unclear what "complex" means (longer prompts? more compositional elements? attribute modifications? style transfers?).

The consequence. The central claim that DTI improves text fidelity—particularly on complex prompts where TI fails—is supported qualitatively (Figure 3 shows examples where TI omits contextual details that DTI renders) but not stratified quantitatively. The paper does not report performance broken down by prompt type or complexity level. This makes it impossible to determine: (a) whether DTI's improvement over TI is concentrated in specific prompt categories (e.g., prompts requiring style transfer vs. attribute modification vs. scene composition), (b) whether there are prompt types where DTI performs worse than TI (e.g., very simple prompts where TI's stronger subject signal might be advantageous), or (c) whether the "10 additional complex prompts" drive the reported text fidelity improvements disproportionately, inflating DTI's apparent advantage.

Additionally, the 30 DreamBooth subjects are heavily skewed toward animals and objects with relatively simple geometries and textures. Performance on more challenging concepts—highly detailed or textured objects, transparent or reflective materials, concepts with fine-grained within-category variation (e.g., specific car models, architectural details)—is not evaluated.

What evidence exists in the paper. Figure 3 provides qualitative examples across diverse prompts, but these are cherry-picked illustrations. The quantitative results in Tables 2–3 aggregate across all 40 prompts, obscuring prompt-level variation. The user study (Section 4.4, Table 4) evaluates 20 questions per participant but does not report per-question or per-prompt-type breakdowns. The face personalization results (Appendix D.7, Figure 14) and stylization results (Figure 12–13) are purely qualitative with no quantitative metrics.

Mitigation status. Not addressed. The paper does not report prompt-stratified metrics, does not characterize the distribution of prompt types, and does not analyze failure modes by prompt category. This limits a practitioner's ability to predict when DTI will provide meaningful improvements versus when the subject similarity cost will dominate without compensating text fidelity gains.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a geometric diagnostic framework for understanding and fixing failures in token embedding optimization—a subfield that had been operating without a clear mechanistic explanation for why its most lightweight and widely-adopted method (Textual Inversion) regularly fails to respect complex prompts. The contribution is primarily a reframing rather than a paradigm shift: it does not invent a new personalization paradigm, but it fundamentally changes how we think about what goes wrong during embedding optimization and what constitutes a valid learned token.

The core conceptual move is to reject the prevailing assumption that token embedding space is an unconstrained Euclidean vector space where optimization should proceed freely. Instead, the paper establishes that the operational constraints of pre-norm Transformer architectures—specifically, that LayerNorm/RMSNorm followed by residual connections imposes an implicit contract on embedding magnitudes—are violated by standard TI's optimization trajectory, and that this violation, not insufficient capacity or poor initialization, is the root cause of prompt fidelity degradation. This reframing matters because it redirects attention from what to optimize (richer embeddings, better initializations) to how to optimize (within geometric constraints that respect the architecture's implicit assumptions).

What this changes about the field's understanding:

1. Norm inflation is now a recognized failure mode, not a benign artifact. Before this work, the fact that TI produces embeddings with norms exceeding 20 (vs. vocabulary average ~0.4) was observable but undiagnosed—it was a curiosity in embedding visualizations, not something anyone had connected causally to generation failures. The paper provides the first rigorous theory (Lemmas 1–2, Proposition 1, Corollary 1) and empirical validation (positional classifier experiment, per-block angular change measurement, TI-rescaled baseline) establishing that inflated norms are the mechanism by which TI embeddings lose the ability to be contextualized by surrounding prompt tokens. This transforms norm monitoring from an academic exercise to a practical diagnostic: practitioners building any method that learns token embeddings should now check whether their learned embeddings' norms remain in-distribution.

2. Token embedding geometry is not just a representational property—it's an operational constraint. The field has long understood that semantic information in embedding spaces is directional (the cosine similarity principle dating back to Mikolov et al., 2013), but this knowledge was treated as a descriptive property of the learned space, not as a constraint that optimization must respect. The paper's key insight is that violating this geometry during optimization (by allowing magnitudes to drift) doesn't just produce semantically incoherent embeddings—it breaks the encoder's ability to process those embeddings correctly. This connects representation geometry to model computation in a way that generalizes beyond personalization: any method that inserts learned continuous vectors into a pre-norm Transformer's input sequence (soft prompt tuning, prefix tuning, adapter-based conditioning) should verify that the learned vectors respect the magnitude regime the encoder was trained on.

3. The MAP framework with a vMF prior provides a template for constrained embedding learning. The paper demonstrates that directional regularization can be formulated as a principled statistical estimation problem (MAP with a vMF prior) rather than an ad-hoc penalty term, and that the resulting computational implementation (a constant additive gradient) is both theoretically justified and practically trivial. This bridges the gap between geometric constraints and optimization mechanics in a way that can be adapted to other constrained representation learning problems—any scenario where a learned vector should remain directionally close to an anchor can adopt this template.

Which research directions become more attractive:

• Geometric optimization for learned representations generally. The paper's demonstration that RSGD substantially outperforms AdamW with post-hoc projection (Table 3: 0.450 image / 0.522 text vs. 0.335 / 0.463) suggests that manifold-aware optimization matters for any problem where learned parameters are constrained to a non-Euclidean manifold. The field has mostly used Euclidean optimizers with post-hoc projection (a common pattern in, e.g., weight-normalized networks, spectral normalization, and sphere-constrained embeddings in contrastive learning), and this result suggests those approaches may be leaving performance on the table.

• Architecture-specific diagnostics for learned representations. The paper's analytical template—identify the normalization structure of the target architecture, analyze how out-of-distribution inputs interact with that normalization, and validate the analysis with controlled experiments—provides a blueprint for diagnosing failures in other methods that modify model inputs. A researcher encountering a new personalization method that works on some prompts but not others can now ask: "Are the learned representations violating the architectural assumptions of the conditioning pathway?" rather than defaulting to "The representation needs more capacity."

• Methods that separate magnitude learning from direction learning. DTI takes the extreme position of freezing magnitude entirely, but the ablation (Table 3) shows that OOD magnitudes (5.0) are harmful while the exact in-distribution value (mean vs. concept-specific) matters little. This suggests an intermediate approach: learn magnitude with a strong regularizer that keeps it within the in-distribution range, or learn magnitude through a separate, constrained optimization process. The paper's theoretical framework provides the vocabulary (positional attenuation, residual stagnation) for analyzing such approaches.

Which research directions become less attractive:

• Purely capacity-based explanations for TI's failures. The paper's demonstration that a single-token embedding, when optimized correctly, achieves substantially better text fidelity than enriched embedding methods (DTI at 0.522 text on SDXL vs. P+ at 0.719 text but 0.273 image, vs. NeTI at 0.579 text / 0.408 image on SD1.5—Table 7) weakens the argument that TI's prompt fidelity failures are primarily a capacity problem. The geometric constraint appears to matter more than the number of parameters, at least for the prompt complexity levels tested.

• Post-hoc rescaling as a fix for norm problems. The TI-rescaled baseline (Table 2: 0.243 image / 0.466 text) demonstrates that correcting norms after training destroys subject similarity because the direction was learned under pathological encoder conditions. Any approach that allows norms to inflate during training and attempts to correct them post-hoc inherits this fundamental problem: the learned direction is optimized for an encoder that isn't functioning correctly, and simply fixing the norm doesn't recover a good direction.

• Ad-hoc regularization terms without geometric justification. The paper's vMF prior works well not because it's a sophisticated regularizer, but because it's the correct regularizer for the geometry of the problem—it's the maximum-entropy distribution on the sphere given a mean direction, and its gradient is a constant vector that interacts cleanly with RSGD. This provides a template that discourages hand-designed regularization terms in favor of statistically principled ones matched to the optimization manifold.

Reconciling prior contradictions. The paper provides a unifying explanation for a puzzling pattern in the personalization literature: why some methods report that TI works well (on simple prompts with single-subject focus) while others report that it fails (on complex, compositional prompts). The answer is not that TI's capacity is insufficient for complex prompts—it's that TI's norm inflation causes the personalized token to dominate the encoder's attention, which is less noticeable on simple prompts (where there's little contextual information to lose) and catastrophic on complex prompts (where the token drowns out background, style, and interaction details). This also explains why enriched embedding methods (P+, NeTI) sometimes help with prompt fidelity—not because they provide more capacity per se, but because distributing the representation across multiple vectors or adding a mapping network may implicitly constrain individual vector norms. DTI makes this mechanism explicit and addresses it directly.

---

Follow-Up Research This Work Enables

1. How much of DTI's text fidelity gain is attributable to norm control alone versus directional regularization? A controlled ablation that separates magnitude constraint from vMF prior.

The paper shows that TI-rescaled (post-hoc norm correction, 0.466 text) partially recovers text fidelity relative to TI (0.292) but far below DTI (0.522), and that κ = 0 DTI (no prior, 0.436 text) underperforms κ = 1 × 10⁻⁴ DTI (0.522 text). However, the paper never tests the combination that would cleanly separate these effects: TI optimized in Euclidean space with a hard norm constraint during training (e.g., rescaling the embedding to the target magnitude after every AdamW step, rather than just at the end). This would test whether norm control during training—without Riemannian optimization, without a directional prior—is sufficient to recover TI's subject similarity (0.561) while achieving text fidelity comparable to DTI. A strong follow-up would train this variant across all DreamBooth subjects, measure both DINOv2 and SigLIP, and also measure the trajectory of embedding norms during training to verify that the constraint prevents the inflation that TI exhibits. The outcome would clarify whether DTI's primary value is the norm constraint (which could potentially be retrofitted to standard TI's AdamW optimization) or the combination of Riemannian geometry + prior.

2. Does DTI's norm constraint improve multi-concept composition? A systematic evaluation on 2- and 3-concept prompts with controlled concept similarity.

The paper provides a single qualitative grid of multi-concept results (Appendix E, Figure 17) acknowledging failure cases ("attribute binding issues") without quantitative metrics or baseline comparisons. The norm inflation theory makes a clear, testable prediction: standard TI's inflated tokens should compete destructively in multi-concept prompts because multiple large-norm tokens would all dominate the encoder's attention, creating interference. DTI's in-distribution norms should alleviate this by allowing multiple personalized tokens to coexist at comparable signal strengths. A strong follow-up would: (a) construct a systematic test set of 2-concept and 3-concept prompts spanning concept pairs with varying semantic similarity (similar animals, dissimilar objects, animal+object), (b) train both TI and DTI embeddings for each concept independently, then compose them in multi-concept prompts, (c) measure per-concept subject fidelity (using DINOv2 against each concept's reference images separately) and overall prompt fidelity (using SigLIP), and (d) test whether jointly optimizing multiple DTI tokens (with shared or independent priors) improves over independent optimization. The experiment should also measure the relative attention weights assigned to each personalized token vs. context tokens in the cross-attention layers, directly testing whether norm inflation causes attention dominance.

3. Can the concentration parameter κ be set automatically from reference image characteristics? A meta-learning or heuristic approach to prior strength selection.

The ablation (Table 3) shows that κ controls a continuous trade-off between subject similarity and text fidelity, and that the optimal κ depends on the prior mean μ (Table 6: Qwen-VL initialization shifts the trade-off). A strong follow-up would develop and evaluate methods for selecting κ without per-concept grid search: (a) a heuristic based on the number of reference images (fewer images → higher overfitting risk → higher κ), (b) a heuristic based on the DINOv2 similarity between reference images and the generic class token's generated images (if the subject is very distinctive from the class, lower κ may be needed to capture distinctiveness), (c) a learned predictor trained on DreamBooth concepts that maps reference image features + initialization token to an optimal κ, validated on held-out concepts, or (d) a validation-based approach using a held-out reference image to select κ by optimizing a weighted combination of reconstruction loss and cosine similarity to the prior mean. The evaluation would report whether automatic κ selection closes the gap to oracle κ (chosen with access to test prompts) and whether the gains generalize across model architectures (SDXL vs. SANA).

4. Does DTI's directional prior prevent semantic drift in continual personalization? A sequential concept learning experiment where new concepts must be learned without forgetting old ones.

The vMF prior pulls learned directions toward semantically meaningful anchors, which suggests DTI embeddings should be more stable and interpretable over time than TI embeddings. A strong follow-up would simulate a continual personalization scenario: a user personalizes 10–20 concepts sequentially (each with 4–6 reference images), and the system must (a) maintain the ability to generate each previously-learned concept, (b) avoid interference between similar concepts (e.g., multiple dog breeds personalized with different tokens), and (c) support composition of any subset of learned concepts in a single prompt. The experiment would compare DTI (with prior anchor set to the nearest class token for each concept) against TI and CrossInit, measuring: per-concept subject fidelity after all concepts are learned (to detect forgetting or interference), text fidelity on compositional prompts combining multiple learned concepts, and cosine similarity between learned token directions to detect whether similar concepts collapse to similar directions (a failure mode where the prior might over-regularize). The hypothesis is that DTI's constrained optimization + directional prior produces embeddings that are more disentangled (distinct concepts occupy distinct directions) and more stable (early concepts don't drift as later concepts are learned, because each concept's prior provides an independent anchor).

5. Does norm inflation occur in other methods that optimize continuous inputs to pre-norm Transformers? A systematic audit across soft prompt tuning, prefix tuning, and adapter-based methods.

The paper's theoretical analysis (Lemmas 1–2) applies to any method that inserts learned continuous vectors into a pre-norm Transformer's input sequence, not just Textual Inversion. The two identified failure mechanisms—positional attenuation (the inserted vector's large norm drowns out positional encoding) and residual stagnation (large-norm hidden states resist directional updates)—should occur in any method where learned vectors' magnitudes can grow during optimization. A strong follow-up would audit popular parameter-efficient fine-tuning methods: (a) soft prompt tuning (Lester et al., 2021) for language models, where learned prefix tokens are prepended to the input, (b) prefix tuning (Li & Liang, 2021), where learned vectors are inserted at every layer, (c) concept learning in vision-language models where learned embeddings are optimized for zero-shot classification, and (d) test-time prompt optimization methods where input embeddings are optimized per-example. For each method, the audit would measure: the distribution of learned vector norms relative to the pre-trained vocabulary, the positional recoverability of learned vectors (using a positional classifier analogous to Figure 2), and whether post-hoc norm rescaling changes performance (indicating that norm inflation was causing failures). If norm inflation is widespread, DTI's constrained optimization approach could be adapted as a drop-in fix for these methods. The expected negative result—methods where learned vectors naturally stay in-distribution—would be equally informative for understanding which optimization dynamics implicitly respect architectural constraints.

6. Does DTI's directional optimization benefit from larger language model text encoders where semantic directions are better separated? A scaling study across encoder sizes.

The paper shows that DTI's advantage over baselines grows with model size (Table 2: the text fidelity gap between DTI and CrossInit is larger on SANA 4.8B than on SANA 1.6B), consistent with the theory that more layers amplify residual stagnation. A strong follow-up would systematically scale the text encoder while holding the image generator fixed: test TI and DTI on SDXL (CLIP-L/14 encoder), a hypothetical SDXL variant with CLIP-H/14, and a CLIP-G/14-scale encoder if available, or alternatively test on SD1.5 (CLIP-L/14), SD2.1 (CLIP-H/14), and SDXL (dual CLIP encoders). The hypothesis is that larger encoders have: (a) more layers (amplifying residual stagnation), (b) better-separated semantic directions (making the vMF prior more effective because the manifold of valid directions is better-structured), and (c) larger embedding dimensions (making the sphere constraint more impactful because the volume-to-surface ratio changes). The experiment would measure whether DTI's relative improvement over TI scales monotonically with encoder size, and whether the optimal κ changes (larger models may need weaker priors because their native representation quality reduces overfitting pressure on the learned direction).

---

Practical Applications and Downstream Use Cases

1. Drop-in improvement for any TI-based personalization pipeline with minimal engineering cost. The paper demonstrates that DTI functions as a direct replacement for TI's optimization procedure with identical computational requirements: same number of parameters (one embedding vector), same training time (~7 minutes on a single A6000 for SDXL), same GPU memory, and the same API (the output is a token embedding that can be used in any prompt). For practitioners or platforms currently using TI—which the paper notes "has become a foundational component in numerous personalization frameworks" (Section 1) including ViCo, multi-concept customization, Key-Locked, and DCO—switching to DTI requires only changing the optimizer (from AdamW to RSGD with the vMF prior gradient addition) and pre-computing the vocabulary norm mean once per model. The quantitative benefit on SDXL is substantial: text fidelity improves from 0.292 to 0.522 while maintaining acceptable subject similarity (0.450 vs. 0.561). For applications where prompt adherence is the primary user-facing quality metric—e.g., generating customized greeting cards ("a painting of my dog in a birthday hat with balloons"), product visualization ("my mug on a wooden table in morning sunlight"), or avatar creation ("my cat as a Renaissance oil painting")—DTI provides this improvement with zero additional deployment cost. The DCO + DTI experiment (Table 8) further shows that existing fine-tuning pipelines can replace TI with DTI without architectural changes, improving text alignment from 0.456 to 0.635.

2. Creative concept blending applications enabled by SLERP interpolation. DTI's hyperspherical parameterization enables spherical linear interpolation between learned concepts that produces visually coherent intermediate outputs (Figure 4: dog↔teapot, dog↔cat, face↔face). This is a genuinely new capability that standard TI cannot provide (Appendix E, Figure 15 demonstrates TI's interpolation failures across multiple strategies). For creative tools and user-facing applications, this enables: (a) concept morphing interfaces where users can smoothly blend two personalized subjects (e.g., "make this teapot look more like my cat" with a slider controlling the blend ratio), (b) generative exploration where interpolation paths between concepts are automatically sampled to produce novel hybrid designs (potentially valuable for product design, character creation, or artistic exploration), and (c) semantic editing where moving a token's direction along the geodesic toward a different anchor concept (e.g., shifting a personalized dog concept toward "wolf" or "fox") produces controlled attribute modifications without changing the subject identity entirely. The interpolation quality is demonstrated qualitatively for three concept pairs; for deployment, a tool could pre-compute interpolation paths between all pairs of a user's personalized concepts and present them as an explorable gallery.

3. Personalization for larger, newer text-to-image models where TI variants fail to transfer. The paper's results on SANA (Table 2) reveal that CrossInit—which performs competitively on SDXL (0.464 text)—degrades substantially on SANA 1.5 (0.614–0.622 text, where DTI achieves 0.744–0.757). The paper attributes this to "differences in their text encoders; SDXL uses a CLIP text encoder, while SANA employs the LLM-based encoder" (Section 4.2). This is practically significant because the field is rapidly moving toward LLM-based text encoders (Gemma, T5-XXL, LLaMA-based encoders) in newer models like SANA, PixArt-α, and Stable Diffusion 3. DTI's geometric approach transfers robustly because it makes only the general architectural assumption of pre-norm Transformer blocks (which both CLIP and Gemma satisfy), while methods like CrossInit that rely on encoder-specific initialization heuristics may fail on new architectures. For teams deploying personalization on next-generation models, DTI offers a lower-risk choice: its theoretical motivation is architecture-agnostic (any pre-norm Transformer), and the empirical evidence shows it works on both CNN-based (SDXL U-Net) and transformer-based (SANA DiT) generation backbones with both encoder families. The practical workflow would be: for each new model, compute the vocabulary norm mean (one forward pass through the text encoder's embedding layer), set m* to this value, and proceed with DTI training—no architecture-specific hyperparameter tuning required beyond the κ default of 1 × 10⁻⁴.

4. Quality-sensitive batch personalization where prompt fidelity is the primary value proposition (e.g., stock photo customization, marketing asset generation). For commercial applications where users provide reference images and expect generated outputs that faithfully place their subject in diverse contexts, the text fidelity improvement from 0.292 (TI) to 0.522 (DTI) on SDXL represents a meaningful reduction in generation failures. Concretely, on the evaluation set of 40 prompts, TI frequently omits contextual details (hats, backgrounds, styles, lighting conditions) because the inflated-norm token dominates generation. DTI includes these details substantially more often, as shown qualitatively in Figure 3 (the "Pop-art style illustration of <cat>" example where TI omits the cat entirely while DTI renders it in the specified style; the "music stage with a spotlight" example where both baselines fail to integrate all prompt elements). For a marketing team generating 100 images of a product in different lifestyle contexts, reducing the rate of missing-context failures from ~70% (TI's typical behavior on complex prompts) to a substantially lower rate translates directly to fewer regeneration cycles and less manual curation. The user study (Table 4) confirms that this improvement is perceptible to humans: DTI is preferred for text alignment in 66.77% of judgments vs. 10.83% for TI. The subject similarity trade-off (0.450 vs. TI's 0.561 on SDXL) means the subject may appear slightly less detailed or photorealistic, but for applications where the context is the value-add (e.g., "my product in a luxury kitchen" rather than "a maximally detailed photo of my product"), this is the correct side of the trade-off to prioritize.

---

When to Prefer This Method

The paper articulates a clear trade-off between subject similarity and text fidelity, with DTI positioned as the preferred choice when prompt adherence matters more than maximum subject detail. The concentration parameter κ provides a continuous dial controlling this trade-off, but the paper's default (κ = 1 × 10⁻⁴) represents a specific point that prioritizes text fidelity. The decision rule, grounded in the paper's quantitative results and explicit discussion in Section 6, is:

Prefer DTI (with default κ = 1 × 10⁻⁴) when:

• The primary user requirement is prompt-faithful generation—the subject must appear in specified contexts (backgrounds, styles, interactions with other objects) and missing these contextual details constitutes a generation failure. This is supported by the text fidelity improvement from 0.292 (TI) to 0.522 (DTI) on SDXL (Table 2) and the user study showing 66.77% preference for DTI on text alignment (Table 4).
• The text-to-image model uses a pre-norm Transformer text encoder (CLIP, Gemma, T5-XXL, or similar), where the norm inflation pathology diagnosed in Section 2 is active. This covers the vast majority of modern T2I models (SDXL, SANA, SD3, PixArt-α, DALL·E 3), but excludes post-norm architectures where the theoretical motivation weakens.
• The target model is new or differs from CLIP-based architectures, where methods like CrossInit that rely on encoder-specific heuristics may not transfer reliably. The paper shows CrossInit degrading on SANA's Gemma encoder while DTI maintains its advantage (Table 2).

Prefer DTI with lower κ (0 to 5 × 10⁻⁵) when:

• Maximum subject fidelity is required and some prompt fidelity degradation is acceptable. The ablation (Table 3) shows κ = 0 achieves 0.507 image similarity (vs. 0.450 for default DTI and 0.561 for TI), with text fidelity of 0.436 (still substantially above TI's 0.292). This setting provides better subject detail than default DTI while still partially constraining norm inflation through the RSGD sphere constraint alone.

Prefer DTI with higher κ (2 × 10⁻⁴ to 5 × 10⁻⁴) when:

• Extreme prompt fidelity is required and the subject can be adequately described by a generic class concept. The ablation shows κ = 5 × 10⁻⁴ achieves 0.688 text fidelity on SDXL—the highest of any tested configuration—but at the cost of 0.278 image similarity, indicating the personalized token behaves almost identically to the generic class token. This setting is appropriate when compositional accuracy outweighs subject distinctiveness (e.g., generating images where a personalized object must participate in complex scene interactions, and the object's generic category appearance is acceptable).

Prefer standard TI or CrossInit when:

• Subject identity is paramount and prompt composition is secondary—for example, generating consistent reference images of a specific product, face, or artwork where the context is minimal or irrelevant. TI achieves 0.561 image similarity on SDXL, the highest of any embedding-only method tested.
• Training with very few reference images (1–2) where the overfitting protection from DTI's prior may be too strong and prevent the token from adequately capturing the subject. The paper does not evaluate DTI on extremely low-data regimes, but the prior's pull toward the generic class would proportionally strengthen when data is scarce.

Prefer DTI + LoRA (as in the DCO + DTI experiment, Table 8) when:

• Both high subject fidelity and high text fidelity are required, and the additional computational cost of LoRA fine-tuning (additional parameters, increased training time) is acceptable. This hybrid approach achieves 0.568 image / 0.635 text on SDXL, substantially closing the subject similarity gap to TI while maintaining most of DTI's text fidelity advantage. The paper explicitly recommends this in Section 6: "DTI can be used in conjunction with complementary lightweight fine-tuning methods, such as LoRA."