Pose Priors from Language Models

ArXiv: 2405.03689

🎯 Pitch

ChatGPT can see where your bodies should touch: by simply asking a large multimodal model to describe contact in an image and converting its words into physical constraints, 3D pose estimates of embracing, wrestling, or yoga become dramatically more realistic—without any human annotations or motion capture data.

1. Executive Summary

This paper introduces ProsePose, a framework that leverages large multimodal models (LMMs) as semantic priors for contact-aware 3D human pose estimation, extracting structured contact constraints from an LMM's natural language descriptions and converting them into differentiable loss functions to refine initial pose estimates. The method is evaluated on two-person interaction datasets—Hi4D, FlickrCI3D, and CHI3D—as well as a single-person yoga dataset (MOYO), using GPT4-V as the primary LMM and BEV or HMR2 as the base pose regressor. ProsePose narrows the gap between heuristic-based optimization baselines and fully supervised methods that require expensive contact annotations, reducing 85% of the joint error gap between the Heuristic baseline and the supervised BUDDI method on Hi4D while improving the percentage of correct contact points (PCC) by over one-third on FlickrCI3D and CHI3D relative to the same heuristic baseline. The framework establishes that LMM-derived contact constraints can meaningfully improve pose estimation quality without any task-specific contact supervision, though gains are bounded by LMM failure modes—particularly chirality disambiguation—and the coarse granularity of LMM-referenced body parts.

2. Context and Motivation

The Core Problem: Estimating 3D Pose Under Physical Contact Is Fundamentally Difficult

The paper addresses a specific, well-recognized failure mode in 3D human pose estimation: when people are in physical contact—whether embracing, dancing, wrestling, or touching their own body in a yoga pose—standard pose estimators fail to capture those contact points accurately. Body parts that should be touching end up with gaps between them; limbs that should be wrapped around another person float in empty space.

This is not a marginal edge case. Physical contact is pervasive in human social interaction. The paper notes in Section 1 that perceiving physical contact is "essential for understanding human behavior," citing research showing that parent-child touch is associated with affection and that certain forms of self-contact signal stress. Contact-rich scenarios—a hug, a handshake, a sports tackle, a yoga posture—are precisely the settings where accurate 3D reconstruction matters most for behavioral analysis, social signal processing, and applications like AR/VR character animation or automated sports analysis.

The technical challenge is straightforward to state but hard to solve: contact almost always implies occlusion. When one person's arm wraps around another's waist, the arm partially obscures the waist, and the waist partially obscures the arm. Monocular pose estimation methods rely on visible 2D keypoints, but occluded keypoints provide no signal. As the paper states in Section 1:

"scenes with contact are challenging for pose estimation methods, as some body parts are frequently occluded. This particularly holds for methods solely relying on 2D keypoints which do not convey contact information."

In other words, the very thing that defines contact—two bodies occupying adjacent space—is what makes the visual evidence ambiguous. A 2D keypoint detector cannot tell you whether two people's hands are clasped together or merely overlapping from the camera's perspective, because depth information is lost in projection. The resulting pose estimates from generic regressors, while plausible in isolation, often violate the physical reality that the bodies should be touching.

The Prior Solution: Annotate Contact Directly—At Great Cost

The paper identifies a clear lineage of prior work that addresses this problem head-on, and the common thread is labor-intensive supervision:

• Müller et al. (2021) [33] focused on self-contact—scenarios like crossed arms or hands touching the body—and required a dataset with annotated contact regions to model these complex poses.
• Fieraru et al. (2020) [10] introduced CHI3D and FlickrCI3D, the first datasets with hand-annotated ground-truth contact labels between pairs of people, mapping which fine-grained body regions are touching.
• Methods like REMIPS [11] and BUDDI [34] train models directly on these contact maps to improve multi-person pose estimation. BUDDI, for instance, learns a diffusion prior over the relative positioning of two interacting bodies, conditioned on the image.
• CloseInt [17] takes a physics-guided approach, training a diffusion model on two-person motion capture data with explicit contact supervision.

These methods work. BUDDI achieves substantially better percentage of correct contact points (PCC) than baseline heuristics—Table 1 shows BUDDI at 81.9 PCC on FlickrCI3D versus 77.8 for the heuristic baseline, and 78.0 PCC on CHI3D versus 74.1 for the heuristic. But they share a fundamental dependency: contact annotations are expensive to acquire. The paper emphasizes this point explicitly in Section 1:

"collecting these datasets is expensive, and existing publicly available datasets include only tens of thousands of images"

And again when motivating their own approach in the related work discussion (Section 2):

"contact annotations, which are crucial for these approaches, are expensive to acquire. Our method does not require any training on such annotations."

This is the central economic constraint that limits existing supervised methods. Motion capture requires specialized multi-camera studios, calibrated environments, and willing participants performing scripted interactions. Manual annotation requires trained annotators to label contact regions frame by frame. The result is that contact-annotated datasets remain small—on the order of thousands of images, not millions—and cover a limited distribution of poses and scenarios. Scaling to the diversity of human interaction visible in in-the-wild photographs (as in FlickrCI3D) is practically impossible with these approaches.

Language Encodes Contact Knowledge—But Prior Language-Pose Work Has the Same Supervision Problem

The paper's motivating insight is that language already describes physical contact richly and at scale. As they state in the first sentence of the paper:

"Language, as a human artifact, encodes a rich set of social and physical interactions. Over centuries, our vocabulary has evolved to describe the nuances of touch, with words and phrases capturing contexts as varied as hugs, handshakes, or postures in sports and yoga."

This is a genuinely interesting observation. If you ask a person (or a vision-language model) to describe an image of two people dancing, they will say things like "one person's arm is around the other person's waist" or "their hands are clasped together." These natural language descriptions contain structured contact constraints—pairs of body parts that should be touching—expressed in a form that humans intuitively understand.

Prior work has explored the connection between language and pose, but the paper identifies a critical limitation shared by all of it: it depends on paired language-pose training data, which is itself expensive to collect. Section 2 details this lineage:

• Text-to-motion generation (e.g., Guo et al., 2022 [15]; Tevet et al., 2023 [44]) uses datasets like KIT [37] and BABEL [38] that pair motion capture with textual action descriptions. These are trained to generate motion sequences from action labels or captions—a different task from contact-aware pose estimation from images.
• PoseScript [7] generates single-person poses from fine-grained textual descriptions, but requires motion capture data annotated with detailed pose language.
• PoseFix [8] introduces a labeled dataset for modifying a pose given a natural language description of the desired change (e.g., "raise the left arm"), and trains a model specifically on this data.
• PoseGPT [9] uses language as auxiliary training data in a pose regressor, but the paper explicitly notes that "PoseGPT does not produce better pose estimates than previous state-of-the-art regressors (i.e. regressors that do not use language) and applies only to the one-person setting."
• Wang et al. (2022) [46] uses a text-only LM to improve action-conditioned human-object pose estimation, but critically relies on a database of action-pose pairs to classify the input pose first—the LM is used to refine a retrieved pose, not to generate constraints from an image directly.

The paper summarizes the shared dependency: "whereas all prior work relies on training data with pairs of language and pose, which is expensive to collect, our method leverages the existing knowledge in an LMM to reason about pose from a given image." This is the key conceptual pivot. Instead of training a model on paired language-pose data, they ask: can we use a pretrained large multimodal model that already understands both images and language to extract contact constraints directly, without any additional training?

The Unexplored Question: Can Pretrained LMMs Perceive Contact?

This leads to the paper's central motivating question, stated explicitly in Section 1:

"Since written language discusses our physical interactions at great length, can large multimodal models (LMMs) trained on images and text correctly perceive physical contact in human pose?"

This question is non-obvious and practically important. Large multimodal models like GPT-4V are trained on vast corpora of image-text pairs from the internet. They have likely seen countless examples of people described as hugging, dancing, wrestling, doing yoga, shaking hands, and so on. But whether this training translates into the ability to produce structured, spatially accurate contact constraints—not just high-level descriptions like "they are hugging" but specific region pairs like "Person 1's arm touches Person 2's back"—is an open empirical question.

The paper identifies several reasons this might fail:

1. Hallucination: LMMs are known to generate plausible-sounding but factually incorrect statements about images. If the model confidently claims "hand touches shoulder" when the hand is actually near but not touching the shoulder, the resulting loss function would actively mislead the pose optimization.

2. Chirality: Can the model distinguish left from right? "Left hand touches right shoulder" is a much more precise constraint than "hand touches shoulder," but requires spatial reasoning about body-part identity that may be beyond current LMM capabilities.

3. Granularity: Do LMMs reason at the level of "upper arm" versus "forearm," or do they produce coarser descriptions like "arm"? The optimization's effectiveness depends on how precisely the contact can be localized.

4. Pose-to-constraint translation: Even if the model can describe contact, can that description be converted into a tractable optimization objective that meaningfully guides 3D mesh refinement?

How This Paper Positions Itself

The paper positions itself at the intersection of three research threads—3D pose estimation, language-pose models, and LMM capabilities—while carving out a genuinely distinct space:

Relative to contact-supervised methods: ProsePose does not compete with BUDDI or REMIPS on their own terms. It does not aim to outperform fully supervised methods that have been trained on annotated contact data. Instead, it asks: how much of the gap between unsupervised heuristics and fully supervised methods can we close using LMM priors alone? The paper frames this as a scalability argument—if LMMs can provide useful contact priors, they could reduce or eliminate the need for expensive annotation pipelines. As stated in the abstract, this offers "a scalable alternative to traditional methods that rely on human annotations or motion capture data."

Relative to prior language-pose work: ProsePose is the first method to use pretrained LMMs for contact prediction in pose estimation without any paired language-pose training data. The paper explicitly distinguishes itself: "we bridge this gap by leveraging large multimodal models (LMMs) as priors for reconstructing contact poses" (abstract). And in Section 2: "Our work differs from previous work on language and pose in several ways. First, whereas all prior work relies on training data with pairs of language and pose, which is expensive to collect, our method leverages the existing knowledge in an LMM to reason about pose from a given image."

Relative to the LMM capabilities literature: This work serves as a case study in operationalizing LMM outputs for a structured 3D vision task. Rather than using an LMM as an end-to-end predictor (which would require generating pose parameters directly), the paper develops a pipeline that extracts intermediate structured representations—contact constraints—and uses them within a classical optimization framework. This hybrid approach (LMM for semantic reasoning + optimization for geometric precision) is a design pattern that could generalize to other vision tasks where LMMs have useful implicit knowledge but lack the output modality to express it directly.

The specific contributions the paper claims (Section 1, end) are: (1) a framework for applying LMMs as contact prediction tools in pose estimation, (2) demonstration that this framework improves pose quality in both one-person and two-person settings, and (3) an analysis of framework components and LMM failure modes. The paper does not claim to solve contact-aware pose estimation or to match supervised methods—it claims to show that LMMs are "promising tools for 3D pose estimation" with "implications beyond touch" (Section 5), establishing a new direction rather than a new state-of-the-art.

3. Technical Approach

3.1 Reader Orientation

The paper builds ProsePose, a system that takes an image of one or two people in physical contact, generates a rough 3D pose estimate using an off-the-shelf regressor, then refines that estimate by asking a large multimodal model (LMM) to describe which body parts are touching and converting that description into a mathematical loss function that pulls the corresponding mesh regions together during optimization. The core idea is that pretrained LMMs possess implicit knowledge about human pose and contact—knowledge they express in natural language—and that this language can be structured into constraints and operationalized as a differentiable objective, providing contact supervision without any manually annotated contact data or task-specific training.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three sequential stages, illustrated in Figure 2a:

1. Pose Regression (initialization): Given an input image $I$ and bounding boxes $B$ for the person(s) of interest, a pretrained 3D pose regressor (BEV for two-person, HMR2 for single-person) produces an initial estimate $X_p$ of each person's 3D body parameters—joint rotations, body shape, global orientation, and translation.

2. LMM Constraint Generation: The image is cropped to the bounding boxes, extraneous people are masked out via segmentation, and the result is fed to an LMM (GPT4-V by default) with a structured prompt. The LMM is asked to list all pairs of body parts that are touching, producing a set of contact constraints $C = \{c_1, ..., c_m\}$ where each $c$ is a pair of coarse body-part region names (e.g., ("hand", "back"), ("arm", "waist (front)")). To mitigate hallucination, $N$ independent responses are sampled from the LMM, yielding $N$ constraint sets $\{C_1, ..., C_N\}$.

3. Constrained Optimization: Each constraint set $C_j$ is converted into a loss function by mapping the named regions to vertex sets on the 3D body mesh, computing the minimum distance between those vertex sets, summing across all constraint pairs, and selecting the assignment of chirality (left/right) and person-ordering that minimizes the total distance. The losses from all $N$ constraint sets are averaged, yielding $\mathcal{L}_{\text{LMM}}$. This LMM loss is combined with several auxiliary losses—a pose prior, shape regularization, 2D keypoint consistency, penalization of large deviations from the initial pose, and an interpenetration penalty—and minimized via gradient descent over the body parameters $\theta, \beta, t$ in two stages.

The output is a refined set of body parameters $X' = [\theta', \beta', t']$ whose corresponding 3D mesh more closely respects the predicted contacts.

3.3 Roadmap for the Deep Dive

• First, the pose representation and body model (SMPL-X), because the entire optimization operates on these parameters and vertex regions—understanding the parameterization is prerequisite.
• Second, the vertex regions and contact constraint definition, since these are the bridge between LMM natural-language output and geometric mesh operations.
• Third, the LMM constraint generation pipeline—prompt design, canonicalization, chirality assignment, filtering, and averaging—since this is the novel contribution and the most complex subsystem.
• Fourth, the loss function construction in detail—how a natural-language constraint becomes a differentiable scalar $\mathcal{L}_{\text{LMM}}$ that can be minimized.
• Fifth, the full optimization objective and its auxiliary losses, showing how the LMM loss integrates with standard pose estimation components.
• Sixth, the backoff mechanism and practical implementation details—what happens when the LMM fails, and how hyperparameters are chosen.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems-building paper whose core idea is that pretrained LMMs can serve as zero-shot contact annotators for 3D human pose estimation, and that their natural-language contact descriptions can be converted into tractable optimization losses through a pipeline of prompt engineering, structured parsing, region-to-vertex mapping, and multi-sample aggregation.

Pose Representation: The SMPL-X Body Model

The paper uses the SMPL-X body model (Pavlakos et al., 2019) to represent each person as a parameterized function that maps a compact set of parameters to a 3D mesh. This choice is standard in the pose estimation literature and is inherited from prior work—specifically BUDDI [34], which the paper builds on directly.

Each person $p \in \{0, 1\}$ is parameterized by four components:

• Pose parameter $\theta \in \mathbb{R}^{d_\theta \times 3}$: a vector of axis-angle rotations for each of the $d_\theta$ joints in the SMPL-X skeleton. An axis-angle representation encodes a 3D rotation as a unit vector indicating the axis of rotation, multiplied by the rotation angle. This is the standard representation used in SMPL-family models because it is differentiable and compact.

• Shape parameter $\beta \in \mathbb{R}^{d_\beta}$: a low-dimensional vector (typically 10 dimensions for SMPL-X) controlling the body's shape variation—how tall, heavy, or lean the person is. These coefficients are the weights of a linear shape space learned via principal component analysis on a large corpus of human scans.

• Global rotation $\Phi \in \mathbb{R}^3$: an axis-angle rotation that orients the entire body in the world coordinate frame. This determines which direction the person is facing.

• Global translation $t \in \mathbb{R}^3$: the 3D position of the body's root joint in world coordinates. For two-person scenes, the relative translation between $t_0$ and $t_1$ is crucial—it determines how close the two people are placed in the scene.

The full parameter set for each person is denoted $X_p = [\theta_p, \beta_p, \Phi_p, t_p]$, and the tuple $(X_0, X_1)$ is abbreviated as $X$. The single-person case is simply the two-person case with $X_0 = X_1$, meaning the same body model is used to reason about self-contact—"Person 0" and "Person 1" refer to the same mesh, and a constraint like ("hand", "foot") means one part of the mesh should touch another part of the same mesh.

What the body model produces. Given parameters $X_p$, a differentiable function (the SMPL-X forward kinematics and skinning) produces a 3D mesh:

• $V \in \mathbb{R}^{d_v \times 3}$: the positions of $d_v$ vertices on the body surface mesh. This is the geometric representation of the person's shape and pose—a triangulated surface with roughly 10,475 vertices for SMPL-X.

• $J \in \mathbb{R}^{d_j \times 3}$: the 3D positions of $d_j$ skeleton joints (e.g., wrist, elbow, shoulder), computed as a linear combination of mesh vertices. Joints are a coarser representation used for evaluation metrics and keypoint-based losses.

From the joints, 2D keypoints are computed by projecting onto the image plane:

$K_{\text{proj}} = \Pi(J) \in \mathbb{R}^{d_j \times 2}$

where $\Pi$ is the perspective projection function parameterized by camera intrinsics predicted from the image by the regressor (in practice, this comes from the SMPLify-style fitting procedure used in prior work). This equation says: take the 3D joint positions $J$, apply perspective projection using the estimated camera matrix, and output 2D pixel coordinates $K_{\text{proj}}$. These are compared against detected 2D keypoints in the loss function $\mathcal{L}_{2D}$.

Why SMPL-X rather than a simpler skeleton. The body model must be a mesh, not just a skeleton, because the contact loss operates on surface vertices. Contact is about surfaces touching—two skeletal joints being coincident would not capture the geometric reality of two bodies pressing together. The mesh provides a dense sampling of the body surface, enabling the minimum-distance computation in Equation 2 to reflect whether skin is actually in contact.

Vertex Regions: The Bridge from Language to Geometry

The central challenge of the paper is translating the LMM's natural language output—phrases like "arm," "waist (front)," "hand"—into geometric operations on a 3D mesh. This requires a mapping from region names to vertex sets, and the design of this mapping is a non-trivial engineering choice driven by the mismatch between LMM vocabulary and the granularity of existing contact annotations.

The grain-size mismatch. Prior work on contact (specifically Fieraru et al., 2020 [10] and BUDDI [34]) partitions the body into 75 fine-grained regions—distinguishing, for example, between "upper arm (front)," "upper arm (back)," "lower arm (front)," "lower arm (back)," "hand (palm)," and "hand (back)." This fine granularity is useful for precise contact localization (is the touch on the front or back of the upper arm?) but is incompatible with how LMMs describe bodies. As the paper states in Section 3.1:

"since our constraints are specified by a LMM trained on natural language, the referenced body parts are often coarser in granularity."

No LMM, when asked to describe which body parts are touching, will reliably say "Person 1's upper arm (front) touches Person 2's lower back (left)." Instead, it will say "arm touches back" or, at best, "hand touches waist."

The solution: coarse-to-fine region aggregation. The paper defines a fixed set of coarse regions by merging the fine-grained regions from [10]. For the two-person setting, the coarse regions are: head, neck, chest, stomach, waist (back), waist (front), back, shoulder (back), shoulder (front), arm, hand, leg, foot, butt. These are illustrated in Appendix Figure 7 with color coding—for example, the "arm" region is the union of all fine-grained upper-arm, lower-arm, and elbow regions (front and back), while the "back" region includes "waist (back)" and "shoulder (back)" as subsets (the paper explicitly notes these overlaps in Figure 7's caption).

For the single-person yoga setting (Appendix Figure 8), the region set is slightly different: head, stomach, back, shoulder, arm, hand, leg, foot, butt. The chest, neck, and waist regions are omitted because "they tend to be less important for contacts in yoga poses," and the front/back shoulder distinction is merged into a single "shoulder" region for the same reason.

Formally, a coarse region $R \in \mathbb{R}^{d_r \times 3}$ is a set of $d_r$ vertices that is a subset of the full mesh ($R \subset V$). The conversion from a coarse region name to a set of vertices proceeds in two steps:

1. Coarse name to fine-grained names: A lookup table maps each coarse region name to a list of fine-grained region names from the 75-region partition of [10]. For instance, "arm" maps to {upper_arm_front, upper_arm_back, lower_arm_front, lower_arm_back, elbow}.

2. Fine-grained names to vertices: A mapping provided by BUDDI [34] maps each fine-grained region name to a set of SMPL-X vertex indices. This mapping is fixed and mesh-topology-specific—it encodes which vertices belong to which anatomical region.

The result is that any coarse region name mentioned by the LMM can be converted to a concrete set of 3D vertex positions on the current mesh (since the mesh vertices $V$ are a function of the optimizable parameters $X_p$).

Constraint definition. A contact constraint $c$ is a pair of coarse regions:

$c = (R_a, R_b)$

where $R_a$ is a region from one mesh (either Person 0 or Person 1, or from the same mesh in the self-contact case) and $R_b$ is a region from the other mesh. The constraint says: the geometric distance between these two vertex sets should be as small as possible—ideally zero, indicating contact. Figure 2b illustrates this visually: one mesh's "waist (front)" region and the other mesh's "arm" region should have vertices that are coincident.

Why this region design matters. The coarse region set is the interface language between the LMM and the optimization. If the regions are too fine-grained, the LMM cannot name them consistently. If they are too coarse (e.g., a single "torso" region), the optimization cannot localize contact precisely enough to meaningfully improve the pose. The paper's choice represents a empirical balance, validated through the prompt design (Section 7.1) that explicitly lists the allowable region names so the LMM is constrained to this vocabulary.

LMM Constraint Generation: From Image to Structured Predictions

This is the novel subsystem of the paper—the pipeline that converts an image into a set of contact constraints $C$ by querying an LMM. The design addresses two fundamental challenges: (1) LMMs output unstructured natural language, not parseable data structures, and (2) LMMs hallucinate, producing false constraints that would mislead the optimization.

Image preprocessing. Before prompting the LMM, the image is prepared to focus the model's attention on the relevant people. Given bounding boxes $B$ for the subjects of interest (from ground truth for FlickrCI3D, from keypoint detectors for other datasets), the image is cropped to those bounding boxes. For the FlickrCI3D dataset, which contains in-the-wild images with potentially many people, an additional segmentation step using Segment Anything [24] masks out any individuals who are not the subjects of interest. The paper notes that "while cropping and masking the image may remove information, we find the LMMs are relatively robust to missing context, and more importantly, this allows us to indicate which individuals to focus on." This is a critical pragmatic choice: without cropping, the LMM might describe contact between background people or fail to identify which of several people in the image are the subjects.

Prompt design. The prompt is a structured instruction that constrains the LMM's output format and vocabulary. The full prompt for the two-person case (provided in Appendix Section 7.1) is:

You are a helpful assistant. You follow all directions
correctly and precisely.
For each image, identify all pairs of body parts of
Person 1 and Person 2 that are touching.
Write all of these in a Markdown table where the
first column is "Person 1 Body Part" and the second
column is "Person 2 Body Part".
You can pick which is Person 1 and which is Person 2.
The list of possible body parts is: head, neck, chest,
stomach, waist (back), waist (front), back, shoulder
(back), shoulder (front), arm, hand, leg, foot, butt.
Do not include left/right.
List ALL pairs you are confident about.
If you are not confident about any pairs, output an
empty table.
Carefully write your reasoning first, and then write
the Markdown table.

Several design choices in this prompt are deliberate and non-obvious:

• Chain-of-thought via "write your reasoning first": The prompt asks the LMM to produce free-form reasoning before generating the structured table. This is motivated by the chain-of-thought technique [48], which has been shown to improve language model performance on reasoning tasks. By explicitly thinking about the pose before listing constraints, the model is more likely to produce coherent, grounded predictions. The reasoning is not used in the optimization—it exists solely to improve the quality of the structured output.

• No left/right chirality: The prompt explicitly instructs "Do not include left/right." This is a data-driven decision. The paper reports in Section 4.3 and Table 5 that asking for left/right labels sometimes leads to worse results, with the authors concluding that "LMMs struggle with disambiguating chirality." By omitting chirality from the prompt, the method avoids one class of LMM error at the cost of less specific constraints—a "hand touches shoulder" constraint is weaker than "left hand touches right shoulder" but is more likely to be correct. Chirality is resolved later, during optimization, by enumerating possibilities.

• Closed vocabulary: The prompt provides an exhaustive list of valid body-part names. This prevents the LMM from inventing region names that cannot be mapped to vertices (e.g., "hip," "thigh," "forearm"), which would create parse failures in the downstream pipeline. The paper notes this as "careful construction of the prompt" in the Introduction.

• Empty table as uncertainty signal: The prompt explicitly allows the LMM to output an empty table if it is not confident. This is crucial for the backoff mechanism (Section 3.2, end): if too many sampled responses are empty, the system infers low LMM confidence and falls back to a baseline optimization that does not use $\mathcal{L}_{\text{LMM}}$.

• Flexible Person 1/Person 2 assignment: The prompt allows the LMM to arbitrarily assign which person is "Person 1" and which is "Person 2." This avoids forcing the LMM to match a particular convention that might conflict with its internal reasoning. The ambiguity is handled downstream by computing the loss for both possible person-orderings and taking the minimum (Equation 3).

The prompt for the single-person yoga setting is similarly structured but asks "Which body parts of the yogi are touching each other," includes "ground" as a possible region, and asks the LMM to "Describe and name the yoga pose, and then write the Markdown table"—again using chain-of-thought reasoning before the structured output.

Sampling multiple responses. To mitigate hallucination, the LMM is queried $N$ times with the same prompt and image (temperature = 0.7), producing $N$ constraint sets $\{C_1, C_2, ..., C_N\}$. The default values are $N = 20$ for GPT4-V experiments and $N = 5$ for ablation experiments with other LMMs (GPT-4o, LLaVA). The paper justifies this in Section 3.2 by analogy to self-consistency [47], a technique where multiple samples are generated and the most consistent answer is selected for code generation tasks. Here, instead of selecting a single most-consistent constraint set, the losses from all constraint sets are averaged—every sample contributes to the optimization objective, with hallucinated constraints diluted by the averaging. The empirical evidence for this choice is Figure 5, which shows that PA-MPJPE decreases and PCC increases as the number of samples grows on the FlickrCI3D validation set.

Canonicalizing region names. The LMM does not always use the exact region names from the prompt. The output parsing step checks for several variants (listed in Appendix Section 7.2): "waist" (which is mapped to the union of "waist (front)" and "waist (back)"), "shoulder" (union of "shoulder (front)" and "shoulder (back)"), and left/right variants of hands, arms, feet, legs, and shoulders. If the LMM includes a left/right label despite being instructed not to, the parser uses it directly—it does not attempt to enumerate both chiralities for that constraint, because the LMM has provided explicit (if possibly incorrect) information. If the LMM uses plural forms (e.g., "hands"), the constraint is treated as applying to both the left and right hand via the chirality condition described below.

Filtering by frequency. After canonicalizing region names across all $N$ constraint sets, constraints that appear fewer than $f$ times are discarded. The threshold $f$ is a hyperparameter: $f = 1$ for two-person experiments (meaning any constraint that appears at least once is kept) and $f = 10$ for the one-person yoga setting. The higher threshold for yoga reflects that self-contact constraints in complex poses tend to be more variable and less reliable from the LMM. This frequency filtering is a simple form of ensemble confidence: constraints that are consistently predicted across multiple samples are more likely to be real than those that appear in only one or two hallucinatory responses.

Chirality assignment. Because the LMM is asked not to specify left/right, each constraint like ("hand", "back") is ambiguous: which hand? Which side of the back? The system resolves this during optimization by enumerating all possible assignments of left/right to the unlabeled regions. For a constraint set with $k$ unlabeled chiral limbs, there are $2^k$ possible assignments (since each limb can be either left or right), and each assignment produces a different loss value when evaluated on the current pose parameters. The system selects the assignment that yields the minimum loss—that is, the assignment that makes the meshes closest to satisfying the predicted contacts.

This is a "minimum-over-assignments" approach that is computationally tractable because $k$ is typically small (a handful of limbs per constraint set) and because the minimum-distance computation is cheap. However, it has an important property: the selected chirality assignment can change across optimization iterations as the pose parameters update, meaning the loss function effectively adapts which limbs it pulls together based on which assignment is geometrically most plausible at each step. This is not a fixed labeling but a dynamic one.

The chirality condition for repeated region types. Appendix Section 7.3 describes a special case: "if the same body part (e.g. 'hand') is mentioned in at least two separate rows of the table output by the LMM (without any 'left' or 'right' prefix) or is mentioned in the plural form (e.g. 'hands'), we enforce that both the left and right limbs of this type must participate in the loss." The motivation is that when both hands are in contact (e.g., both hands on the other person's back), the LMM might list the constraint twice—("hand", "back") and ("hand", "back")—which appear identical without chirality labels. Without this condition, the minimum-over-assignments procedure might assign both constraints to the same hand (e.g., right hand touching back twice), which would not capture the actual contact configuration. The condition ensures that when a region type appears multiple times, the optimization must use different limbs for each occurrence.

Prompt ablation results. Table 5 in the appendix evaluates four alternative prompting strategies on the Hi4D and FlickrCI3D validation sets:

1. Default prompt (no left/right, no person labels): PA-MPJPE 83 (Hi4D) and 58 (Flickr).
2. Prompt asking for left/right labels (with the LMM prepending "left" or "right" to each limb): PA-MPJPE 80 and 62. Worse on FlickrCI3D.
3. Labeled image + left/right prompt (where Person 1 and Person 2 are visually labeled in the image, and the LMM is asked for chirality): PA-MPJPE 82 and 65. Worse on both datasets.
4. Caption-then-constrain (a two-stage approach: first generate a descriptive caption, then feed the caption to a text-only LM to extract contact constraints): PA-MPJPE 84 and 60. Comparable to default on Hi4D, slightly worse on FlickrCI3D.

The default prompt (no chirality, no labeling) performs best or near-best, confirming that asking LMMs for spatial detail beyond their capabilities introduces more errors than it resolves. The paper explicitly flags chirality as a limitation in Section 4.3: "prompting the LMM for left/right labels sometimes leads to worse results, suggesting that LMMs struggle with disambiguating chirality."

What the LMM actually produces. The qualitative examples in Figures 3 and 4 show concrete predictions. For a tango image (Figure 3, top), GPT4-V's top 3 constraints (with counts) are: "Hand, Hand ×20" (both times), "Hand, Shoulder (front) ×21," "Hand, Hand × 17," plus several less frequent constraints. The model correctly identifies hand-hand and hand-shoulder contact but cannot specify which hand touches which shoulder. For a rugby tackle (Figure 3, second row), the top constraints are "Arm, Waist (front) × 15" and "Back, Shoulder (front) × 13"—capturing the wrapping motion of a tackle without needing to specify left/right arms or which player is which.

Loss Function Construction: From Constraints to Differentiable Optimization

Given a set of constraints $C$ from the LMM, the system must produce a scalar loss that can be minimized by gradient descent over the pose parameters $X$. This conversion involves three sequential steps, each addressing a different source of ambiguity.

Step 1: Per-constraint distance. For a single constraint $c = (R_a, R_b)$, the paper defines the distance as the minimum Euclidean distance between any vertex in region $R_a$ and any vertex in region $R_b$:

$\text{dist}(c) = \min_{v_a \in R_a, v_b \in R_b} ||v_a - v_b||_2$

where $v_a, v_b \in \mathbb{R}^3$ are individual vertex positions on the meshes, and $R_a, R_b$ are sets of 3D vertex coordinates (subsets of the mesh vertices $V$ that correspond to the named body parts).

What it computes: the length of the shortest line segment connecting any point on region $R_a$ to any point on region $R_b$. If the regions are touching, this distance is zero (or very small). If they are far apart, the distance is large. The $\min$ over vertex pairs means the loss is sensitive only to the closest points—if most of region $R_a$ is far from $R_b$ but one vertex is close, the distance reflects that one vertex. This is appropriate for contact: we care whether any part of the hand touches any part of the back, not whether the average hand position is near the average back position.

Why this form: the minimum distance is the natural geometric measure of contact proximity. An alternative would be the average distance, but this would penalize poses where the hand is touching the back at one point but the rest of the hand is far away—a situation that is physically correct (contact is often at a small patch, not across the entire region). Another alternative would be the Chamfer distance (bidirectional average of nearest-neighbor distances), but this would penalize the non-contacting parts of the regions, which is undesirable because the LMM only asserts that some part of each region is in contact, not that the entire regions are coincident.

Computational efficiency note. The paper states that "the number of vertices in each region can be very large. To make this computation tractable, we first take a random sample of vertices from $R_a$ and from $R_b$ before computing distances between pairs of vertices in these samples." This is a stochastic approximation: rather than computing all $|R_a| \times |R_b|$ pairwise distances (which can be thousands × thousands), the system subsamples both vertex sets and computes distances only among the subsampled vertices. The minimum over the subsampled set approximates the true minimum over the full set, with the approximation becoming exact as the sample size grows.

Step 2: Summing over constraints with person-ordering ambiguity. A constraint set $C_j$ contains multiple individual constraints $c_1, c_2, ..., c_m$. The loss for the entire set is the sum of per-constraint distances, but with a critical complication: the LMM does not specify which person is "Person 1" and which is "Person 2." The constraint "Arm, Back" could mean $R_{\text{arm}}$ from Person A touching $R_{\text{back}}$ from Person B, or $R_{\text{arm}}$ from Person B touching $R_{\text{back}}$ from Person A. The system must therefore compute the loss for both orderings and select the minimum.

Let $c^\top = (R_b, R_a)$ denote the flipped constraint. The summed distance is:

$\text{dist}_{\text{sum}}(C_j) = \min\left(\sum_{c \in C_j} \text{dist}(c), \sum_{c \in C_j} \text{dist}(c^\top)\right)$

What it computes: for each constraint set, compute the total distance under two hypotheses—(1) the LMM's "Person 1" means mesh 0 and "Person 2" means mesh 1, and (2) the opposite assignment. Take the smaller total distance. This is a hard assignment: the same person-ordering hypothesis is applied to all constraints in the set, not independently per-constraint, because the LMM consistently uses "Person 1" and "Person 2" within a single response.

Why this form: the $\min$ over orderings reflects the fact that the LMM's assignment of Person 1/Person 2 is arbitrary (the prompt says "You can pick which is Person 1 and which is Person 2"). The optimization should not be penalized because the LMM happened to assign the labels in the opposite order from the mesh indices. Computing the minimum is the standard way to handle this label symmetry—it's analogous to the Hungarian matching used in set prediction losses.

Step 3: Averaging over samples. The final LMM loss is the mean of the summed distances across all $N$ sampled constraint sets:

$\mathcal{L}_{\text{LMM}} = \frac{1}{N} \sum_{j=1}^{N} \text{dist}_{\text{sum}}(C_j)$

If a constraint set $C_j$ is empty (the LMM output an empty table), then $\text{dist}_{\text{sum}}(C_j) = 0$—an empty constraint set contributes nothing to the loss, rather than being treated as evidence that nothing should be touching.

What it computes: the average over samples of the best-person-ordering total contact distance. Each sample's constraint set pulls the optimization in the direction of satisfying its predicted contacts, with the strength of the pull proportional to how many samples predict that contact (since frequently predicted constraints appear in more terms of the sum).

Why this form: averaging over samples is the key mechanism for mitigating hallucination. If the LMM hallucinates a constraint in one sample (e.g., "hand touches head" when the hand is actually near the waist), that single constraint contributes $\text{dist}(c_{\text{hallucinated}})$ to the sum for that sample, and is averaged with $N-1$ other samples that may not include it. The hallucinated constraint's influence is diluted by a factor of $1/N$. Conversely, if a true constraint appears in 18 out of 20 samples, it contributes strongly to the average and dominates the optimization. This is analogous to ensembling: each sample is a noisy "vote" for which constraints exist, and the average represents the model's consensus.

Handling empty constraint sets at the system level. If the number of empty constraint sets (samples where the LMM outputs an empty table) is at least $t$, the system infers that the LMM has low confidence and backs off entirely: $\mathcal{L}_{\text{LMM}}$ is not used, and the optimization proceeds with only the auxiliary losses (the "Heuristic" baseline). The threshold $t$ is 2 for CHI3D and $N$ (i.e., all samples must be non-empty, equivalent to requiring at least one non-empty set since $t = N$ means backoff only if all samples are empty) for all other datasets. This is a graceful degradation mechanism: if the LMM cannot identify any contacts, the system falls back to a standard optimization that doesn't use LMM information, avoiding the risk of optimizing against hallucinated or absent constraints.

The loss coefficient $\lambda_{\text{LMM}}$. In the full objective, the LMM loss is weighted by $\lambda_{\text{LMM}} = 1000$ for the two-person experiments and $\lambda_{\text{LMM}} = 10000$ for the single-person yoga setting. These values were chosen on the validation sets and reflect the need to balance the LMM loss against the other terms. The large coefficient (1000–10000) indicates that the LMM loss is measured in millimeters (vertex distances are typically on the order of 1–100 mm), while other losses are in different units (e.g., 2D pixel errors, unitless shape penalties), requiring scaling to make their gradients comparable.

The Full Optimization Objective

The refinement optimization jointly minimizes the LMM loss and five auxiliary losses. The optimization operates on a subset of the body parameters—specifically $\theta$ (pose), $\beta$ (shape), and $t$ (translation)—while the global rotation $\Phi$ is kept fixed (it is not listed among the optimized variables in Equation 5, though this is implied rather than explicitly stated). The objective is:

$[\theta', \beta', t'] = \argmin \left( \lambda_{\text{LMM}}\mathcal{L}_{\text{LMM}} + \lambda_{\text{GMM}} \mathcal{L}_{\text{GMM}} + \lambda_{\beta} \mathcal{L}_{\beta} + \lambda_{\theta} \mathcal{L}_{\theta} + \lambda_{2D} \mathcal{L}_{2D} + \lambda_{P} \mathcal{L}_{P} \right)$

where $[\theta', \beta', t']$ are the optimized parameters, and each $\lambda$ is a weighting coefficient for its corresponding loss term. The paper takes all coefficient values directly from BUDDI [34] for the two-person setting, inheriting those hyperparameter choices.

Two-stage optimization. Following BUDDI [34], the optimization is divided into two sequential stages:

• Stage 1: Optimize all three parameters $\theta, \beta, t$ simultaneously for at most 1000 steps using the Adam optimizer [23] with learning rate 0.01.

• Stage 2: Keep the shape $\beta$ fixed at its Stage 1 final value, and optimize only $\theta$ and $t$ for at most 1000 additional steps.

The two-stage design reflects a standard pose fitting practice: shape is primarily determined by the initial regressor and body proportions are less affected by contact, so fixing it after an initial refinement stage prevents the optimizer from distorting body shape to achieve contact constraints (e.g., making arms unnaturally long to reach each other). The optimizer is Adam with learning rate 0.01, a relatively high learning rate for this type of optimization, reflecting the need to make substantial pose adjustments (on the order of centimeters) within the 2000-step budget.

Auxiliary loss 1: Gaussian Mixture pose prior ($\mathcal{L}_{\text{GMM}}$). From Bogo et al. (2016) [4], this loss measures the negative log-likelihood of the pose parameters $\theta$ under a Gaussian mixture model trained on a large corpus of natural human poses from motion capture. It penalizes unnatural joint configurations—elbows bending backward, knees hyperextending, or limbs twisted into impossible orientations. The GMM prior encodes what human poses "look like" in a statistical sense, preventing the optimizer from finding solutions that satisfy the contact and keypoint constraints but create physically impossible body configurations. The coefficient $\lambda_{\text{GMM}}$ is inherited from BUDDI.

Auxiliary loss 2: Shape regularization ($\mathcal{L}_{\beta}$).

$\mathcal{L}_{\beta} = ||\beta||_2^2$

where $||\cdot||_2^2$ is the squared L2 norm (sum of squared components) of the shape parameter vector $\beta$.

What it computes: the squared magnitude of the shape coefficients, which penalizes extreme deviations from the SMPL-X mean body shape.

Why this form: the SMPL-X shape space is zero-mean by construction (the mean shape corresponds to $\beta = \vec{0}$). The L2 penalty acts as a Gaussian prior centered at the mean shape, discouraging unrealistic body proportions that might otherwise be exploited to satisfy contact constraints (e.g., giving a person an extremely narrow waist so an arm can wrap around it in a dance pose). The coefficient $\lambda_{\beta}$ controls the tradeoff between fitting the data and staying near the mean shape.

Auxiliary loss 3: Initial pose deviation ($\mathcal{L}_{\theta}$).

$\mathcal{L}_{\theta} = ||\theta' - \theta||_2^2$

where $\theta$ is the initial pose from the regressor and $\theta'$ is the current optimized pose.

What it computes: the squared Euclidean distance between the initial and current joint rotation vectors. A large value means the optimization has substantially changed the pose from its initialization.

Why this form: this is a "trust-region" penalty that prevents the optimizer from drifting too far from the regressor's estimate. The regressor (BEV or HMR2) produces a reasonable initial pose; this loss ensures that refinements are local adjustments rather than wholesale reconfigurations. Without it, the LMM loss and 2D keypoint loss might jointly converge to a dramatically different pose that satisfies the contact constraints but loses the regressor's global pose information. The penalty is an L2 distance in rotation space, treating all joint rotations equally.

Auxiliary loss 4: 2D keypoint consistency ($\mathcal{L}_{2D}$). This is the data term that grounds the 3D optimization in the image:

$\mathcal{L}_{2D} = \sum_{j=1}^{d_j} \gamma (\mathbf{K}_{\text{proj}} - \mathbf{K})^2$

where $\mathbf{K} \in \mathbb{R}^{d_j \times 2}$ are pseudo-ground-truth 2D keypoints detected by OpenPose [5] and ViTPose [49], $\gamma \in \mathbb{R}^{d_j}$ is the per-keypoint confidence from the detectors (element-wise multiplied with the squared error), and $\mathbf{K}_{\text{proj}}$ are the 2D projections of the 3D joints as defined in Equation 1.

What it computes: the confidence-weighted sum of squared pixel errors between the projected 3D joints and the detected 2D keypoints. Each keypoint $j$ contributes $\gamma_j \cdot (x_{\text{proj},j} - x_{\text{detected},j})^2$ to the sum, so high-confidence detections (e.g., clearly visible joints) contribute more than low-confidence ones (e.g., occluded joints). The sum is over all $d_j$ joints for all people in the scene.

Why this form: confidence weighting is essential because keypoint detectors assign low confidence to occluded joints—precisely the joints involved in contact. Without confidence weighting, the optimizer would try to match noisy, inaccurate detections of occluded keypoints, potentially pulling meshes away from correct contact configurations. The confidence weights effectively say "trust the detector where it's confident, and use other cues (the LMM loss) where it's not."

Auxiliary loss 5: Interpenetration penalty ($\mathcal{L}_{P}$). This loss prevents the meshes from intersecting—a common failure mode when bodies are brought close together.

$\mathcal{L}_{P} = \sum_{x \in V_{0,1}} \min_{v_1 \in V_1} ||x - v_1||_2^2 + \sum_{y \in V_{1,0}} \min_{v_0 \in V_0} ||y - v_0||_2^2$

where $V_0$ and $V_1$ are the vertex sets of the two meshes, $V_{0,1}$ is the subset of $V_0$ vertices that lie inside $V_1$ (computed via winding numbers to test point-in-mesh containment), and $V_{1,0}$ is the symmetric set of $V_1$ vertices inside $V_0$.

What it computes: the sum of squared distances from penetrating vertices to the nearest point on the surface of the other mesh. The first term sums over all vertices of Person 0 that are inside Person 1's mesh, pushing them outward to the surface of Person 1. The second term does the same for Person 1 vertices inside Person 0. The squared distance means the penalty grows quadratically with penetration depth, strongly discouraging deep intersections.

Why this form: contact optimization naturally pulls meshes together, and without a penetration penalty, they would pass through each other rather than stopping at surface contact. The winding-number computation identifies which vertices have crossed the boundary. For efficiency, this loss is computed on low-resolution versions of the meshes (roughly 1000 vertices per mesh—approximately 1/10th of the full resolution). The quadratic penalty ($||\cdot||_2^2$ rather than $||\cdot||_2$) creates a smooth gradient that increases with penetration depth, which is better behaved for gradient-based optimization than a hard boundary constraint.

Single-person adaptations. For the yoga setting, the paper makes two modifications informed by validation-set experiments (Section 7.5): (1) the GMM pose prior $\mathcal{L}_{\text{GMM}}$ is removed because "the complex yoga poses are out of distribution for the GMM prior"—a statistical model of everyday poses would penalize yoga poses as unnatural, which is exactly wrong for this domain; and (2) the weight on $\mathcal{L}_{\theta}$ (initial pose deviation) is doubled to compensate for the removed GMM prior, providing stronger anchoring to the HMR2 initialization. These changes highlight that the auxiliary loss configuration is domain-dependent and that priors useful for everyday interactions may be harmful for specialized pose domains.

Backoff Mechanism and Handling LMM Failures

The system includes explicit failure-handling logic because LMMs are unreliable and the contact prediction task is difficult. The paper identifies two failure modes that trigger a backoff to the baseline optimization (the "Heuristic" for two-person, "HMR2+opt" for single-person):

Backoff trigger 1: Too many empty constraint sets. As described above, if the number of empty constraint sets (samples where the LMM outputs "no contacts" by producing an empty table) is at least $t$, the system infers low LMM confidence and falls back. The threshold $t$ is 2 for CHI3D (where even one non-empty set out of 20 is sufficient to proceed, but 0 is not) and $N$ for other datasets (all samples must be non-empty, i.e., $t = N = 20$ means backoff only if $N$ out of 20 are empty—a stricter criterion requiring at least one non-empty sample). The different threshold for CHI3D reflects that this motion-capture dataset has controlled conditions where the LMM is more likely to produce consistent (if not always correct) predictions.

Backoff trigger 2: Optimization divergence. The paper states that the system also backs off "when the optimization diverges" (Section 3.2). The exact divergence criterion is not specified, but typically this means that the loss fails to decrease or that the mesh parameters move to numerically unstable regions (e.g., extreme shape coefficients causing mesh self-intersections that the penetration loss cannot resolve).

The Heuristic baseline. When ProsePose backs off, it uses the same optimization procedure described in Section 3.3 but without $\mathcal{L}_{\text{LMM}}$—the five auxiliary losses $\mathcal{L}_{\text{GMM}}, \mathcal{L}_{\beta}, \mathcal{L}_{\theta}, \mathcal{L}_{2D}, \mathcal{L}_{P}$ are applied as-is. Additionally, for the two-person setting, the Heuristic includes "a term that minimizes the minimum distance between the two meshes" (Section 4.1, Baseline description), which is a generic contact-encouraging loss that simply pulls the two bodies as close as possible without specifying which parts should touch. This is a weak signal compared to the LMM's structured constraints but is better than nothing.

Backoff frequency in practice. The paper reports specific backoff rates: 13 out of 241 Hi4D examples (5.4%), 106 out of 1403 FlickrCI3D examples (7.6%), 224 out of 431 CHI3D examples (52.0%), and 43 out of 76 MOYO examples (56.6%). The high backoff rates on CHI3D and MOYO suggest that the LMM often produces empty constraint sets for these datasets—possibly because CHI3D's motion-capture settings have sparse visual context and MOYO's complex yoga poses are difficult for the LMM to parse contact from. The paper does not analyze why the backoff rate differs across datasets, which leaves open the question of what image properties trigger LMM uncertainty.

Why the backoff mechanism matters architecturally. The backoff logic means ProsePose is not an end-to-end system that always uses the LMM. It is a conditional pipeline: use the LMM when confident, fall back to classical optimization when not. This hybrid design is pragmatic—it avoids the worst-case scenario where LMM hallucinations degrade performance below the baseline—but also means that the reported improvements come primarily from the subset of examples where the LMM produces useful predictions. For CHI3D, the paper makes this explicit: "On the subset of images where we do not default to the heuristic (i.e. on images where GPT4-V predicts enough non-empty constraint sets), the PA-MPJPE for ProsePose and BEV is 86 and 87, respectively." So on the 52% of CHI3D examples where ProsePose is actually used, it slightly outperforms BEV in joint error—while on the 48% where it backs off, it performs identically to the Heuristic (which underperforms BEV, leading to the overall worse joint error in Table 1).

Summary of Design Choices and Their Justifications

• Coarse region vocabulary instead of fine-grained body parts: enables LMMs to name body parts using natural language while still providing sufficiently localized constraints for optimization. The set was chosen to be both LMM-friendly (common words like "arm," "back") and geometrically meaningful (mappable to vertex sets).

• No chirality from the LMM: empirical finding that LMMs cannot reliably distinguish left from right limbs (Table 5), so chirality is resolved via enumeration during optimization rather than asked of the LMM. This is a deliberate tradeoff: less precise constraints (hand vs. left hand) in exchange for higher reliability.

• Multi-sample averaging with $N=20$ samples: mitigates hallucination by diluting spurious constraints across the ensemble, analogous to self-consistency. Figure 5 provides empirical validation.

• Minimum distance over vertex pairs rather than average or Chamfer distance: contact is a pointwise phenomenon—if any part of region A touches any part of region B, the constraint is satisfied. Average distance would penalize the non-contacting portions of the regions.

• Minimum-over-person-orderings for the summed constraint loss: the LMM's Person 1/Person 2 labeling is arbitrary, so both assignments must be considered. The $\min$ reflects that the optimization should only be penalized under the best interpretation.

• Two-stage optimization (pose+shape+translation, then pose+translation): prevents the shape parameter from distorting to satisfy contact constraints after initial rough alignment.

• Backoff to Heuristic when LMM is uncertain: graceful degradation that avoids the risk of optimizing against hallucinated or absent constraints. The threshold $t$ is dataset-specific, tuned on validation sets.

• Removal of GMM prior for yoga poses: yoga poses are out-of-distribution for a general-purpose pose prior, so the prior would penalize correct solutions. Domain knowledge (yoga is about extreme poses) overrides the general-purpose statistical model.

4. Key Insights and Innovations

Innovation 1: Recasting LMMs as Zero-Shot Contact Annotators—Language as a Scalable Substitute for Manual Contact Labels

The paper's deepest conceptual move is the reframing of large multimodal models not as end-to-end pose regressors but as zero-shot structured annotators for a task—contact region labeling—that has historically required expensive human supervision. This is not an obvious framing. The dominant paradigm in the contact-aware pose estimation literature (Fieraru et al., 2020 [10]; Müller et al., 2021 [33]; BUDDI [34]; REMIPS [11]) treats contact as a supervision signal: you collect motion capture data or manually annotate which body regions touch, you train a model on those annotations, and the model learns to predict contact from images. The entire pipeline depends on the existence and quality of the annotation dataset. ProsePose asks a fundamentally different question: can a pretrained model that has never been fine-tuned on contact annotations nevertheless produce useful contact constraints, simply by virtue of having seen enough image-text pairs to develop an implicit understanding of how bodies interact?

This is a conceptual pivot from "how do we get better at predicting contact given training data?" to "what contact knowledge already exists in models trained for entirely different purposes?" It parallels a broader intellectual trend in the LMM era—using pretrained models as zero-shot classifiers, annotators, or verifiers—but applies it to a domain (3D geometry + fine-grained spatial reasoning) where the generalization is far from guaranteed. The paper explicitly sets up this contrast in the Introduction:

"If LMMs can accurately identify contact points, they could decrease the cost of curating such datasets."

The key word is decrease, not eliminate. The paper is not claiming that LMMs replace human annotators entirely, but rather that they can produce contact annotations at a scale and cost that manual labeling cannot match, potentially enabling the collection of much larger contact-annotated datasets than currently exist (which, as the paper notes in Section 2, include only "tens of thousands of images"). This positions LMMs not as competitors to supervised methods but as enablers of future supervised methods that can be trained on LMM-generated pseudo-labels.

Evidence that this reframing matters. The paper does not outperform BUDDI—a fully supervised method—in any metric on any dataset (Table 1). But this is not a failure; it is evidence that zero-shot LMM priors are surprisingly competitive with supervised methods in terms of narrowing the gap between unsupervised heuristics and full supervision. On Hi4D, ProsePose closes 85% of the PA-MPJPE gap between Heuristic (no contact supervision) and BUDDI (full contact supervision). On FlickrCI3D, it closes over one-third of the PCC gap. These numbers suggest that LMM-derived constraints capture a substantial fraction of what supervised contact models provide, despite using no contact-annotated training data whatsoever. The "Oracle" row in Table 1—using ground-truth coarse contacts in the same optimization framework—reveals the ceiling: even perfect coarse contacts achieve only 86 PA-MPJPE on Hi4D, meaning the remaining gap between ProsePose (93) and BUDDI (89) is partly attributable to LMM errors (chirality, hallucination) but also partly to the intrinsic limitation of coarse contact constraints as an optimization signal.

Why this is fundamental, not incremental. Prior work on language and pose (PoseScript [7], PoseFix [8], PoseGPT [9]) all trains models on paired language-pose data, which requires collecting language descriptions alongside ground-truth poses. ProsePose is the first to show that a pretrained model's existing knowledge—knowledge acquired during generic vision-language pretraining, not from pose-specific annotation—can be extracted and operationalized for structured 3D reasoning. This is not an incremental improvement to an existing language-pose pipeline; it is an entirely different paradigm for how language models interface with geometric vision tasks. The paper is demonstrating a capability of existing LMMs that was not previously known or exploited, not proposing a new training objective.

Innovation 2: The Hybrid LMM + Classical Optimization Architecture—Semantics from Language, Precision from Geometry

The paper's architectural innovation is the clean decomposition of a contact-aware pose estimation system into two modules with fundamentally different strengths: an LMM that provides semantic reasoning about contact (which body parts are touching? what is the nature of the interaction?) and a classical geometry-based optimizer that provides metric precision (where exactly should the meshes be positioned to satisfy those constraints?). Neither module alone could solve the problem; their integration is what makes the system work.

This decomposition solves a specific limitation of LMMs for 3D vision tasks: LMMs operate in language space and do not output pose parameters, mesh vertices, or spatial coordinates. One could imagine fine-tuning an LMM to regress pose parameters directly, but the paper does not pursue this path—likely because (a) it would require task-specific training data and (b) LMMs are not architecturally designed for precise metric reasoning (as the chirality failures in Appendix 8.4 demonstrate). Instead, ProsePose uses the LMM for what it is good at—producing structured natural language descriptions of spatial relationships—and leaves the metric optimization to a purpose-built geometric pipeline with differentiable rendering, collision detection, and physics-inspired losses.

This design pattern—query an LMM for structured constraints, then optimize a domain-specific model to satisfy those constraints—is potentially generalizable beyond pose estimation. Any task where (a) there exists a domain-specific differentiable model (physics simulators, CAD constraints, robotic kinematics) and (b) the LMM can articulate constraints in a structured but not fully precise form could adopt this architecture. The paper does not make this generalization argument explicitly, but Figure 2a effectively diagrams a reusable template: regressor initialization → LMM constraint extraction → constrained optimization.

Evidence that the decomposition is necessary. The paper's ablation study (Section 4.1, bottom) shows the relative importance of different loss terms. Removing $\mathcal{L}_{\text{LMM}}$ increases PA-MPJPE from 81 to 138—a 70% degradation—while removing $\mathcal{L}_{2D}$ increases it from 81 to 130—a 60% degradation. Both components are essential; neither dominates. The LMM loss provides the contact information that 2D keypoints cannot (because occluded keypoints are low-confidence or undetected), while the 2D keypoint loss prevents the optimizer from drifting into poses that satisfy contact but are inconsistent with the visible evidence. The remaining auxiliary losses ($\mathcal{L}_{\text{GMM}}$, $\mathcal{L}_{\beta}$, $\mathcal{L}_{\theta}$, $\mathcal{L}_{P}$) have smaller individual effects, confirming that the core innovation is not the optimization framework itself (which largely inherits from BUDDI [34] and SMPLify [4]) but the source of the contact signal embedded within it.

The "language as intermediate representation" idea. The paper implicitly argues that natural language is a useful intermediate representation for a geometric reasoning task. This is non-obvious—why would describing contact in words be better than directly predicting contact maps (as BUDDI does) or contact vertices (as Müller et al., 2021 [33] does)? The answer, suggested by the paper's results, is that language is the native output modality of LMMs, and forcing them into a different output space (numeric coordinates, heatmaps, parameters) would require fine-tuning that defeats the zero-shot advantage. The paper accepts the limitation that language is imprecise and coarse (e.g., "arm" instead of "upper arm, anterior surface") and compensates with chirality enumeration, multi-sample averaging, and region-to-vertex mapping. Architectural constraints dictate the solution: work with the LMM's strengths rather than trying to reshape it.

Why this is a structural contribution, not a training trick. The decomposition is not specific to GPT4-V, SMPL-X, or the contact prediction task. It is a systems design pattern for combining large pretrained models with classical optimizers—a pattern that the broader vision community is beginning to explore but that this paper instantiates in a particularly clear and well-evaluated form. The backoff mechanism (where ProsePose reverts to a classical optimizer when the LMM is uncertain) makes the architecture gracefully degradable, which is essential for deployment—the system never performs worse than the baseline, and LMM improvements (better models, better prompts) yield immediate accuracy gains without architectural changes.

Innovation 3: Multi-Sample Averaging as a Hallucination Mitigation Strategy for Structured Output

While "self-consistency" and majority voting are well-established techniques for improving LLM performance on classification and code generation tasks [47, 48], ProsePose applies a related but distinct idea to continuous optimization losses derived from structured outputs. Rather than selecting the single most consistent constraint set (hard selection) or voting on individual constraints (which would require establishing correspondence between samples), the paper averages the loss functions from all $N$ samples, producing a single differentiable objective that reflects the LMM's consensus.

This is a clever formulation because it avoids the need for explicit consensus. If 18 out of 20 samples predict "hand touches back" and 2 samples hallucinate "hand touches head," the hallucinated constraints contribute $\text{dist}(c_{\text{hallucinated}})$ to only 2 of the 20 terms in the average $\mathcal{L}_{\text{LMM}}$. The optimizer sees a weak signal pulling the hand toward the head from those two samples, but a 9× stronger signal pulling the hand toward the back from the 18 correct samples. The hallucination is not filtered out—it is numerically diluted. Furthermore, if the hallucinated constraint would require violating other losses (e.g., moving the hand far from its 2D keypoint or creating an unnatural pose), those other losses resist the weak hallucinatory pull.

Evidence that averaging works better than alternatives. The paper's ablation of $N$ (Figure 5) demonstrates that more samples monotonically improve both PA-MPJPE and average PCC on FlickrCI3D, with the curve showing diminishing returns past ~15–20 samples. This validates that individual LMM samples are noisy (hallucination exists) and that averaging reduces variance. The paper does not compare averaging against alternative aggregation strategies (e.g., hard majority vote on constraint sets, weighted averaging by LMM confidence), so whether averaging is optimal is unproven—but it is demonstrably effective and requires no additional hyperparameters beyond $N$.

The connection to ensemble methods. In machine learning, averaging predictions from multiple models reduces variance (the classic bias-variance decomposition of ensemble methods). Here, the "ensemble" is created by sampling from a single stochastic model (temperature = 0.7) rather than training multiple distinct models. The rationale is the same: individual samples capture different facets of the LMM's uncertain knowledge, and averaging synthesizes them into a more reliable signal. The novelty lies not in the statistical principle but in its application to a new domain—structured geometric constraints from an LMM's natural language output, converted into a loss function that is averaged in loss space rather than in output space.

Why this is more than an implementation detail. The averaging strategy is what makes the system robust enough to be practically useful without supervised training. Without it, a single hallucinated sample could dominate the loss and pull the optimization toward a physically impossible configuration. With it, the system gracefully degrades as hallucination probability increases. The paper reports that the backoff mechanism is triggered for 5–56% of examples depending on the dataset (Section 4.1), but among the remaining examples, the averaging ensures that occasional hallucinations within otherwise non-empty constraint sets don't derail optimization. This reliability is what distinguishes a working system from a promising demo.

Innovation 4: Difficulty-Aware Graceful Degradation via LMM Uncertainty Estimation—Recognizing When the Model Doesn't Know

ProsePose includes an explicit mechanism for the LMM to signal its own uncertainty: the prompt instructs the model to output an empty table if it is not confident, and the system interprets multiple empty samples as low-confidence and falls back to a classical optimizer. This is a form of model self-assessment that is rare in LMM-based vision systems, where the typical approach is to trust the model's output unconditionally or apply post-hoc confidence calibration.

The design is simple—a threshold $t$ on the number of empty constraint sets across $N$ samples—but its implications are significant. It means ProsePose never performs worse than the baseline (Heuristic or HMR2+opt) on any example, because whenever the LMM is uncertain, the LMM loss is simply not used. The system's worst-case performance is bounded by the classical optimizer, and its best-case performance improves as LMM predictions become more accurate. This property—graceful degradation with a performance floor—is essential for any system intended for real-world deployment, but it is often absent from LMM-based pipelines that prioritize novelty over robustness.

Evidence that the backoff is needed and works. The backoff rates are high: 52% of CHI3D examples and 57% of MOYO examples trigger the fallback. Without the backoff, these examples would be optimized with $\mathcal{L}_{\text{LMM}}$ derived from empty or sparse constraint sets, which would either have no effect (wasted computation) or, worse, rely on the few hallucinated constraints that survived frequency filtering. The paper's explicit reporting of backoff rates and the breakout of performance on the non-backoff subset (86 PA-MPJPE for ProsePose on CHI3D non-backoff examples vs. 87 for BEV) is a commendable practice that makes the system's actual behavior transparent.

The conceptual parallel to selective prediction. In the selective prediction literature, models can abstain from making predictions when confidence is low. ProsePose implements abstention at the loss term level: the LMM can abstain from providing contact constraints on a per-image basis. This is a natural extension of the principle to a setting where the LMM is one of several information sources, and its contribution can be gated without sacrificing the other sources (2D keypoints, pose priors, interpenetration). The threshold $t$ functions as a confidence hyperparameter that can be tuned per-dataset to trade off between using the LMM aggressively (low $t$, higher risk of hallucination) and conservatively (high $t$, more fallbacks but safer).

Why this is a conceptual contribution beyond the paper's domain. The backoff mechanism represents a design philosophy for hybrid LMM + classical systems: use the LMM where it adds value, fall back to a trusted baseline where it doesn't. This is distinct from the more common approach of trying to make the LMM work on every input (through better prompting, fine-tuning, or retrieval augmentation). By accepting that the LMM will be unreliable on some fraction of inputs and designing the system accordingly, ProsePose achieves practical robustness without requiring LMM improvements. As LMMs improve over time, the backoff rate will naturally decrease, and the system's performance will improve without any architectural changes—a form of model-agnostic future-proofing.

How these innovations fit together. Innovation 1 (LMMs as zero-shot annotators) provides the motivation: contact annotations are expensive, but LMMs might substitute for a meaningful fraction of that cost. Innovation 2 (hybrid semantic + geometric architecture) provides the mechanism: use the LMM for what it's good at (semantic reasoning) and classical optimization for what it's good at (geometric precision). Innovation 3 (loss averaging) addresses the reliability problem: LMMs hallucinate, but averaging in loss space dilutes hallucinations without rejecting uncertain-but-correct predictions. Innovation 4 (uncertainty-based backoff) provides the safety net: when the LMM is genuinely unsure, fall back to a trusted baseline rather than risking degradation. Together, these four ideas form a complete design philosophy for integrating unreliable but knowledge-rich LMMs into reliable geometric vision pipelines—a philosophy that the paper demonstrates for contact-aware pose estimation but that arguably applies to a much broader class of vision problems.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on four datasets. For two-person interaction: Hi4D [50] (motion capture, 241 test images from a single camera, every fifth contact frame from sequences with marked contact frames), FlickrCI3D [10] (in-the-wild Flickr images, 1403 test images with manual contact map annotations, curated by BUDDI [34] to exclude noisy annotations), and CHI3D [10] (motion capture, 431 validation images used as the test set following BUDDI, distributed across 4 cameras with one "contact frame" per sequence). For single-person self-contact: MOYO [45] (motion capture of yoga poses, 76 test images from a single side-view camera, with 67 examples containing ground-truth self-contact). Dataset processing largely follows BUDDI [34] for the two-person datasets, while the MOYO split is custom-built (79 validation images from the training set, remainder combined with the official validation set for testing).

• Base model(s). Pose initialization uses BEV [42] for two-person experiments and HMR2 [12] for single-person yoga experiments. BEV was chosen because it was trained to predict both body pose parameters and relative scene placement of multiple people, making it appropriate for interaction scenarios. HMR2 was chosen for yoga because "the pose parameter estimates of HMR2 are much higher quality" for single-person settings (Section 4, Implementation details). Both models produce SMPL parameters, which are converted to SMPL-X for evaluation. The LMM is GPT4-V (gpt-4-vision-preview) by default, with ablation experiments using GPT-4o (gpt-4o-08-06) and LLaVA-NeXT 34B [30]. All LMMs are used zero-shot without fine-tuning on contact data, which is the paper's central methodological claim.

• Metrics. Three metrics are reported. Procrustes-aligned Mean Per Joint Position Error (PA-MPJPE) in millimeters: finds the best rigid alignment (rotation, translation, scale) between estimated and ground-truth 3D joint positions before computing per-joint Euclidean error, then averaging across joints. Lower is better. For two-person settings, joint PA-MPJPE is reported (incorporating relative translation and orientation of both people, evaluating the full interaction geometry), while Appendix Table 4 separately reports per-person PA-MPJPE. Percentage of Correct Contact Points (PCC) [34]: the fraction of ground-truth fine-grained contact region pairs (from the 75-region partition of [10]) that are predicted to be within a specified radius $r$. A pair is classified as "in contact" if both regions are within $r$ mm of each other. PCC is averaged across radii $r \in \{0, 5, 10, 15, ..., 95\}$ mm. Higher is better. PCC captures whether the estimated meshes are actually touching at the correct anatomical locations. F1 score on coarse contact predictions: precision and recall of the LMM's raw predicted region pairs against ground-truth pairs derived from mesh distances (threshold 0.01m for Hi4D, 0.02m for MOYO). This metric evaluates LMM accuracy independent of optimization quality. Chirality is ignored in F1 computation.

• Baselines. For two-person experiments: (1) BEV [42] — the raw regressor output, a multi-person pose estimation method using relative depth annotations for spatial placement; (2) Heuristic — BEV initialization followed by the same optimization as ProsePose but without $\mathcal{L}_{\text{LMM}}$, instead including a generic term that minimizes the minimum distance between the two meshes (introduced by BUDDI [34], hyperparameters from that work). The Heuristic is what ProsePose backs off to when LMM confidence is low; (3) BUDDI [34] — a fully supervised method using a learned diffusion prior trained on annotated interaction data to constrain optimization. For single-person experiments: (1) HMR2 [12] — raw regressor output; (2) HMR2+opt — HMR2 initialization followed by the same optimization as ProsePose but without $\mathcal{L}_{\text{LMM}}$ (the backoff method for the yoga setting). The paper also reports an Oracle baseline (coarse ground-truth contacts used in optimization, with correct chirality) and a BUDDI+ProsePose variant where $\mathcal{L}_{\text{LMM}}$ is added to BUDDI's optimization.

• Generation budget / compute accounting. For LMM queries, the paper samples $N = 20$ responses from GPT4-V (default) or $N = 5$ for ablation LMMs (GPT-4o, LLaVA). Each sample is one API call with temperature 0.7. The optimization runs for at most 1000 steps per stage (two stages), using Adam with learning rate 0.01. BUDDI takes 64 seconds per example on average versus 89 seconds for BUDDI+ProsePose on Hi4D validation (Appendix 8.2.3). Sampling 20 programs from GPT-4o takes approximately 16 seconds averaged over 30 examples. The difficulty estimation cost (for backoff decisions and frequency filtering) is folded into the sampling cost—no separate difficulty estimation step exists, unlike the compute-optimal scaling paper from the reference.

• Cross-validation / statistical protocol. The paper does not use cross-validation for test-set evaluation. Hyperparameters ($\lambda_{\text{LMM}}$, $N$, $f$, $t$, prompt design, region vocabulary, optimization coefficients) were "chosen based on experiments on the validation sets" (Section 4, Implementation details). For FlickrCI3D and Hi4D, official validation splits were used. For CHI3D, a sample of the training set was used for development. For MOYO, a custom 79-image validation set was created from the training data. No standard errors, confidence intervals, or statistical significance tests are reported. The test set sizes are: 241 (Hi4D), 1403 (FlickrCI3D), 431 (CHI3D), and 76 (MOYO). Following BUDDI [34], images where BEV or keypoint detectors fail to detect one subject are excluded from evaluation.

Main Quantitative Results

Two-Person Interaction Results: ProsePose Narrows the Gap Between Unsupervised Heuristics and Full Supervision

Table 1 provides the aggregate results across Hi4D, FlickrCI3D, and CHI3D. The headline finding is that ProsePose consistently improves over the strongest baseline that does not use contact supervision (Heuristic) across all three datasets, while falling short of the fully supervised BUDDI—a result that frames ProsePose as closing a meaningful fraction of the supervision gap.

Hi4D dataset (Table 1, column 1). The Heuristic baseline achieves PA-MPJPE of 116 mm; BUDDI achieves 89 mm—a gap of 27 mm attributable to contact supervision. ProsePose achieves 93 mm PA-MPJPE, narrowing that gap by 85% (reducing 23 of the 27 mm difference). In F1 score on coarse contact prediction, ProsePose achieves 24, while the Oracle (ground-truth coarse contacts) achieves 100—indicating that the LMM correctly identifies roughly one-quarter of the ground-truth coarse contact pairs, and that perfect coarse contact prediction would dramatically improve results (Oracle PA-MPJPE drops to 81 mm). BUDDI+ProsePose achieves 88 mm PA-MPJPE, a marginal 1 mm improvement over BUDDI alone, suggesting that when a strong contact prior is already available, the LMM signal provides little additional benefit.

FlickrCI3D dataset (Table 1, column 2). The Heuristic achieves 77.8 average PCC; BUDDI achieves 81.9—a gap of 4.1 points. ProsePose achieves 79.9 PCC, closing over one-third of the gap. In PA-MPJPE, ProsePose achieves 58 mm vs. Heuristic's 67 mm—a 9 mm improvement. The Oracle achieves 86 mm PA-MPJPE and 100 PCC, showing substantial headroom from better contact prediction. BUDDI+ProsePose achieves 83.2 PCC (vs. BUDDI's 81.9, a 1.3 point improvement) and 65 vs. 66 PA-MPJPE—again, marginal gains when combined with full supervision. The F1 score for ProsePose is 13, indicating the LMM's raw predictions are substantially noisy.

CHI3D dataset (Table 1, column 3). This dataset reveals a more complex picture. ProsePose achieves 75.8 PCC (vs. 74.1 for Heuristic) and 23 F1, but PA-MPJPE is 100 mm—worse than BEV at 96 mm and Heuristic at 105 mm. The paper explicitly addresses this seeming contradiction: on the subset of images where ProsePose does not fall back to the Heuristic (224 out of 431 examples trigger the backoff), the PA-MPJPE is 86 mm for ProsePose vs. 87 mm for BEV—marginally better. The overall worse PA-MPJPE is attributed to the Heuristic's poor performance on the backoff examples. The Oracle achieves 83.8 mm PA-MPJPE and 100 PCC. BUDDI achieves 68 mm PA-MPJPE and 78.0 PCC. BUDDI+ProsePose achieves 69 mm PA-MPJPE (worse than BUDDI alone, a 1 mm degradation) but 78.8 PCC (0.8 point improvement).

PCC at specific radii (Table 2). Breaking down PCC by radius provides fine-grained evidence that ProsePose brings meshes closer at correct contact points. On FlickrCI3D at radius 10 mm (a fairly tight contact threshold), ProsePose achieves 39.9 PCC vs. Heuristic's 33.9—a 6.0 point absolute improvement. At 15 mm, ProsePose achieves 57.1 vs. Heuristic's 49.3—a 7.8 point improvement. On CHI3D at 10 mm, ProsePose achieves 35.2 vs. 28.0 for Heuristic (7.2 point improvement) and at 15 mm, 52.5 vs. 45.3 (7.2 point improvement). In all cases, BUDDI outperforms ProsePose (44.2 PCC at 10 mm on FlickrCI3D), and BUDDI+ProsePose outperforms BUDDI alone (49.3 PCC at 10 mm on FlickrCI3D, a 5.1 point gain), suggesting that even with a learned prior, the LMM's explicit contact constraints add value for fine-grained contact precision.

Per-person PA-MPJPE (Appendix Table 4). The per-person breakdown (evaluating each person's pose independently rather than joint error) shows ProsePose achieving 65 mm on Hi4D (same as Heuristic, vs. 76 for BEV and 70 for BUDDI), 31 mm on FlickrCI3D (same as Heuristic, vs. 71 for BEV and 43 for BUDDI), and 49 mm on CHI3D (slightly worse than Heuristic's 48, vs. 51 for BEV and 47 for BUDDI). The per-person metrics show smaller relative gains than joint metrics, confirming that ProsePose's primary contribution is in the relative positioning of the two people (captured by joint PA-MPJPE and PCC) rather than individual pose quality (which is largely determined by the 2D keypoint loss and initial regressor).

Single-Person Yoga Results: Self-Contact Improves Under LMM Guidance

Table 3 reports results on the MOYO yoga dataset. HMR2 (raw regressor) achieves 84 mm PA-MPJPE and 83.0 average PCC—a surprisingly high PCC given that HMR2 was not trained for contact, suggesting that the regressor incidentally captures some self-contact in standard poses. HMR2+opt (optimization without LMM loss) improves to 81 mm PA-MPJPE and 85.2 PCC—the auxiliary losses alone provide modest contact improvement. ProsePose achieves 82 mm PA-MPJPE (slightly worse than HMR2+opt in joint error) but 87.8 PCC—a 2.6 point improvement over HMR2+opt and 4.8 points over raw HMR2. The F1 score is 25. The backoff mechanism is triggered for 43 out of 76 examples (56.6%), meaning ProsePose is only active on 33 examples—yet the PCC improvement on the full test set is measurable, indicating substantial gains on those 33 examples.

At specific radii (Table 3, PCC columns): at 5 mm radius (very tight contact), ProsePose achieves 54.2 vs. 47.7 for HMR2+opt (+6.5 points) and 34.2 for HMR2 (+20 points). At 15 mm, ProsePose achieves 73.8 vs. 65.5 for HMR2+opt (+8.3 points). These gains at tight radii suggest that the LMM loss genuinely pulls contacting body parts closer together, not just at coarse proximity but at near-touching distances.

The Role of Multiple Samples: Hallucination Mitigation in Practice

Figure 5 (on the FlickrCI3D validation set) shows the effect of varying the number of LMM samples $N$. Joint PA-MPJPE decreases (improves) and average PCC increases (improves) monotonically as $N$ increases, with clear diminishing returns beyond approximately 15 samples. This validates the key design choice of multi-sample averaging: individual LMM samples are noisy, and aggregating across more samples reduces variance in the loss function. The paper does not report the absolute numbers at each $N$ value, but the trend lines in Figure 5 demonstrate that a single sample ($N=1$) would produce substantially worse results than the default $N=20$.

Prompt and LMM Variant Ablation: Chirality Requests Degrade Performance

Table 5 in the appendix reports an ablation study varying the LMM and prompt on the Hi4D and FlickrCI3D validation sets with $N=5$ samples. The default GPT4-V with no-chirality prompt achieves 83 mm PA-MPJPE on Hi4D and 58 mm on FlickrCI3D. Key comparisons:

• Asking for left/right labels (Prompt 1 in the prompt ablation study, Section 7.1.1): PA-MPJPE changes to 80 mm on Hi4D (3 mm improvement) but 62 mm on FlickrCI3D (4 mm degradation). The mixed effect supports the paper's conclusion that chirality is unreliable from LMMs—it sometimes helps when correct, but the errors on in-the-wild images offset the gains.

• Labeled image + left/right (Prompt 2): 82 mm on Hi4D and 65 mm on FlickrCI3D. Explicitly labeling Person 1 and Person 2 in the image and asking for chirality produces the worst FlickrCI3D performance among all GPT4-V variants, suggesting that the additional visual annotation does not resolve the underlying spatial reasoning limitation.

• Caption-then-constrain (Prompt 3, two-stage approach generating a prose description then converting to constraints via a text-only LM): 84 mm on Hi4D and 60 mm on FlickrCI3D. Comparable to default on Hi4D, slightly worse on FlickrCI3D. This approach is architecturally similar to Wang et al. (2022) [46]'s pipeline, suggesting that directly prompting for structured constraints is at least as effective as the two-stage caption-then-parse approach.

• GPT-4o (default prompt): 84 mm on Hi4D and 57 mm on FlickrCI3D—comparable to or slightly better than GPT4-V, indicating the framework is not sensitive to the specific LMM version as long as the model is sufficiently capable.

• LLaVA-NeXT 34B (two variants): 86–89 mm on Hi4D and 61–67 mm on FlickrCI3D. LLaVA underperforms GPT4-V, which the paper attributes to general capability differences: "this is in line with holistic multimodal evaluations that indicate that GPT4-V performs better than LLaVA" (Appendix 8.3).

LLaVA full test set results (Appendix Table 6). Using LLaVA+GPT4 (caption-to-table pipeline, $N=5$), PA-MPJPE on Hi4D is 95 mm (vs. 93 for GPT4-V with $N=20$), on FlickrCI3D is 60 mm (vs. 58), on CHI3D is 101 mm (vs. 100), and on MOYO is 82 mm (vs. 82). PCC on FlickrCI3D is 79.7 (vs. 79.9), on CHI3D is 75.2 (vs. 75.8), and on MOYO is 85.2 (vs. 87.8). The consistent but modest underperformance of LLaVA confirms that LMM quality matters—better LMMs produce better contact constraints and better final poses—but also that the framework is not brittle to LMM choice; even a weaker open-source LMM provides useful signal.

GT probability baseline (Appendix Table 5). A random baseline that samples constraints from the dataset-wide empirical distribution achieves 101–110 mm PA-MPJPE on Hi4D (with/without chirality), demonstrably worse than any LMM variant. This confirms that the LMM is not simply reciting common contact patterns—it is extracting image-specific information.

Ablation Studies and Robustness Checks

• Loss component ablation (Section 4.1, Two-person Pose Refinement, final paragraph): On FlickrCI3D validation, removing individual loss terms from the full optimization produces the following PA-MPJPE values: full system 81 mm; remove $\mathcal{L}_{\text{LMM}}$ → 138 mm (degradation of 57 mm, the largest single-factor effect); remove $\mathcal{L}_{\text{GMM}}$ → 85 mm; remove $\mathcal{L}_{\beta}$ → 91 mm; remove $\mathcal{L}_{\theta}$ → 84 mm; remove $\mathcal{L}_{2D}$ → 130 mm (degradation of 49 mm, second-largest effect); remove $\mathcal{L}_{P}$ → 78 mm. The LMM loss and 2D keypoint loss are the two most impactful components, while the interpenetration loss removal actually improves PA-MPJPE (78 vs. 81), suggesting that for these examples, the penetration penalty may conflict with achieving correct contact proximity.

• Number of LMM samples $N$ (Figure 5, Section 4.1): On FlickrCI3D validation, increasing $N$ monotonically reduces PA-MPJPE and increases average PCC, with diminishing returns beyond approximately 15 samples. The figure does not report exact numbers, but the visual trends validate the core strategy and suggest the default $N=20$ is near-optimal for the cost-accuracy tradeoff.

• LMM choice (Table 5 and Appendix Table 6): GPT4-V (default), GPT-4o, and LLaVA all produce improvements over the Heuristic baseline, with performance ranking corresponding to general LMM capability (GPT-4o ≈ GPT4-V > LLaVA). The framework is robust to LMM choice—even the weaker LLaVA provides gains—but LMM quality directly impacts result quality. On MOYO, LLaVA achieves no PCC improvement over HMR2+opt (both at 85.2 PCC, Appendix Table 6), suggesting that yoga-specific contact reasoning may exceed LLaVA's training distribution.

• Prompt design (Table 5): Four prompt variants are compared. The default prompt (no chirality, chain-of-thought reasoning, explicit region vocabulary, empty table for uncertainty) performs best or near-best across datasets. Asking for chirality degrades performance on FlickrCI3D (62 vs. 58 PA-MPJPE). The two-stage caption approach performs comparably to direct constraint prompting. Labeling people in the image does not help. The negative results on chirality-specific prompts are informative: they demonstrate that the LMM's contact perception is genuinely coarse, and forcing it to produce more specific outputs introduces errors rather than resolving ambiguity.

• Chirality assignment strategy (Appendix Section 7.3 condition): The paper describes a special condition where repeated mentions of the same unlabeled body part type (e.g., "hand" appears in two different constraint rows) forces the optimization to use both left and right limbs. This is not empirically ablated—no results are reported comparing optimization with and without this condition—so its importance is asserted based on conceptual motivation rather than experimental evidence.

• Backoff mechanism (Section 4.1, CHI3D analysis): The paper reports a subset analysis on CHI3D: on the 207 of 431 examples where ProsePose does not default to the Heuristic, PA-MPJPE is 86 mm for ProsePose vs. 87 mm for BEV. This subset analysis demonstrates that the LMM loss provides genuine improvement where it is applied, and the overall worse PA-MPJPE (100 vs. 96) is attributable to the Heuristic's poor performance on the backoff examples rather than ProsePose being harmful when active.

• BUDDI + ProsePose combination (Tables 1 and 2): Adding $\mathcal{L}_{\text{LMM}}$ to the fully supervised BUDDI optimization produces mixed but generally positive effects on PCC: +1.3 points on FlickrCI3D (81.9 → 83.2), +0.8 points on CHI3D (78.0 → 78.8), and improved fine-grained PCC at specific radii (e.g., +5.1 points at 10 mm on FlickrCI3D, Table 2). However, PA-MPJPE is essentially unchanged or slightly worse (88 vs. 89 on Hi4D, 65 vs. 66 on FlickrCI3D, 69 vs. 68 on CHI3D). This suggests that LMM constraints provide complementary contact information even when a strong learned contact prior exists, but the contact improvements do not translate to overall joint error reductions—possibly because BUDDI already achieves near-optimal joint configurations and additional contact tightness comes at the cost of other pose quality dimensions.

• Oracle coarse contacts (Tables 1, 2, 3): Using ground-truth coarse region pairs with correct chirality in the optimization serves as an upper bound. On Hi4D, the Oracle achieves 81 mm PA-MPJPE and 100 F1; ProsePose achieves 93 mm and 24 F1. The 12 mm gap in PA-MPJPE between Oracle and ProsePose represents the headroom achievable through improved LMM contact prediction. On MOYO, the Oracle is not reported (no test-set contact annotations exist for MOYO).

• Running time (Appendix 8.2.3): BUDDI averages 64 seconds per example on Hi4D validation; BUDDI+ProsePose averages 89 seconds—a 39% increase. Sampling 20 programs from GPT-4o takes 16 seconds on average (30-example measurement). The LMM sampling cost is thus a modest fraction of the total pipeline, and the optimization dominates runtime.

Critical Assessment

The experiments demonstrate a clear and consistent pattern: ProsePose improves contact-aware pose estimation over heuristic baselines that lack contact supervision, and these improvements are attributable to the LMM-derived contact constraints rather than to the optimization framework itself (established by the loss ablation and the comparison to the Heuristic baseline). The improvements are meaningful in magnitude—85% gap closure on Hi4D PA-MPJPE, one-third gap closure on PCC for FlickrCI3D and CHI3D—and robust across four datasets, two pose regressors, and multiple LMMs. The empirical case that LMMs can serve as useful zero-shot contact priors is solid.

However, several important claims from the paper require qualification when examined against the experimental evidence:

Claim: "Our method produces compelling reconstructions... accurately capturing the semantics of physical and social interactions" (Abstract). The quantitative results support "improves over baselines that lack contact supervision." Whether the reconstructions are "compelling" and "accurately capturing semantics" is a stronger claim that the qualitative examples (Figures 3, 4) illustrate but the quantitative metrics only partially capture. The PCC metric measures whether meshes are closer at correct contact points—it does not assess whether the configuration of contact (who is touching whom, how, with what body language) matches the semantic interaction. The Oracle row in Table 1 shows that even with perfect coarse contacts, PA-MPJPE on Hi4D is 81 mm—meaning 19 mm of error remains from other sources (2D keypoints, pose priors, mesh topology). The LMM's contribution, while real, is bounded by this remaining error. Furthermore, the F1 scores of 13–25 (Tables 1, 3, 6) indicate that the LMM correctly predicts only a minority of ground-truth contact pairs—so "accurately capturing semantics" is relative to having no contact information at all, not relative to a complete contact description.

Claim: "LMMs can serve as powerful tools for contact prediction and pose estimation, offering an alternative to costly manual human annotations or motion capture data" (Abstract). The experimental evidence supports "useful" more than "powerful." The LMM's F1 on coarse contact prediction is 24 on Hi4D, 13 on FlickrCI3D, 23 on CHI3D, and 25 on MOYO—meaning roughly 75–87% of ground-truth contacts are missed or incorrectly predicted. The gap between ProsePose and the Oracle (which uses perfect coarse contacts) is substantial (93 vs. 81 mm PA-MPJPE on Hi4D, 79.9 vs. 86 PCC on FlickrCI3D). The paper's framework demonstrates that LMM priors can reduce but not replace the need for annotated data, and the authors acknowledge this implicitly by framing LMMs as tools for "creating larger datasets with contact annotations" rather than as standalone replacements for supervised methods (Section 5). The experiments do not test whether LMM-generated pseudo-labels could train a model like BUDDI—that crucial follow-up experiment is absent.

Claim: The framework "improves pose estimates compared to strong baselines that do not use contact supervision" (Introduction). This claim is well-supported across datasets and metrics. The improvement is consistent in PCC (Tables 1, 2, 3) and mixed in PA-MPJPE (clear on Hi4D and FlickrCI3D, absent or slightly negative on CHI3D and MOYO). The PA-MPJPE degradation on CHI3D is attributed to the Heuristic backoff, not to ProsePose being harmful when active—the subset analysis supports this interpretation, but it also means that on 52% of CHI3D examples, the method provides no benefit over the Heuristic (since those examples fall back). The high backoff rates on CHI3D (52%) and MOYO (57%) mean the method is inactive on roughly half of the test examples, and the overall gains come disproportionately from the subset where the LMM is confident. This is a legitimate engineering solution (graceful degradation), but it means the method's impact is conditional on LMM confidence, which is itself influenced by dataset characteristics (motion-capture lab settings vs. in-the-wild photos, camera angle, pose complexity).

Underexplored aspects of the experimental design:

• Single LMM family for main results: The primary experiments use GPT4-V, a proprietary model with unknown training data and architecture. While GPT-4o and LLaVA ablations show the framework works with other LMMs, the paper does not report what fraction of GPT4-V's training data might overlap with the test datasets—a concern for any zero-shot evaluation using web-trained models. If GPT4-V was trained on images from Flickr or motion capture datasets similar to those in the evaluation, the "zero-shot" claim weakens. The paper does not address this.

• Small test sets with no uncertainty quantification: Hi4D has 241 test images, MOYO has 76, CHI3D has 431. These are small enough that individual examples can meaningfully shift aggregate metrics. The paper reports no confidence intervals, standard deviations, or statistical tests. A paired bootstrap or permutation test comparing ProsePose to the Heuristic per-example would clarify whether the reported improvements are statistically reliable or within sampling noise. The FlickrCI3D test set (1403 images) is larger and thus more reliable, but the paper treats all datasets as equally definitive.

• Dataset-specific hyperparameters: The backoff threshold $t$ differs per dataset (2 for CHI3D, $N$ for others), the frequency filter $f$ differs (1 for two-person, 10 for one-person), the LMM loss weight $\lambda_{\text{LMM}}$ differs (1000 for two-person, 10000 for one-person), the GMM prior is removed for yoga, and the initial pose loss weight is doubled for yoga. Each of these was tuned on validation sets. This is standard practice, but it means the reported numbers reflect dataset-specific optimization that limits the strength of cross-dataset comparisons and raises questions about how the method would perform on a new dataset without validation-set tuning.

• Missing ablation: effects of cropping and segmentation: The image preprocessing (cropping to bounding boxes, masking extraneous people with Segment Anything) is described as important for "indicating which individuals to focus on" (Section 3.2), but no experiment reports performance without these steps. Given that cropping removes context (the surrounding scene, other people), there is a plausible tradeoff between focus and information loss that goes unevaluated.

• Missing ablation: vertex subsampling for distance computation: The minimum-distance computation uses random vertex subsampling for tractability (Section 3.2), but the subsample size is not specified and no experiment tests sensitivity to this parameter. If the subsample is too small, the minimum distance estimate becomes noisy, potentially affecting optimization gradients.

• No experiment on generalizing beyond contact: The paper frames contact as one instance of LMMs providing structured constraints for 3D vision, but all experiments are on contact-aware pose estimation. There is no demonstration—even qualitative—of the framework applied to a different constraint type (e.g., "person is sitting on a chair," "hand is above head," "feet are on the ground"), which would substantiate the broader claim that "this work provides evidence that LMMs are promising tools for 3D pose estimation, which likely has implications beyond touch" (Section 5).

• The Heuristic baseline is relatively weak: The Heuristic minimizes the overall minimum distance between the two meshes—it pulls the bodies together but does not specify where they should touch. This is a low bar for contact-aware optimization. A stronger unsupervised baseline would incorporate symmetry heuristics (e.g., if hands are close in 2D, pull the corresponding 3D hand regions together) or learned contact priors from non-contact-annotated interaction data. The paper's choice of baseline is inherited from BUDDI [34], but it means the measured improvement from LMM constraints is partly improvement over a deliberately weak contact model.

• No direct comparison to other language-pose methods: The paper distinguishes itself from PoseScript [7], PoseFix [8], and PoseGPT [9] conceptually, but does not implement or evaluate any of them as baselines—even on the single-person MOYO dataset where they would be applicable. This makes it difficult to assess whether the LMM-as-zero-shot-annotator approach is better or worse than methods that train on paired language-pose data (even if those methods use a different type of language supervision).

• Camera angle sensitivity (Appendix 8.2.4): The paper reports that F1 varies from 0.31 to 0.42 across camera angles on CHI3D, with front and back cameras scoring below 0.37. This sensitivity is noted but not systematically analyzed or mitigated—the paper does not report whether camera angle is a confound in the other datasets or whether certain camera angles disproportionately trigger the backoff mechanism.

What the experiments demonstrate vs. what they claim: The experiments convincingly demonstrate that (1) pretrained LMMs can extract structured contact constraints from images without fine-tuning, (2) those constraints, when converted to differentiable losses and combined with standard pose optimization, improve contact metrics over a heuristic baseline, and (3) the improvement is robust to LMM choice, prompt variations, and dataset characteristics. What the experiments do not demonstrate is that LMM-derived constraints can substitute for supervised contact data in training downstream models, that the framework generalizes to constraint types beyond contact, or that the improvements are practically sufficient for applications requiring highly accurate contact reconstruction. The paper's claims in Section 5 are appropriately measured ("LMMs may be useful in creating larger datasets," "promising tools," "likely has implications beyond touch"), and the experimental evidence supports these measured claims. The stronger language in the abstract and introduction ("compelling reconstructions," "powerful tools," "accurately capturing the semantics") somewhat overstates what the quantitative metrics establish.

6. Limitations and Trade-offs

LMM Failures on Chirality and Spatial Reasoning: A Hard Ceiling on Constraint Accuracy

The assumption or constraint. The entire ProsePose framework depends on the LMM's ability to produce spatially accurate, semantically correct contact constraints. The paper identifies a specific, pervasive failure mode: LMMs cannot reliably distinguish left from right limbs. Section 4.3 states this explicitly:

"prompting the LMM for left/right labels sometimes leads to worse results, suggesting that LMMs struggle with disambiguating chirality."

The prompt design works around this by instructing the LMM not to specify chirality (Section 3.2), pushing the ambiguity into the optimization stage. But the underlying problem is broader than chirality—Appendix 8.4 shows LMMs hallucinating contacts that are semantically plausible for the pose category but factually wrong for the specific image (e.g., predicting "hand touches foot" for a yoga pose where the hand actually touches the knee, Figure 10, example c).

The consequence. Inaccurate LMM constraints directly mislead the optimization. The loss function $\mathcal{L}_{\text{LMM}}$ pulls the specified body regions together regardless of whether they should actually touch. A hallucinated constraint like "hand, head" when the hand is near the waist creates a spurious force that can degrade pose quality—the hand may be pulled toward the head at the expense of 2D keypoint consistency and natural pose priors. Even when constraints are correct in content but wrong in chirality, the minimum-over-assignments procedure may select the geometrically closest but physically incorrect limb pairing, producing a pose where the right hand touches the left shoulder when the image shows the left hand touching the right shoulder. The paper's multi-sample averaging only dilutes these errors, not eliminates them—if 5 of 20 samples hallucinate the same incorrect constraint, the spurious signal is 25% as strong as a true constraint voted unanimously.

What evidence exists in the paper. Table 5 quantifies the impact: asking for left/right labels degrades FlickrCI3D PA-MPJPE from 58 to 62–65 mm, and the captioned approach (which forces the LMM to commit to specific spatial relationships) also underperforms the default prompt. The F1 scores in Tables 1 and 3 reveal the scale of the underlying constraint accuracy problem: F1 = 24 on Hi4D, 13 on FlickrCI3D, 23 on CHI3D, and 25 on MOYO—meaning the LMM correctly identifies only 13–25% of ground-truth coarse contact pairs (considering both precision and recall). The "Oracle" row in Table 1 demonstrates the ceiling: with perfect coarse contacts and correct chirality, Hi4D PA-MPJPE would drop from 93 to 81 mm—a 12 mm gap entirely attributable to LMM prediction errors. Appendix 8.4 (Figure 10) provides qualitative examples of hallucination and chirality failures. Appendix 8.2.4 quantifies camera-angle sensitivity: F1 varies from 0.31 to 0.42 across CHI3D camera angles, with front and back cameras performing worst.

Mitigation status. The paper partially mitigates this through multi-sample averaging (dilution), frequency filtering (threshold $f$ removes rare constraints), and the backoff mechanism (abstaining entirely when the LMM is uncertain, triggered for 5–57% of examples depending on the dataset). However, none of these address the root cause—the LMM's fundamental inability to perform the fine-grained spatial reasoning required for precise contact annotation. The paper does not propose any architectural improvement to the LMM, any fine-tuning strategy to improve contact prediction, or any verification mechanism to filter incorrect constraints beyond simple frequency thresholding. Section 4.3 acknowledges this as a limitation and suggests future work on "eliciting more fine-grained constraints from an LMM," but provides no concrete pathway. The chirality issue is specifically called out: "Improving this approach depends in large part on correctly identifying limbs as left/right."

High Backoff Rates: The Method Is Inactive on a Large Fraction of Examples

The assumption or constraint. ProsePose includes a backoff mechanism that deactivates $\mathcal{L}_{\text{LMM}}$ and falls back to a heuristic baseline when the LMM produces too many empty constraint sets (Section 3.2). This is architecturally necessary—without it, the system would optimize against absent or unreliable constraints—but it means the method's contribution is conditional on LMM confidence. The paper reports specific backoff rates in Section 4.1 and Section 4.2:

"The backoff method (which is the heuristic) is used in 13/241 Hi4D examples, 106/1403 Flickr examples, and 224/431 CHI3D examples."

"The backoff method is used in 43/76 examples" (MOYO, Section 4.2).

The consequence. On Hi4D and FlickrCI3D (backoff rates of 5.4% and 7.6%), the method is almost always active, and the aggregate improvements reflect genuine LMM contribution. On CHI3D (52.0% backoff) and MOYO (56.6% backoff), the method is inactive on roughly half of the test examples. On those examples, ProsePose performs identically to the Heuristic or HMR2+opt baseline—it provides no benefit whatsoever. The reported aggregate metrics on CHI3D and MOYO are therefore averages over a mixture distribution: the LMM improves performance on the ~48% of examples where it is active, but the ~52% where it falls back dilute the average. This means that for a practitioner deploying ProsePose on a new dataset, the expected improvement depends critically on the LMM's confidence rate on that dataset, which is unknown a priori and varies substantially (from 5% to 57% backoff across the four test datasets). The paper's PA-MPJPE on CHI3D (100 mm) is worse than BEV (96 mm) despite ProsePose being better than BEV on the active subset (86 vs. 87 mm)—meaning the backoff mechanism, while preventing degradation relative to the Heuristic, can still produce aggregate performance worse than the raw regressor if the baseline heuristic underperforms the regressor.

What evidence exists in the paper. The paper explicitly reports backoff counts for each dataset (Section 4.1 for two-person, Section 4.2 for MOYO) and provides a subset analysis for CHI3D:

"On the subset of images where we do not default to the heuristic (i.e. on images where GPT4-V predicts enough non-empty constraint sets), the PA-MPJPE for ProsePose and BEV is 86 and 87, respectively."

This is the full extent of the evidence—the paper does not analyze why the backoff rate differs across datasets, what image properties trigger empty LMM responses, or whether the active vs. backoff subsets differ systematically in difficulty, pose type, or image characteristics.

Mitigation status. The paper treats the backoff mechanism as a feature (graceful degradation) rather than a limitation, and does not attempt to reduce backoff rates. There is no analysis of what causes high backoff rates (camera angle? pose complexity? occlusion? lighting?) and no proposed strategy for converting borderline uncertain cases into confident predictions. The threshold $t$ is tuned per-dataset on validation sets (2 for CHI3D, $N$ for others), which means even the decision boundary for backoff requires dataset-specific calibration—a practitioner with a new dataset would need a validation set to set this hyperparameter. The paper does not suggest future work on reducing backoff rates or predicting LMM confidence without the expense of full sampling.

Coarse Region Granularity: Constraints Are Too Imprecise for Fine-Grained Contact

The assumption or constraint. The paper maps LMM natural language output to a vocabulary of 14–16 coarse body-part regions (Section 3.1, Appendix 7.2). The "arm" region, for example, is the union of all fine-grained upper-arm, lower-arm, and elbow regions (front and back). A constraint like ("arm", "back") tells the optimizer that some part of the arm should touch some part of the back—but not whether the contact is at the upper arm, forearm, or hand, nor whether it is on the upper back, lower back, or shoulder blade. The paper acknowledges this explicitly in Section 3.1:

"since our constraints are specified by a LMM trained on natural language, the referenced body parts are often coarser in granularity."

The consequence. The minimum-distance loss (Equation 2) pulls the closest vertices of the two regions together, but this provides no information about where on each region the contact should occur. The optimizer can satisfy ("arm", "back") by bringing any point on the arm to any point on the back—possibly the elbow to the shoulder blade in a pose where the actual contact is the hand on the lower back. This ambiguity limits the precision of the pose refinement: even with perfectly accurate coarse constraints from the LMM and perfect optimization, the resulting pose may have contact at the wrong anatomical locations within the named regions. The PCC metric (Section 4, Tables 1–3) is computed on 75 fine-grained regions, and improvements in coarse contact do not guarantee improvements at the fine-grained level. The gap between the Oracle (perfect coarse contacts, correct chirality) and perfect pose estimation reflects this: even with ideal constraints, PA-MPJPE on Hi4D is 81 mm (Table 1)—the coarse granularity is a fundamental information bottleneck that no amount of LMM accuracy or optimization quality can overcome.

What evidence exists in the paper. The PCC results at specific radii (Table 2) indirectly demonstrate this. ProsePose improves PCC across all radii, but even at 15 mm (a relatively generous contact threshold), it achieves only 57.1 PCC on FlickrCI3D and 52.5 on CHI3D—meaning 43–48% of fine-grained contact points are still missed. The Oracle coarse contacts achieve 100 F1 but only 86 PA-MPJPE on Hi4D (Table 1), showing that even perfect constraints at the coarse level leave substantial pose error. The paper does not ablate region granularity—there is no experiment with finer or coarser region sets to quantify the granularity-accuracy tradeoff.

Mitigation status. The paper acknowledges this limitation in Section 4.3: "Another limitation is the use of coarse regions. Future work could improve by eliciting more fine-grained constraints from an LMM." However, no concrete approach is proposed. The paper's prompt design (Section 7.1) provides a fixed list of coarse region names to the LMM, explicitly constraining the vocabulary to this granularity. Whether LMMs can produce finer-grained constraints (e.g., "upper arm," "forearm," "lower back") with acceptable accuracy is untested—it may be that the coarse vocabulary is not just a design choice but a necessary compromise because LMM accuracy degrades at finer granularity, as it does with chirality. The paper does not explore this tension.

No Quantification of the Annotation-Cost Savings: The Central Scalability Claim Is Unmeasured

The assumption or constraint. The paper's core motivation is that LMMs can substitute for expensive manual contact annotations. The abstract states the method offers "a scalable alternative to traditional methods that rely on human annotations or motion capture data," and Section 5 concludes that "LMMs may be useful in creating larger datasets with contact annotations, which are otherwise expensive." This implies that using ProsePose to generate pseudo-labels for training a supervised method (like BUDDI) would be cheaper or faster than collecting human annotations, while achieving comparable downstream performance.

The consequence. The paper provides no evidence for this claim. There is no experiment training BUDDI (or any supervised method) on ProsePose-generated pseudo-labels and comparing to BUDDI trained on human annotations. The ProsePose framework costs approximately 16 seconds of LMM API time plus 25–89 seconds of optimization per image (Appendix 8.2.3), and the resulting poses have F1 scores of 13–25 on contact prediction (Tables 1, 3). Whether these noisy pseudo-labels are sufficient to train a model that approaches the performance of one trained on clean human annotations is an open question—the paper demonstrates that LMM constraints improve optimization-based refinement when used as a loss term, but this is a different use case from generating training labels for a learned prior. The optimization framework benefits from LMM constraints even when they are noisy because the multi-sample averaging and auxiliary losses provide regularization. A supervised model trained on noisy pseudo-labels has no such protection—it would learn to replicate the LMM's systematic errors (chirality confusion, coarse granularity, camera-angle sensitivity). Without a training experiment, the scalability claim remains an untested hypothesis.

What evidence exists in the paper. The paper compares ProsePose (zero-shot LMM) to BUDDI (trained on human annotations) in Tables 1–2, showing that BUDDI outperforms ProsePose across all metrics. The gap—BUDDI's advantage—represents the value of clean human annotations over LMM pseudo-labels in the optimization setting. The BUDDI+ProsePose results (adding LMM loss to BUDDI's optimization) show small PCC improvements (1.3 points on FlickrCI3D, 0.8 on CHI3D), suggesting that LMM constraints provide complementary information even when a strong learned prior exists. But no experiment tests the central use case: generating a large dataset with LMM pseudo-labels and training BUDDI on it. The paper does not report the cost of LMM API calls (which are commercial and priced per token), the cost of human annotation for the datasets used, or any cost-benefit analysis.

Mitigation status. Not addressed. The paper does not acknowledge this as a gap between its motivation and its experiments. The conclusion (Section 5) frames the contribution as evidence that LMMs "may be useful" for dataset creation, which is appropriately cautious, but the abstract and introduction use stronger language ("scalable alternative," "decrease the cost") that implies demonstrated cost savings. No future work is proposed on training with LMM pseudo-labels, cost analysis, or annotation quality assessment.

Single LMM Family with Unknown Training Data: The Zero-Shot Claim Has an Unquantified Leakage Risk

The assumption or constraint. The primary experiments use GPT4-V, a proprietary model whose training data, architecture, and scale are not publicly documented. The paper claims that ProsePose leverages "the existing knowledge in an LMM to reason about pose from a given image" without any task-specific training (Section 2), which is a zero-shot generalization claim. This claim depends on the LMM's contact knowledge arising from general vision-language pretraining, not from exposure to the specific test datasets or closely related contact-annotated data.

The consequence. If GPT4-V was trained on images from Hi4D, CHI3D, FlickrCI3D, or MOYO—or on similar motion-capture and interaction datasets, or on web pages that describe and analyze these datasets—then the apparent zero-shot contact prediction capability may partly reflect memorization of the test data rather than generalizable contact understanding. The same concern applies to any images in the test sets that appear in GPT4-V's training web corpus (particularly FlickrCI3D, which is sourced from Flickr—a website likely included in web-scale training data). The paper provides no analysis of data contamination, no attempt to identify whether test images were in the LMM's training set, and no discussion of the implications for its zero-shot claims. For datasets like Hi4D and CHI3D (motion capture in controlled lab settings), the risk of direct image overlap is low, but the risk of the LMM having been trained on research papers, dataset documentation, or blog posts describing these datasets—and thus having prior knowledge of the types of poses and contacts they contain—is non-trivial for a model trained on a significant fraction of the public web.

What evidence exists in the paper. The paper conducts ablation experiments with GPT-4o and LLaVA-NeXT 34B (Tables 5, 6) and reports that performance is comparable with GPT-4o and worse with LLaVA, which the paper attributes to "the general capabilities of these LMMs" (Appendix 8.2.2). The LLaVA results (Appendix Table 6) are particularly informative: using a much smaller open-source LMM (34B parameters), ProsePose still improves over the heuristic baselines on most metrics, though gains are smaller (e.g., MOYO PCC drops from 87.8 to 85.2, matching HMR2+opt). This suggests that contact knowledge is not unique to GPT4-V and that the zero-shot capability generalizes across LMMs. However, the possibility of test-set contamination applies to LLaVA as well, since it was also trained on web-scale image-text data. Neither LMM's training data is analyzed for overlap with the test datasets.

Mitigation status. Not addressed. The paper does not mention test-set contamination as a concern, does not attempt to detect whether test images appear in GPT4-V's training data, and does not discuss the implications of proprietary training data for reproducibility or for the strength of the zero-shot claim. The use of GPT-4o and LLaVA as ablation LMMs provides some evidence that the capability is not exclusive to a single model, but it does not address contamination since all three models may share overlapping training corpora. The paper also does not address the reproducibility concern: GPT4-V (gpt-4-vision-preview) is a specific API endpoint that may be deprecated or changed, making exact replication of the paper's results impossible for future researchers without access to the same model version. The code release (https://prosepose.github.io) partially mitigates reproducibility for the optimization components, but the LMM inference—the novel component—is dependent on a commercial API.

Small Test Sets Without Statistical Significance: The Measured Improvements Are Not Characterized for Reliability

The assumption or constraint. The paper evaluates on test sets of 76 (MOYO), 241 (Hi4D), 431 (CHI3D), and 1403 (FlickrCI3D) images. The three smaller datasets are small enough that the reported aggregate metrics could be sensitive to a handful of examples. The paper reports no confidence intervals, standard deviations, standard errors, or statistical significance tests for any comparison. The evaluation protocol (Section 4, Metrics) reports raw averages only.

The consequence. For datasets like MOYO (76 examples), a difference of 2–3 mm PA-MPJPE or 2–3 points PCC could fall within sampling noise. The paper reports ProsePose as achieving 82 mm PA-MPJPE vs. 81 mm for HMR2+opt on MOYO—a 1 mm difference (ProsePose is slightly worse in joint error). Whether this difference is statistically meaningful or attributable to a few outlier examples is unknown. Similarly, on CHI3D, ProsePose achieves 100 mm PA-MPJPE vs. 96 mm for BEV—a 4 mm degradation. Without uncertainty quantification, a practitioner cannot assess whether deploying ProsePose on a similar-sized dataset would reliably improve or possibly degrade performance. The subset analysis on CHI3D (86 mm for ProsePose vs. 87 mm for BEV on the 207 active examples) is a 1 mm difference on an even smaller sample. The paper's key efficiency claim—that ProsePose "reduces 85% of the gap in PA-MPJPE between Heuristic and the fully supervised BUDDI" on Hi4D—is a point estimate on 241 examples; the uncertainty around that 85% figure is unknown.

What evidence exists in the paper. The paper reports the number of examples in each dataset (Section 4.1, 4.2) and the backoff counts, which further reduce effective sample sizes for the active ProsePose condition. The two-fold cross-validation protocol used in the reference example's compute-optimal scaling analysis (for strategy selection on small validation folds) is absent here—hyperparameters are chosen on separate validation sets, but test-set evaluation uses the full test set without cross-validation or resampling. The paper does not discuss statistical power, minimum detectable effect sizes, or the relationship between dataset size and the reliability of the reported improvements.

Mitigation status. Not addressed. The paper includes no error bars in any figure, no confidence intervals in any table, and no discussion of statistical significance. This is consistent with common practice in the 3D pose estimation literature—BUDDI [34] and other cited works also report point estimates without uncertainty quantification—but it limits the strength of the comparative claims, particularly on the smaller datasets. The FlickrCI3D results (1403 examples) are more robust by virtue of sample size, but even there, the PCC difference between ProsePose (79.9) and Heuristic (77.8) is 2.1 points, and whether this is a reliable difference is not statistically characterized. The paper's ablation studies (Table 5, Figure 5) are conducted on validation sets of unspecified size but presumably smaller than the test sets, compounding the uncertainty.

7. Implications and Future Directions

How This Work Changes the Landscape

ProsePose introduces a new capability claim for pretrained large multimodal models—that they can serve as zero-shot structured annotators for a fine-grained 3D geometric reasoning task—and in doing so opens a genuinely distinct research direction at the intersection of vision-language models and geometric computer vision. The contribution is not a new state-of-the-art on contact-aware pose estimation (BUDDI outperforms ProsePose on every metric, Table 1) but rather a proof of existence: LMMs, without any task-specific fine-tuning or contact-annotated training data, can extract structured spatial constraints from images at a level of detail sufficient to meaningfully improve 3D pose optimization in a domain where annotation has historically been the binding constraint on progress.

This shifts the conversation in several concrete ways:

From "can LMMs do precise spatial reasoning?" to "how do we operationalize coarse spatial reasoning from LMMs?" The paper demonstrates that LMMs cannot reliably perform fine-grained spatial reasoning—they fail at chirality (Table 5, Section 4.3), produce hallucinated contact constraints (Appendix 8.4, Figure 10), and operate at a coarse region granularity that is substantially less precise than what supervised methods require (75 fine-grained regions used in PCC computation vs. 14–16 coarse regions used in the LMM vocabulary, Appendix Figure 9 vs. Figure 7). Yet despite these failures, the coarse constraints remain useful when embedded in a classical optimization framework with multi-sample averaging and graceful backoff. This suggests that the research question should not be "can we make LMMs as precise as supervised annotators?" but rather "how do we build systems that extract what LMMs can provide and compensate for what they cannot?" The paper's architecture—LMM for semantic reasoning, classical optimization for geometric precision, uncertainty-based backoff for safety—is a design template for this approach.

Reconciliation of contradictory impulses in the language-pose literature. Prior work on language and pose split into two camps: (1) methods that train on paired language-pose data (PoseScript [7], PoseFix [8], PoseGPT [9]) and thus require expensive annotations, and (2) methods that use text-only language models for pose refinement (Wang et al., 2022 [46]) but rely on pre-existing pose-action databases. ProsePose demonstrates a third path: using pretrained vision-language models that require neither paired language-pose training nor pose databases. This resolves a latent tension—the field has known that language contains useful pose information, but has been constrained by the annotation bottleneck in extracting it. The paper shows the bottleneck is not fundamental; it is a property of the training paradigm, and the emergence of pretrained LMMs bypasses it for at least some contact-related reasoning.

Reframing the scalability argument for contact-aware pose estimation. The dominant narrative in the contact pose literature is that progress requires more and better annotated data—CHI3D, FlickrCI3D, BUDDI's training data. ProsePose reframes this: if LMMs can produce plausible contact constraints at scale (16 seconds per image for GPT-4o sampling, Appendix 8.2.3), then the bottleneck shifts from acquiring annotations to verifying and using them. A pipeline that generates LMM pseudo-labels for millions of web images, filters them by multi-sample consistency, and trains a BUDDI-like prior on the result is now a conceptually viable (if unproven) path to scaling contact-aware pose estimation beyond small motion-capture datasets. The paper does not achieve this—it only demonstrates the first step—but the demonstration makes the downstream vision concrete where it was previously speculative.

Making certain research directions more attractive. The paper strongly suggests that improving LMM spatial reasoning—particularly chirality disambiguation and finer-grained region identification—would yield direct improvements in pose estimation. The Oracle row in Table 1 shows that perfect coarse contacts would improve Hi4D PA-MPJPE from 93 mm to 81 mm—a 12 mm gap that represents the headroom from better constraint prediction. A LMM that could reliably produce constraints at the granularity of the 75 fine-grained regions (Appendix Figure 9) rather than the 14–16 coarse regions would shift the Oracle ceiling even further. This makes LMM spatial reasoning benchmarks (chirality classification, fine-grained body part identification) directly relevant to the pose estimation community in a way they were not before.

Making certain research directions less attractive. The paper indirectly argues against end-to-end LMM-based pose regression in the near term. The chirality failures (Figure 10a, 12, 13) and hallucination patterns (Appendix 8.4) suggest that LMMs lack the metric precision required for direct 3D output. The paper's hybrid design—LMM for constraints, classical optimization for geometry—outperforms the alternative paradigm of training an LMM to output pose parameters directly (as PoseGPT attempted). This suggests that for the foreseeable relevant timeline, research effort is better spent on improving the interface between LMMs and geometric optimizers (better prompts, better constraint formalisms, better region vocabularies) than on trying to make LMMs output precise spatial coordinates.

The scope of the shift. This is not a paradigm shift. It is a capability demonstration and architectural template that opens a new subfield—"LMM-guided geometric optimization"—within pose estimation. The paper's results are modest in absolute terms (ProsePose does not beat supervised methods, backoff rates reach 57% on some datasets), and the work does not resolve the fundamental limitations it identifies (chirality, granularity, hallucination). But by showing that even with these limitations LMM constraints provide measurable value, and by providing a complete, replicable pipeline for extracting and operationalizing those constraints, it creates a foundation that subsequent work can improve incrementally. The paper's value is as a first demonstration of feasibility for a new class of methods, not as a solution to contact-aware pose estimation.

Follow-Up Research This Work Enables

Training BUDDI on LMM-generated pseudo-labels and measuring the annotation-cost tradeoff. The paper's central scalability claim—that LMMs can reduce the cost of creating contact-annotated datasets—is stated but completely untested. The most direct follow-up would be: use ProsePose to generate coarse contact pseudo-labels (with multi-sample frequency filtering to select high-confidence predictions) for a large corpus of unlabeled interaction images, train BUDDI's diffusion prior on those pseudo-labels, and compare to BUDDI trained on the same number of human-annotated examples. The key measurement is the pseudo-label efficiency curve: how many LMM-labeled images are required to match the performance of K human-labeled images, and what is the relative cost in dollars and time? This experiment would directly test the paper's motivating claim and quantify the annotation-cost savings. A strong negative result—e.g., LMM pseudo-labels being too noisy to train a prior that generalizes beyond the LMM's own systematic errors—would be equally valuable, as it would reveal that ProsePose's optimization-time benefits (averaging across samples, regularization from auxiliary losses) do not transfer to the training-data-generation use case.

Fine-grained region prompting with chirality-aware chain-of-thought. The paper identifies coarse region granularity and chirality as two major LMM failure modes (Section 4.3) but does not test whether the LMM can produce finer-grained constraints under different prompting strategies. A targeted follow-up would design a hierarchical prompting protocol: first, ask the LMM for coarse contact regions as in ProsePose; then, for each predicted coarse contact pair, ask the LMM to localize the contact to sub-regions (e.g., for "hand, back," ask "which part of the hand and which part of the back?"). Similarly, test whether a two-stage chirality prompt—first predict contacts without left/right, then ask a separate LMM call (or the same LMM with the constrained image region) to disambiguate chirality for each specific contact—improves over the single-stage approach ablated in Table 5. The experiment would measure F1, PA-MPJPE, and PCC when using the resulting fine-grained chiral constraints against the current coarse no-chirality baseline, ideally on the same test sets to enable direct comparison. A positive result would demonstrate that LMM limitations are partly prompt-engineering limitations rather than fundamental capability gaps, while a negative result would strengthen the paper's implicit claim that LMMs genuinely cannot perform this level of spatial reasoning regardless of how the question is framed.

Extending the constraint vocabulary beyond contact to spatial relationships. The paper frames contact as one instance of a broader class of spatial constraints that LMMs could articulate—but never demonstrates the generalization. A direct extension would define a richer constraint vocabulary: relative position ("Person 1 is to the left of Person 2"), body-part orientation ("Person 1's face is turned toward Person 2"), support relationships ("Person 1 is sitting on a bench"), and inter-object constraints ("right hand is on the basketball"). Each constraint type would require a new loss function (e.g., a relative-position loss based on bounding box overlap or center-of-mass vector comparison), but the ProsePose architecture—LMM prompt → structured parse → differentiable loss → optimization—remains identical. The evaluation would use existing datasets with relevant ground truth (e.g., PROX for human-scene interaction, Human3.6M for relative pose, or a custom dataset with spatial relationship annotations), measuring whether the generalized constraints improve pose estimation accuracy for the relevant relationship types. This would test the paper's concluding claim that the work "likely has implications beyond touch" (Section 5) and transform ProsePose from a contact-specific tool into a general framework for LMM-guided 3D scene understanding.

Why-causes-what analysis of LMM failures: dataset bias, camera angle, and pose complexity. The paper observes that F1 varies from 0.31 to 0.42 across camera angles on CHI3D (Appendix 8.2.4) and that backoff rates range from 5.4% (Hi4D) to 56.6% (MOYO) (Sections 4.1, 4.2), but provides no analysis of why. A systematic diagnostic study would categorize LMM predictions (and failures) along multiple axes: camera angle (front, side, back, oblique), pose type (hug, handshake, dance, tackle), occlusion severity (fraction of keypoints undetected by OpenPose/ViTPose), body-part visibility (which coarse regions are visible in the image), and dataset domain (motion capture vs. in-the-wild). For each axis, the study would compute per-category F1, backoff rate, and PA-MPJPE improvement over the Heuristic, identifying the conditions under which LMM contact prediction is reliable vs. unreliable. This would serve two purposes: (1) it would guide practitioners on when ProsePose is likely to help (e.g., "use it for side-view in-the-wild images of standing interactions; don't use it for front-view motion-capture images of seated poses"), and (2) it would provide a diagnostic benchmark for future LMMs—a new model could be evaluated on whether it improves performance specifically on the failure categories identified in this analysis, rather than on aggregate metrics that are dominated by easy examples.

Active difficulty estimation and adaptive sampling. The paper uses a fixed N = 20 samples for all images and a fixed backoff threshold t, but Figure 5 shows diminishing returns beyond ~15 samples—implying that some images could use fewer samples without performance loss. An adaptive sampling strategy would: (1) begin with a small number of LMM samples (e.g., N = 5), (2) estimate constraint-set consistency (e.g., the fraction of samples that produce non-empty sets, or the Jaccard similarity between sample constraint sets), (3) if consistency is high (the LMM is confident and self-consistent), stop sampling; if consistency is low, draw additional samples up to a maximum N_max, (4) if consistency remains below a threshold after N_max samples, back off. The evaluation would measure the average number of LMM samples used per image against the PA-MPJPE/PCC achieved, comparing to the fixed N = 20 baseline. A successful adaptive strategy would maintain accuracy while substantially reducing API cost, particularly on "easy" images where the LMM is already confident—which, based on the high backoff rates for MOYO and CHI3D, might be a minority of images, but the cost savings on that minority could be significant in large-scale deployment. This directly addresses the unexplored tradeoff between difficulty estimation cost and optimization benefit that the paper identifies in Section 3.2.

Cross-dataset generalization of the prompt and hyperparameters. The paper tunes prompt design, region vocabulary, f, t, and λ_LMM on each dataset's validation set (Section 4, Implementation details; Appendix Section 7.5). A robustness stress-test would fix all hyperparameters and the prompt to the values optimized on one dataset (say, FlickrCI3D) and evaluate on the other three datasets (Hi4D, CHI3D, MOYO) without any per-dataset tuning. This tests whether the method truly works "out of the box" or requires dataset-specific calibration. A strong result (performance degrades minimally) would support the paper's implicit claim of generality and make ProsePose practically deployable without validation sets. A weak result (large degradation on transfer, particularly on MOYO where the region vocabulary and GMM prior differ) would reveal that the method is more domain-specific than the paper suggests and that practical deployment requires dataset-specific calibration—a finding that would motivate research on automated hyperparameter selection or learning-based constraint-to-loss mapping that generalizes across domains.

Practical Applications and Downstream Use Cases

Bootstrapping contact-annotated datasets for training supervised priors. The most direct application of ProsePose is generating pseudo-labels at scale: for a large corpus of web images depicting human interaction (e.g., sports photography, dance performances, social media photos of people hugging or shaking hands), run ProsePose to produce contact-constrained 3D pose estimates, filter by multi-sample consistency (keeping only images where the LMM produces consistent non-empty constraint sets across many samples), and use the resulting pose+contact pairs as training data for a model like BUDDI. The paper's numbers provide a rough cost estimate: 16 seconds of LMM API time per image for 20 GPT-4o samples (Appendix 8.2.3), plus 25–89 seconds of optimization. At scale, this could generate tens of thousands of pseudo-labeled contact images for a few hundred dollars of API cost—orders of magnitude cheaper than motion capture or manual annotation. The key unknown (which the follow-up experiment above would resolve) is whether the resulting pseudo-labels are accurate enough to train a prior that outperforms the LMM itself—a critical threshold for this application to be viable.

Improving markerless motion capture for interaction-heavy scenarios. Motion capture systems that rely on multi-view 2D keypoint triangulation (e.g., in sports biomechanics or film production) frequently fail during close physical contact because occluded keypoints produce noisy or missing 3D joint estimates. A production pipeline could integrate ProsePose as a post-processing step: after initial 3D reconstruction from multi-view keypoints, run the LMM constraint generation on each view independently (or on a representative view), average the resulting constraints across views for robustness, and use the constrained optimization to refine poses specifically at contact frames. The quantitative gains on Hi4D (which IS a markerless motion capture dataset with controlled multi-view capture) are directly relevant: ProsePose reduces PA-MPJPE from 116 mm (Heuristic) to 93 mm on contact frames (Table 1), with the largest improvements at tight contact radii (Table 2). For a biomechanics application where correctly measuring inter-person distance at contact is critical (e.g., analyzing tackle safety in rugby), the PCC improvements at 5–10 mm radii could directly translate to more accurate contact force estimates.

Self-contact analysis for physical therapy and yoga instruction. The MOYO results (Table 3) demonstrate that ProsePose improves self-contact accuracy in yoga poses: PCC improves from 85.2 (HMR2+opt) to 87.8, with a particularly large gain at the tight 5 mm radius (54.2 vs. 47.7, a 6.5-point absolute improvement). For applications in physical therapy—where a system evaluates whether a patient is correctly touching their hand to their foot, or whether their arm is properly positioned against their back—these fine-grained contact improvements could distinguish between correct and incorrect form. Deploying ProsePose on monocular video from a smartphone camera (rather than the motion-capture MOYO dataset) would provide a low-cost at-home assessment tool: the patient records themselves performing a prescribed stretch, ProsePose extracts contact constraints, and the optimization verifies whether the correct body parts are within a clinically meaningful distance threshold. The 56.6% backoff rate on MOYO (43 of 76 examples) is a concern—roughly half of yoga poses would not benefit—but specific poses where the LMM is confident (the 33 active examples) could be prioritized in a curated exercise set.

Human-robot interaction and social robot behavior analysis. For robots that need to understand human social touch (e.g., a care robot determining whether a person is being hugged, pushed, or guided by the arm), contact configuration is a critical perceptual signal that is poorly captured by standard pose estimators. ProsePose could run on the robot's camera feed, generating contact constraints in real time (16 seconds for LMM sampling would need optimization, possibly via a faster inference engine or smaller LMM, but the architectural pattern transfers). The robot would receive not just 3D joint positions (from which it must infer contact) but explicit contact region pairs—"Person 1's hand is on Person 2's shoulder"—that directly inform social state estimation. The FlickrCI3D results, which show ProsePose working on in-the-wild photographs with diverse interactions (dancing, tackling, holding hands, Figure 3), suggest that the contact predictions generalize to unconstrained social scenarios rather than being limited to lab settings. The key limitation for real-time use is the LMM sampling latency; a distilled or smaller LMM fine-tuned specifically for contact constraint prediction (using ProsePose's GPT-4V outputs as training labels—a self-distillation loop) would be a natural next step for deployment.