OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation

ArXiv: 2511.13655

🎯 Pitch

Self-supervised Earth observation models frequently collapse during training when predicting in latent space, but OlmoEarth fixes this with one simple change: replacing the target encoder with a frozen random linear projection. This single modification—which the authors call Latent MIM Lite—meets the performance of unstable latent methods while remaining as stable as a pixel-space autoencoder, achieving state-of-the-art on the majority of 24 embedding and 29 fine-tuning tasks across a massive evaluation spanning 12 competing foundation models.

---

1. Executive Summary

OlmoEarth proposes a new family of spatio-temporal, multimodal foundation models for Earth observation built around a novel self-supervised training formulation called Latent Masked Image Modeling of Linear, Invariant Token Embeddings (Latent MIM Lite) — a simplification of Latent MIM that replaces the target encoder with a frozen random linear projection to stabilize training — combined with modality-aware masking (reconstructing entirely masked bandsets rather than only randomly masked tokens) and a modality patch discrimination loss (contrasting predicted tokens only against targets from the same bandset to eliminate easy negatives from cross-modality comparisons). Evaluated against 12 other foundation models across 24 embedding tasks and 29 fine-tuning tasks spanning both research benchmarks (GEO-Bench, BreizhCrops, CropHarvest, PASTIS, MADOS, Sen1Floods11) and real-world datasets from 7 partner organizations, OlmoEarth achieves best performance on 15 of 24 kNN/linear probe tasks and 19 of 29 full fine-tuning tasks, establishing a Pareto frontier of performance versus computational efficiency across model sizes from ViT Nano (1.4M parameters) to ViT Large (300M parameters) — with the finding that test-time performance scales reliably only when training stability is preserved through the frozen-target design, as the baseline Latent MIM approach suffered representation collapse during training.

2. Context and Motivation

The Core Problem: Earth Observation Foundation Models Are Unstable and Inaccessible

Earth observation (EO) data — satellite imagery, radar, and derived maps — presents a uniquely challenging combination of properties for machine learning. It is spatial like images, sequential like video or text (time series of observations over months or years), and highly multimodal (different sensors like Sentinel-1 SAR, Sentinel-2 multispectral, Landsat-8 thermal, plus derived products like land cover maps and elevation data). A model that can reason across space, time, and sensor modalities simultaneously would be enormously valuable for applications ranging from deforestation monitoring to crop yield forecasting to disaster response. However, building such models has proven difficult for two distinct reasons the paper identifies.

The first is technical: training instability. Foundation models for EO have increasingly adopted self-supervised learning in latent space — a family of approaches where the model doesn't predict raw pixels but instead predicts representations of those pixels produced by a separate "target encoder." This shift from pixel-space reconstruction (masked autoencoders, or MAE) to latent-space prediction (Latent Masked Image Modeling, or Latent MIM, and related methods like I-JEPA) is motivated by evidence that latent-space objectives produce richer, more transferable features for downstream tasks. However, the paper explicitly documents a critical failure mode with these approaches:

"When attempting to replicate existing work we frequently saw training instability, representation collapse, and models underperforming their stated potential."

The authors observe that the baseline Latent MIM approach — where a learned target encoder produces the latent targets that the model is trained to predict — collapses during training (Table 4, first row: "Full Latent MIM collapsed during training"). This collapse manifests as the model's representations degenerating to a trivial solution rather than learning meaningful features. Other work, such as Mo et al. (2024) and Assran et al. (2023), has documented similar instabilities in latent-space self-supervised learning. The paper frames this as a fundamental tension: MAE-style pixel reconstruction is stable but produces weaker representations, while latent-space methods produce stronger representations but are unstable. There is no established middle ground.

The second problem is practical: inaccessibility. The paper argues that despite promising research results from a growing number of EO foundation models — Anysat, Galileo, CROMA, Panopticon, TerraMind, TESSERA, Prithvi v2, Satlas, Clay, CopernicusFM, and others — adoption for real-world tasks lags behind, "especially in the non-profit sector." The stated barriers are that foundation models are "large, complex to train, and expensive to deploy." Environmental and humanitarian organizations that could benefit enormously from EO modeling (mangrove conservation groups, food security researchers, ecosystem mapping initiatives) lack the GPU infrastructure, deep learning expertise, and data management tooling to operationalize these models. The paper positions this as a "last-mile problem" — the technology exists in research papers but has not been translated into usable tools for the organizations that need it most.

These two problems are connected: if training were stable and reproducible, more organizations could train their own models or fine-tune open ones. If deployment were simpler, organizations could build on existing models rather than starting from scratch. The paper addresses both: the technical contribution (Latent MIM Lite) solves the stability problem, and the platform contribution (OlmoEarth Platform) solves the accessibility problem.

Why This Matters: Real-World Stakes

The paper grounds its motivation in concrete use cases. The partner organizations described in Section 5.1 and Appendix B are not hypothetical — they are actively using EO data for mission-critical work:

• Global Mangrove Watch maps coastal mangrove forests for carbon sequestration monitoring and coastal protection. Their existing approach uses a random forest model achieving 95.3% F1, but only covers about half of relevant coastal regions and operates on a yearly cadence. A foundation model that could be fine-tuned on their data and run monthly or on a rolling basis would dramatically improve both coverage and temporal resolution.
• Global Ecosystem Atlas is building a comprehensive map of world ecosystems, a task requiring labeling tens of thousands of data points and training classification models. The platform has enabled them to label more than 15,000 data points in three months.
• Amazon Conservation monitors forest loss drivers (mining, logging, agriculture) to prioritize enforcement and litigation against illegal deforestation.
• African Wildlife Foundation maps land use and land cover in southern Kenya for conservation planning.
• NASA JPL uses live fuel moisture content predictions to assess wildfire risk.
• CGIAR/IFPRI and NASA Harvest work on crop type mapping for food security.

These organizations share a common profile: they possess domain expertise and sometimes labeled data, but lack the machine learning infrastructure to build and deploy custom models. The paper argues that improving foundation model training stability and providing an end-to-end platform directly serves these use cases by lowering the barrier to entry.

The scale of the data these organizations work with reinforces the practical motivation. The paper's pretraining dataset spans 285,288 locations globally with up to 12 monthly timesteps across multiple sensors, sampled from 120 OpenStreetMap feature categories. Downstream evaluation covers 24 embedding tasks and 29 fine-tuning tasks across 18 research benchmarks and 19 partner datasets — an unusually comprehensive evaluation that reflects the diversity of real-world EO applications.

Prior Approaches and Their Shortcomings

The paper situates itself against a landscape of prior work that has evolved through several phases, each with identifiable limitations.

Early Work: Contrastive Learning

Initial approaches to EO pretraining used contrastive objectives — Tile2Vec (Jean et al., 2019), Seasonal Contrast (Mañas et al., 2021), Geography-Aware Contrastive Learning (Ayush et al., 2021). These methods train models to distinguish between different geographic locations or temporal snapshots. While foundational, contrastive methods require careful negative sampling and struggle to leverage the full richness of multimodal, multi-temporal inputs. They also tend to produce features optimized for image-level similarity rather than the dense, per-pixel understanding needed for segmentation and detection tasks.

The Shift to Masked Modeling

More recently, masked modeling — inspired by BERT in language (Devlin et al., 2018) and MAE in vision (He et al., 2022) — has become the dominant paradigm. Early EO masked models reconstructed raw pixel values: SatMAE (Cong et al., 2022), Scale-MAE (Reed et al., 2023), and Lightweight Transformer (Tseng et al., 2023). These are stable — pixel reconstruction with an L1 or Smooth L1 loss is well-behaved — but the representations they learn tend to be weaker for downstream tasks. The problem is that reconstructing pixels forces the model to spend capacity on low-level details (exact spectral values, sensor noise) rather than semantic content. The paper's own ablation (Table 6, first row) confirms this: an MAE baseline achieves an average rank of 5.1 out of 6 variants across 10 tasks, substantially worse than the final Latent MIM Lite recipe.

The Shift to Latent Space

Following developments in natural imagery (I-JEPA by Assran et al., 2023; DINOv3 by Siméoni et al., 2025; Latent MIM by Wei et al., 2024), EO models have increasingly adopted reconstruction in latent space. Leading models in this category include:

• Anysat (Astruc et al., 2024): A multimodal EO model using latent-space objectives.
• Galileo (Tseng et al., 2025): Uses a FlexiViT-style architecture with variable patch sizes and trains on both observational data and supervised maps. Galileo treats maps as valid model inputs, meaning the encoder must learn to represent map modalities alongside observations — allowing the model to potentially use map information to predict observations during training.
• Panopticon (Waldmann and Shah, 2025): Achieves strong performance on embedding tasks, as shown in the paper's evaluation tables.
• TerraMind (Jakubik et al., 2025): A large-scale generative multimodal model that avoids instability by using a frozen tokenizer — for image modalities, they train a quantized autoencoder and use its encoder as a frozen tokenizer during multimodal masked modeling. This is conceptually similar to the paper's approach but achieved through a different mechanism (vector quantization vs. random projection).

The paper explicitly acknowledges that latent approaches "work well but have documented instabilities," citing Mo et al. (2024) and Assran et al. (2023). The core tension is that the target encoder — the component that produces the latent targets the model tries to predict — must be learned jointly with the main model, and this co-adaptation can lead to collapse: the target encoder and the predictor find a trivial equilibrium where both produce constant or degenerate outputs that minimize the loss without learning useful representations.

Specific Shortcomings of Prior Work

The paper identifies several concrete gaps that prior models fail to address:

1. Training instability in latent-space methods. This is the central technical gap. The paper's own development history (Table 4) shows that "Full Latent MIM" (standard latent-space training with a learned target encoder) achieved only 7.9 mIoU on PASTIS versus 35.2 for Latent MIM Lite with the frozen random projection — and the full Latent MIM approach actually collapsed during training. This is not a marginal improvement but a qualitative difference between a broken model and a working one. Prior work either avoided latent space entirely (staying with pixel reconstruction and accepting weaker features) or attempted to manage instability through careful hyperparameter tuning (which the paper found insufficient for reliable replication).

2. Masking strategies not adapted to multimodal EO data. Standard random masking, which works well for natural images and text, is problematic for multimodal remote sensing because "any token in the input will have many similar tokens either in space, time, or at a different aligned modality." If you mask 50% of Sentinel-2 tokens randomly, the unmasked Sentinel-2 tokens from nearby spatial locations or adjacent timesteps are highly correlated with the masked ones, making the reconstruction task too easy. The paper notes that "random masking is too easy of a task unless you use a very high masking ratio," citing Galileo's use of 90% masking to compensate. Skewed masking ratios create their own problems — the model sees very little context, making the task artificially hard in a different way.

3. Contrastive losses not adapted to multimodal token spaces. Latent MIM's patch discrimination loss contrasts a predicted token against all target tokens in the input. For natural images, this is reasonable because all tokens come from the same distribution (different spatial patches of the same RGB image). For multimodal EO, target tokens come from fundamentally different distributions — Sentinel-2 optical bands, Sentinel-1 SAR backscatter, Landsat thermal bands, elevation data. The paper argues that "tokens from different modalities have very different distributions so distinguishing between them is easy," meaning "a significant amount of the loss comes from these 'easy' negatives." The model expends gradient signal learning to distinguish between modalities (a trivial task) rather than learning the harder, more useful within-modality distinctions.

4. Supervised data integration requires architecture changes. Models like Galileo and TerraMind train on both observations and supervised maps by treating maps as valid encoder inputs. This means the encoder must learn to represent map modalities during training and potentially uses map information to predict observations. The paper argues this complicates the learning task and may not be optimal: "we theorize that our approach simplifies learning for the encoder while maintaining the benefits of training with supervised data." The paper's approach — always masking maps entirely from the encoder input and using them only as decoder targets — means the encoder never sees map data during training, eliminating the need to model it while still benefiting from the supervised signal through the decoder loss.

5. Global representation methods are inconsistent. For tasks requiring image-level understanding (classification), some foundation models use a dedicated <CLASS> token. Others pool tokens across the entire input. The paper identifies a subtle problem with pooling: "tokens encoded from the same modality share semantics but tokens from different modalities may look very different from each other. We want to be able to average tokens from all modalities together and get a sensible global representation." Without explicit encouragement, tokens from different modalities may inhabit different regions of the embedding space, making naive pooling produce incoherent global representations. The paper addresses this with an instance-level contrastive loss that explicitly encourages cross-modal representation alignment.

6. Evaluation is fragmented and incomparable. The paper points out that "there is no standard evaluation suite for remote sensing models. While there are some established standard practices [reed2023scale, fuller2024croma, tseng2025galileo], they are not always followed." Different papers evaluate on different benchmarks, use different training recipes, and report different metrics, making it impossible to compare models fairly. The paper addresses this by running a comprehensive evaluation of 12 other models within a unified framework — same training recipes, same hyperparameter sweeps, same metrics — enabling for the first time a genuinely comparable assessment across the field.

7. Precomputed embeddings sacrifice flexibility. AlphaEarth Foundations (Brown et al., 2025) and TESSERA (Feng et al., 2025) offer precomputed global embeddings as an alternative to full model access. While more accessible (no GPU needed for inference), this approach has significant limitations: embeddings are annualized (making sub-annual or real-time predictions impossible), the embedding model itself cannot be fine-tuned, and retrieving embeddings from cloud storage can be expensive (the paper reports $290 to export embeddings for a 100K-crop dataset from Google Earth Engine). Table 7 shows that while AlphaEarth embeddings are competitive with frozen OlmoEarth embeddings, full fine-tuning of OlmoEarth substantially outperforms both — a capability that is impossible with precomputed embeddings only.

How This Paper Positions Itself

The paper positions OlmoEarth as addressing both halves of the problem: the technical instability that makes latent-space training unreliable, and the practical inaccessibility that prevents real-world adoption.

On the technical side, the paper proposes Latent MIM Lite as a deliberately simple modification that breaks the feedback loop responsible for collapse — replace the learned target encoder with a frozen random projection. This is justified by theoretical and empirical work on random projections (Blum, 2005; Bingham and Mannila, 2001; Siddharth et al., 2020) showing that random linear projections preserve enough structure to serve as meaningful representation targets. The key insight: the target doesn't need to be optimal (which the learned target encoder attempts to be, but at the cost of potential collapse); it just needs to be stable and informative. The frozen random projection is provably stable (no parameters to update) and empirically informative enough to produce strong downstream features.

This positions OlmoEarth as striking a "middle ground between two common approaches" — stable like MAE but with the representational quality of latent methods. The paper explicitly presents Latent MIM Lite not as a completely novel paradigm but as a careful simplification of existing Latent MIM training that resolves its primary failure mode.

On the practical side, the paper positions the OlmoEarth Platform as solving the "last-mile problem" that separates research from impact. This is a deliberate framing: the technical contribution enables the models, but the platform contribution enables their use. The paper describes an end-to-end workflow — data collection, labeling, fine-tuning, inference, and map publishing — all accessible without GPU infrastructure or deep learning expertise. This positioning is reinforced by the case studies in Section 5.1 showing real organizations actively using the platform for mangrove monitoring and ecosystem mapping.

The paper's most distinctive positioning is its breadth: it simultaneously contributes a training methodology, a family of models at four scales, a comprehensive benchmark of 12 competing models, and a deployment platform. The evaluation scope — 24 embedding tasks and 29 fine-tuning tasks across research benchmarks and real-world partner data — is unusually comprehensive and reflects the paper's argument that EO foundation models should be judged not just on research benchmarks but on their utility for real-world environmental and humanitarian applications.

3. Technical Approach

3.1 Reader Orientation

OlmoEarth is a family of Vision Transformer-based models that processes multimodal satellite imagery time series — combining observations from different sensors (Sentinel-1 radar, Sentinel-2 multispectral, Landsat-8 thermal) with derived maps (land cover, elevation, crop data) — to produce embeddings or task-specific predictions for Earth observation applications. The core problem it solves is that existing self-supervised training methods for this kind of rich, multimodal data either produce weak representations (pixel-space reconstruction) or are unstable and collapse during training (latent-space prediction with learned target encoders); the solution is a deliberately simple modification — replacing the learned target encoder with a frozen random linear projection — that breaks the feedback loop responsible for collapse while maintaining the representational benefits of latent-space modeling, combined with a modality-aware masking strategy and loss function designed specifically for the redundancy and heterogeneity of Earth observation data.

3.2 Big-Picture Architecture (Diagram in Words)

The OlmoEarth system has five major components:

1. Input Data Pipeline — Assembles multimodal, multi-temporal satellite observations (Sentinel-1, Sentinel-2, Landsat-8) and derived maps (WorldCereal, OpenStreetMap, WorldCover, Cropland Data Layer, SRTM elevation, Canopy Height) into spatially and temporally aligned samples covering 2.56 km × 2.56 km regions over one-year periods with up to 12 monthly timesteps.

2. FlexiViT-Style Patch Embedding — Converts raw pixel data from each modality into token representations using a flexible patch-embedding layer that supports variable effective patch sizes (1 to 8) by resizing the input image to simulate different patch resolutions, combined with 2D sinusoidal positional encodings, sinusoidal temporal encodings, and learnable modality encodings.

3. Modality-Aware Masking Strategy — Determines which parts of the input are visible to the encoder versus used as reconstruction targets, operating at the bandset level (groups of spectral bands captured at the same native resolution) rather than individual tokens, with a key constraint that map-derived modalities are always masked from the encoder input and used only as decoder targets.

4. Transformer Encoder-Decoder Architecture — A ViT encoder (4 size variants from Nano at 1.4M parameters to Large at 300M parameters) with full self-attention across space, time, and modalities processes the unmasked input tokens; a shallower 4-layer decoder with cross-attention to the encoder outputs predicts the latent representations of the masked target tokens.

5. Latent MIM Lite Training Objective — A frozen, randomly initialized linear projection layer (one per modality) projects raw input patches into a token-space target; the decoder's predictions are compared to these targets using a modality-aware patch discrimination loss (contrastive loss comparing predicted tokens only against targets from the same bandset), combined with an instance contrastive loss on globally pooled encoder outputs to encourage cross-modal alignment.

Information flows as follows: a multimodal sample enters the system → the patch embedding layer converts raw pixels to tokens with positional/temporal/modality encodings → the masking strategy selects which bandsets are encoded versus decoded → the encoder processes visible tokens through full self-attention → the decoder cross-attends to encoder outputs and predicts representations for masked tokens → the frozen random projection produces target representations for the ground-truth masked patches → the modality patch discrimination loss and instance contrastive loss are computed and backpropagated.

3.3 Roadmap for the Deep Dive

• First, the input data and tokenization pipeline — because understanding what information the model receives and how it is converted into tokens is prerequisite to understanding the masking and training strategies that follow.
• Second, the modality-aware masking strategy — since this fundamentally reframes the pretraining task from "fill in randomly masked holes" to "reconstruct entirely missing modalities from partial views of other modalities," and determines which tokens the encoder sees versus what the decoder must predict.
• Third, the Latent MIM Lite target construction and why the frozen random projection solves the collapse problem — because this is the paper's core technical innovation and motivates the entire training design.
• Fourth, the modality patch discrimination loss — since this is where predictions are compared to targets, and the modality-aware negative selection is a key contribution that distinguishes OlmoEarth from standard patch discrimination.
• Fifth, the instance contrastive loss — because this separately addresses the global representation quality needed for classification tasks, complementing the local patch-level objective.
• Sixth, the full training recipe and hyperparameters — to make the approach reproducible and to understand the engineering choices behind the results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methods and systems paper whose core idea is that stable, high-quality self-supervised learning for multimodal Earth observation can be achieved by replacing the learned target encoder in Latent MIM with a frozen random linear projection, which eliminates the collapse-inducing co-adaptation between target and predictor networks, and that this simplified latent-space training, combined with modality-aware masking and a bandset-specific contrastive loss, produces representations that outperform both pixel-space reconstruction methods and prior latent-space approaches across a comprehensive suite of research benchmarks and real-world tasks.

---

Input Data and Tokenization

OlmoEarth's pretraining dataset consists of 285,288 samples drawn from locations worldwide, with each sample covering a spatial region of 2.56 km × 2.56 km and a temporal window of one year containing up to 12 monthly timesteps. The sample locations are selected based on OpenStreetMap features: the authors choose 120 categories of map features (ranging from roads to geothermal power plants), enumerate all 2.56 km × 2.56 km tiles containing each category, and randomly sample up to 10,000 tiles per category, with the one-year time range sampled uniformly between January 2016 and December 2024. Many categories appear in fewer than 10,000 tiles globally, so the final count of 285,288 reflects natural limits on feature occurrence.

Each sample can contain data from three satellite observation modalities and six derived map modalities. The observations are:

• Sentinel-1: Synthetic Aperture Radar (SAR) data, which provides information about surface texture and structure independent of cloud cover or lighting conditions.
• Sentinel-2: Multispectral optical imagery capturing reflected sunlight across multiple wavelength bands, providing information about vegetation health, land cover type, water bodies, and surface materials.
• Landsat-8: Optical and thermal imagery with different band specifications than Sentinel-2, offering complementary spectral information and a longer historical archive.

The derived maps are:

• WorldCereal (Van Tricht et al., 2023): A global map of crop type and cropping season information.
• OpenStreetMap: Crowd-sourced geographic data containing roads, buildings, land use boundaries, and other infrastructure features.
• WorldCover (Zanaga et al., 2022): An ESA-produced global land cover map at 10-meter resolution.
• Cropland Data Layer (USDA NASS): Agricultural land cover data for the United States, specifying crop types at field-level detail.
• SRTM (Shuttle Radar Topography Mission): A near-global digital elevation model providing terrain height information.
• Canopy Height Map (Tolan et al., 2024): A global map of forest canopy height derived from satellite observations.

All modalities are resampled to a uniform spatial resolution of 10 meters per pixel. The authors experimented with adding NAIP aerial imagery at 2.5 meters per pixel and ERA5 climate reanalysis data at 160 meters per pixel but "found no significant improvement on our evaluations," so these were excluded from the final training.

A critical design choice is the further subdivision of Landsat and Sentinel-2 into bandsets, which are groups of spectral bands that share the same native sensor resolution. Landsat-8 consists of 2 bandsets, while Sentinel-2 consists of 3 bandsets. The exact band grouping is specified in the released source code rather than the paper text, but the principle is that bands captured at the same spatial resolution by the sensor's optics are grouped together. This subdivision matters because the masking strategy and the modality patch discrimination loss both operate at the bandset level rather than the individual band level — tokens from bands within the same bandset are treated as coming from the same "modality" for the purposes of negative sampling in the contrastive loss.

Tokenization. The raw pixel data is converted into tokens through a FlexiViT-style patch embedding layer (Beyer et al., 2023). However, the paper uses a simplified variant:

"instead of doing that confusing pseudo-inverse stuff from FlexiViT we keep the actual projection weights the same size and resize the input image to mimic changing the patch size. It's probably basically equivalent."

In standard FlexiViT, a single set of patch embedding weights is used across multiple patch sizes by applying a pseudo-inverse resizing operation to the weights themselves. The OlmoEarth approach is simpler and more direct: the projection layer weights remain at a fixed size, and the input image is resized to simulate different effective patch sizes. During training, the model uses a random effective patch size sampled uniformly from the range {1, 2, ..., 8} and takes a random square crop from the input with side length in tokens uniformly sampled from the range {1, 2, ..., 12}. This means the spatial input can range from a 1×1 pixel region (patch size 1, 1-token crop) to a 96×96 pixel region (patch size 8, 12-token crop). This variable-resolution training forces the model to learn representations that are robust to spatial scale — a critical property for Earth observation, where the same physical object (a field, a forest, a building) can appear at very different pixel footprints depending on sensor resolution and image extent.

After the patch embedding projects the pixels to a token vector, three types of encoding are added:

• 2D sinusoidal positional encoding (sincos): Encodes the spatial position of each token within the 2D grid of the input crop, using standard sine and cosine functions at different frequencies. This is the same positional encoding scheme used in the original Transformer and Vision Transformer architectures.
• Sinusoidal temporal encoding: Encodes the timestamp of each observation within the one-year window. The paper does not specify the exact formulation, but sinusoidal temporal encodings typically use the same sine/cosine functions applied to the normalized time value (e.g., fraction of year).
• Learnable modality embedding: A unique learned vector for each modality (Sentinel-1, Sentinel-2 bandset 1, Sentinel-2 bandset 2, Sentinel-2 bandset 3, Landsat-8 bandset 1, Landsat-8 bandset 2, and each of the six map modalities). This embedding is added to every token from that modality and allows the model to distinguish which sensor or data source produced each token.

The sum of the patch embedding vector, the positional encoding, the temporal encoding, and the modality embedding constitutes the input representation for each token that enters the transformer.

---

Modality-Aware Masking Strategy

Standard masked image modeling for natural photographs randomly masks some fraction of input patches (typically 50-75% for MAE-style approaches) and trains the model to reconstruct the masked patches from the visible context. The paper argues this is inadequate for multimodal Earth observation data because of the high redundancy across space, time, and modalities:

"With multimodal remote sensing data, any token in the input will have many similar tokens either in space, time, or at a different aligned modality. Random masking is too easy of a task unless you use a very high masking ratio."

If you randomly mask 50% of Sentinel-2 tokens, the remaining unmasked Sentinel-2 tokens from nearby spatial locations, from adjacent monthly timesteps, or from the other Sentinel-2 bandset will be highly correlated with the masked ones. The model can solve the reconstruction task by interpolating from these correlated neighbors without learning meaningful semantic features. The paper notes that Galileo addresses this by using a 90% masking ratio, but extremely high masking ratios create their own problem: the model sees very little context, making the task artificially difficult in a way that may not transfer to downstream tasks where full context is available.

OlmoEarth's solution is a modality-aware masking strategy that operates at the bandset level rather than the individual token level. For each training example, the masking strategy assigns every bandset to one of four categories:

• Not selected: The bandset is completely ignored for this example — neither encoded nor decoded.
• Encode only: Tokens from this bandset are randomly masked (some fraction hidden from the encoder) and the visible tokens are input to the encoder. The masked tokens are NOT used as decoder targets — they are simply dropped.
• Decode only: All tokens from this bandset are hidden from the encoder input. They are used as targets for the decoder to predict.
• Encode and decode: Tokens from this bandset are randomly masked, the visible tokens go to the encoder, and the masked tokens are used as decoder targets. This is the standard masked modeling setup.

This strategy reframes the pretraining problem:

"This masking strategy re-frames the problem slightly from reconstructing data that has been partially masked to reconstructing missing bandsets from partial views of other bandsets."

When a bandset is assigned to "decode only," the encoder receives zero tokens from that bandset and must reconstruct it entirely from information in other bandsets. For example, if Sentinel-2 bandset 1 is "decode only" and Sentinel-2 bandset 2 is "encode only," the model must predict the high-resolution visible bands from the lower-resolution bands, or from Sentinel-1 radar data, or from Landsat-8 thermal data, or from temporal context — a genuinely challenging cross-modal reasoning task that requires semantic understanding.

When all bandsets are "encode and decode" (the standard random masking setup), the task is easier because the encoder still sees other tokens from the same bandset that are spatially or temporally correlated with the masked ones. The modality-aware masking deliberately removes this shortcut by sometimes hiding entire bandsets entirely.

Map modalities are always masked from the encoder. A critical constraint: during training, map-derived modalities (WorldCereal, OpenStreetMap, WorldCover, Cropland Data Layer, SRTM, Canopy Height) are never provided as encoder input. They can only be "decode only" or "not selected." The rationale is pragmatic:

"Maps can change over time–indeed downstream tasks are often detecting this kind of change–so we only rely on observations for inference."

If the model were allowed to use map data as encoder input during training, it would learn to depend on these high-quality labels. But at inference time, maps may be outdated or unavailable (if a map from 2020 says a region is forest, but deforestation occurred in 2023, the map is now misleading). By ensuring the encoder never sees map data during pretraining, the model learns to extract all necessary information from the raw satellite observations alone. The maps still provide a supervised training signal — the decoder must predict them as targets — but the encoder develops representations that support this prediction without ever directly ingesting the map values.

This design also unifies self-supervised and supervised pretraining under the same architecture and loss function:

"We project each modality, whether observations or maps, through a frozen random projection into token space. Loss is calculated the same for both types of modalities. We do not need to add on specific predictor heads for supervised data or adjust our training strategy or loss."

Models like Galileo and TerraMind train on both supervised and unsupervised data, but they treat supervised maps as valid model inputs. Their encoders must therefore learn to model map modalities and may use map information to predict observations or other maps during training. The paper argues that this complicates the encoder's learning task unnecessarily — it must build internal representations of map semantics in addition to observation semantics. OlmoEarth's approach keeps the encoder's job clean (process observations only) while still leveraging supervised data through the decoder loss.

---

Latent MIM Lite: The Frozen Random Projection Target

This is the paper's central technical contribution. To understand why it matters, we first need to understand the standard Latent MIM approach and its failure mode.

Standard Latent MIM. In Latent Masked Image Modeling (Wei et al., 2024), the model does not predict raw pixel values. Instead, it predicts representations of those pixels produced by a separate target encoder — a neural network (typically a smaller transformer or the exponentially moving average of the main model) that projects raw image patches into a latent space. The main model's decoder produces predictions in this same latent space, and a contrastive or similarity-based loss compares the predictions to the target representations. This is analogous to how knowledge distillation works: the target encoder provides "soft targets" that contain richer information than raw pixels.

The advantage: by predicting in a learned representational space rather than pixel space, the model is forced to capture semantic structure — the target encoder compresses away low-level pixel details (exact spectral values, sensor noise) and retains higher-level features, so reconstructing these targets requires semantic understanding. Empirically, latent-space models produce better downstream features than pixel-space models.

The problem: the target encoder and the main model co-adapt during training. If the target encoder is updated (e.g., via exponential moving average of the main model's weights), there is a feedback loop: the main model learns to predict the target encoder's outputs, the target encoder updates to reflect the main model's new representations, and this cycle can degenerate to a trivial equilibrium where both produce constant or degenerate outputs that minimize the loss but contain no useful information. This is representation collapse, and the paper explicitly encountered it:

"Full Latent MIM collapsed during training."

Table 4 shows the consequences: the standard Latent MIM baseline achieves only 32.2% accuracy on m-so2sat, 68.4% on m-eurosat, and 7.9 mIoU on PASTIS — and this is presumably the best checkpoint before collapse, since collapsed models produce essentially random representations.

How Latent MIM Lite solves this. The paper's insight is elegantly simple: the target encoder doesn't need to be learned. Replace it with a randomly initialized, frozen linear projection that is never updated during training:

"We replace the target encoder of Latent MIM with a linear projection from image patches to token space that is randomly initialized and never updated during training. This simple modification stabilizes training but maintains the representative power of modeling in latent space."

For each modality, a separate linear projection matrix is randomly initialized (using a standard scheme like Kaiming or Xavier initialization, though the paper doesn't specify which). This matrix projects the raw pixel values of a patch into a token vector of the same dimensionality as the model's internal representations. The projection is applied once per modality and the resulting target tokens are stored; they do not change during training.

Why this works theoretically. The paper cites work on random projections showing that randomly projected data preserves meaningful structure:

"Randomly projecting raw input data extracts valuable features both from a theoretical and practical standpoint [blum2005random, siddharth2020randpro, bingham2001random]. Thus our predictions are operating in a true latent space of our input data."

Random projections approximately preserve pairwise distances (Johnson-Lindenstrauss lemma), meaning that similar patches produce similar target vectors and dissimilar patches produce dissimilar target vectors. The key property: the random projection is a deterministic, fixed transformation of the input. There are no parameters to update, so there is no co-adaptation, no feedback loop, and no possibility of collapse. The training objective becomes: learn representations that can predict these fixed random projections of the input. This is conceptually similar to the framework of randomized target functions — the model is learning to approximate a random but information-preserving function of the input, which is provably a meaningful learning task because the function captures the data manifold's structure.

What is lost. The random projection is not optimal — a learned target encoder could in principle produce more informative targets by adapting to the specific structure of the data. The paper acknowledges this implicitly: "While it's possible this approach is too simplistic in more diverse domains like natural image processing, empirical results show a clear benefit in our domain of Earth observation data." For Earth observation, where the data is highly structured and the modalities are relatively limited in diversity compared to natural images, the random projection appears to be "good enough" — it captures enough structure to drive meaningful learning without the fragility of a learned target.

Quantifying the improvement. Table 4 shows the jump from standard Latent MIM to Latent MIM Lite:

• m-so2sat accuracy: 32.2% → 42.2% (+10.0 percentage points)
• m-eurosat accuracy: 68.4% → 87.2% (+18.8 percentage points)
• PASTIS mIoU: 7.9 → 35.2 (+27.3 mIoU points)

The PASTIS improvement is particularly dramatic — from a nearly useless model (7.9 mIoU, barely above random for 19 classes) to a functional one (35.2 mIoU). This confirms that the baseline failure was not just suboptimal performance but actual collapse, and that the frozen projection completely eliminates this failure mode.

Unifying self-supervised and supervised training. An additional benefit of the random projection approach is that it treats all modalities identically regardless of whether they are observations or maps. Each modality (Sentinel-1, Sentinel-2 bandset 1, WorldCereal map, etc.) gets its own frozen random projection matrix. The decoder produces predictions in the same token space for all modalities, and the loss compares predictions to the projected targets. There is no architectural distinction between "self-supervised prediction" (predicting a masked observation) and "supervised prediction" (predicting a map from observations). The loss is computed identically:

"We project each modality, whether observations or maps, through a frozen random projection into token space. Loss is calculated the same for both types of modalities."

This is in contrast to many multimodal models that require separate prediction heads for different data types (e.g., a classification head for maps, a reconstruction head for images). OlmoEarth's unified architecture emerges naturally from the Latent MIM Lite design and is not an additional feature bolted on.

---

Transformer Architecture and Decoder

The architecture is a standard Vision Transformer (ViT) encoder-decoder, with the decoder intentionally made smaller to concentrate representational capacity in the encoder.

Encoder. Four size variants are provided, based on standard ViT sizes:

The encoder processes all unmasked input tokens — those from bandsets assigned to "encode only" or "encode and decode" that survive random masking — through full self-attention. "Full self-attention" means every token attends to every other token regardless of spatial location, timestep, or modality. This is computationally expensive (quadratic in the total number of tokens) but enables the model to learn cross-modal and cross-temporal relationships without architectural constraints. Given the relatively small spatial extent (up to 12×12 = 144 tokens spatially, times up to 12 timesteps, times available modalities), the total token count is manageable even with quadratic attention.

Decoder. For each model size, the decoder shares the same hidden dimension and number of attention heads as the encoder but has a depth of only 4 layers (compared to 4, 12, 12, or 24 layers for the encoder). The explicit rationale:

"We design a smaller decoder so that the encoder does the majority of the modeling."

This is a common design in masked autoencoders — the decoder only needs to produce the target representations for masked tokens, not perform complex reasoning, so a shallow decoder suffices. The encoder does the hard work of building representations from visible tokens, and the decoder's job is to map those representations to the target token space through cross-attention.

Mask token handling. Masked positions (tokens that are hidden from the encoder) are represented in the decoder by a learned <MASK> embedding vector. This mask token is added to the appropriate positional, temporal, and modality embeddings for that position, ensuring the decoder knows where in space, time, and sensor-space the missing token belongs. The decoder then cross-attends to the encoder's output tokens (which represent the visible input) to predict the target latent vector for each masked position.

Global representation. For downstream tasks that require a single embedding for the entire input (e.g., image classification), OlmoEarth does not use a dedicated <CLASS> token. Instead, the encoder output tokens are average pooled across all spatial locations, timesteps, and modalities to produce a single global representation vector. This pooled vector is what is used for kNN classification and as input to linear probes or fine-tuned decoders. The instance contrastive loss (described below) is specifically designed to make this pooling operation well-behaved by encouraging tokens from different modalities to inhabit a common representational space.

---

Modality Patch Discrimination Loss

Once the decoder produces predicted latent vectors for the masked target tokens, these predictions must be compared to the ground-truth target tokens (produced by the frozen random projection). The loss function for this comparison is a crucial design choice.

Standard patch discrimination loss. In standard Latent MIM, a contrastive formulation called patch discrimination loss is used instead of a regression loss like Smooth L1. The intuition: predicting the exact latent vector values (regression) doesn't incentivize the model to produce diverse, well-separated representations. Instead, patch discrimination frames the problem as a classification task: for each predicted token, the goal is to make it similar to its corresponding ground-truth target token and dissimilar from all other ground-truth target tokens in the same input.

Formally, for a predicted token vector $\hat{z}_i$ (the decoder's output for a specific masked position) and a set of target token vectors $\{z_j\}$ (the frozen random projections of all masked patches in the input), the patch discrimination loss is:

$\mathcal{L}_{\text{patch}}(i) = -\log \frac{\exp(\text{sim}(\hat{z}_i, z_i) / \tau)}{\sum_j \exp(\text{sim}(\hat{z}_i, z_j) / \tau)}$

where $\text{sim}(\hat{z}_i, z_j)$ is the cosine similarity between the predicted vector $\hat{z}_i$ and target vector $z_j$, $\tau$ is a temperature parameter controlling the sharpness of the softmax distribution, $z_i$ is the target token corresponding to the same masked position as $\hat{z}_i$ (the positive example), and the sum over $j$ runs over all other target tokens in the input (the negative examples).

What this computes: For each predicted token, the loss is the negative log-likelihood of correctly identifying its matching target token among all target tokens. The numerator $\exp(\text{sim}(\hat{z}_i, z_i) / \tau)$ measures similarity to the correct target (should be high); the denominator sums similarities to all targets (correct and incorrect), creating a softmax over the set of possible matches. Minimizing this loss pushes the predicted embedding toward its correct target and away from all other targets.

Why contrastive rather than regression: A regression loss (MSE) would penalize the model for any deviation from the exact target values, but many different latent vectors could represent equivalent semantic content — the target space may be redundant. Contrastive loss cares only about relative similarity, allowing the model to produce vectors that are functionally equivalent (similar to the correct target, dissimilar to others) without being numerically identical. Additionally, the contrastive formulation explicitly pushes representations of different patches apart, preventing the model from collapsing to a single trivial representation.

The problem with standard patch discrimination for multimodal EO. In standard image modeling, all target tokens come from the same distribution — they are random projections of different spatial patches from the same RGB image. The contrastive task is genuinely challenging because all negatives are from the same modality and require fine-grained discrimination.

In OlmoEarth, target tokens come from fundamentally different distributions:

• Sentinel-1 SAR tokens have very different statistical properties from Sentinel-2 optical tokens.
• Landsat-8 thermal tokens differ from Sentinel-2 visible tokens.
• Map tokens (e.g., WorldCover land cover classes projected through their random matrix) differ from observation tokens.
• Even within Sentinel-2, tokens from different bandsets (different native resolutions) have different distributions.

The paper argues that distinguishing between modalities is trivially easy:

"Tokens from different modalities have very different distributions so distinguishing between them is easy. Yet there are so many tokens from other modalities that a significant amount of the loss comes from these 'easy' negatives."

If the loss function contrasts a predicted Sentinel-2 token against all target tokens — including Sentinel-1, Landsat-8, and map tokens — a large fraction of the negative comparisons are trivial. The model can achieve low loss simply by learning modality-specific features that make it easy to reject cross-modal negatives, without learning the harder within-modality discrimination that actually requires semantic understanding. The gradient signal from these easy negatives dominates the loss because there are many more cross-modality negative pairs than within-modality ones, diluting the training signal for the hard task.

Modality-aware patch discrimination loss. OlmoEarth's solution: restrict the contrastive comparison so that a predicted token is only compared against target tokens from the same bandset:

"Instead we contrast tokens only with other tokens in their respective bandset. This focuses the model training on a more challenging but more productive objective."

The loss for a predicted token $\hat{z}_i$ from bandset $b$ becomes:

$\mathcal{L}_{\text{mod-patch}}(i) = -\log \frac{\exp(\text{sim}(\hat{z}_i, z_i) / \tau)}{\sum_{j \in \mathcal{B}_b} \exp(\text{sim}(\hat{z}_i, z_j) / \tau)}$

where $\mathcal{B}_b$ is the set of all target tokens belonging to the same bandset $b$ as the predicted token, $z_i$ is the correct target for position $i$ (which is necessarily in $\mathcal{B}_b$), and $j$ ranges over all target tokens in that same bandset — but NOT over target tokens from other bandsets.

What this computes: The identical contrastive loss formulation, but with the negative set restricted to within-bandset tokens. For a predicted Sentinel-2 bandset-2 token, the negatives are only other Sentinel-2 bandset-2 target tokens — not Sentinel-1, not Landsat-8, not maps, not even Sentinel-2 bandset-1 tokens. This makes the task harder: the model must discriminate between patches within the same modality, which requires capturing fine-grained semantic differences rather than coarse modality-level features.

Why this form: By eliminating easy cross-modality negatives, the loss focuses gradient signal on the within-modality discrimination that transfers to downstream tasks. A model that can tell the difference between two Sentinel-2 patches (e.g., "this is a wheat field" vs. "this is a corn field") has learned something semantically useful; a model that can tell Sentinel-2 apart from Sentinel-1 has learned a trivial sensor-identification shortcut. The modality-aware loss explicitly incentivizes the former and ignores the latter.

Table 4 quantifies the benefit: adding modality patch discrimination to the Latent MIM Lite + modality masking baseline improves m-so2sat from 53.6% to 55.3%, m-eurosat from 90.2% to 91.5%, and PASTIS from 46.6 to 48.1 mIoU. These are incremental gains on top of an already-strong baseline, suggesting the modality-aware loss refines representations that are already good.

---

Instance Contrastive Loss

The patch discrimination loss operates on local, per-token representations — it ensures that the decoder can predict individual masked patches correctly. However, many downstream tasks (classification, regression) require a single global representation of the entire input that captures the overall content (e.g., "this 2.56 km × 2.56 km region over this one-year period is a mangrove forest"). Generating good global representations requires two things: (1) the per-token representations must contain the relevant information, and (2) the aggregation operation (average pooling in OlmoEarth's case) must produce a coherent vector.

The issue with naive pooling of multimodal tokens is that tokens from different modalities may inhabit different regions of the embedding space:

"Tokens encoded from the same modality share semantics but tokens from different modalities may look very different from each other. We want to be able to average tokens from all modalities together and get a sensible global representation of an input."

If Sentinel-1 tokens cluster in one region of the embedding space and Sentinel-2 tokens cluster in a completely different region, averaging them produces a vector that is not representative of either — it sits somewhere in between, potentially in an empty region of the space with no semantic meaning. The instance contrastive loss addresses this by explicitly encouraging cross-modal alignment in the encoder's output space.

Formulation. The loss follows the SimCLR framework (Chen et al., 2020): generate two different "views" of the same input, encode them, and contrast the resulting global representations against each other as positives, with global representations from other inputs in the batch serving as negatives.

The paper generates the two views not through data augmentation (the standard SimCLR approach) but through different random masking:

"We run random masking twice, then encode both batches with our encoder, pool the resulting tokens, and apply contrastive loss to the pooled representations."

For each input sample, the modality-aware masking strategy is applied twice independently, producing two different mask configurations. Both masked inputs are passed through the encoder (sharing the same encoder weights). The output tokens from each encoding are average-pooled across space, time, and modalities to produce two global representation vectors $g^{(1)}$ and $g^{(2)}$.

The instance contrastive loss is:

$\mathcal{L}_{\text{inst}} = -\log \frac{\exp(\text{sim}(g^{(1)}_i, g^{(2)}_i) / \tau)}{\sum_{j=1}^{B} \exp(\text{sim}(g^{(1)}_i, g^{(2)}_j) / \tau)}$

where $g^{(1)}_i$ and $g^{(2)}_i$ are the global representations from the two masked views of sample $i$, $\tau$ is a temperature parameter, $\text{sim}$ is cosine similarity, and $B$ is the batch size (or, due to memory constraints, the micro-batch size of 32 — see below). The sum in the denominator runs over all pairs where the second view comes from a different sample ($j \neq i$), making these the negative examples.

What this computes: For each sample, the loss is the negative log-likelihood of correctly identifying the matching (second-view) representation among all second-view representations in the batch. The model must make the two representations of the same sample more similar to each other than to representations of other samples.

Why this helps with cross-modal pooling: By forcing two differently masked views of the same sample to produce similar global representations, the loss incentivizes the encoder to produce tokens that are invariant to which modalities are visible. If Sentinel-1 tokens are masked in view 1 but visible in view 2, both views must still produce similar global representations, which means the pooling operation must be robust to the presence or absence of specific modalities. This pushes tokens from different modalities into a shared representational space where averaging across modalities is meaningful. Additionally, the contrastive formulation separates representations of different samples globally, ensuring that the pooled vectors capture sample-level identity rather than just local patch features.

Scaling and micro-batch handling. The instance contrastive loss is most effective with large batch sizes, because more negative examples create a harder and more informative contrastive task. However, full-batch contrastive loss requires storing all representations and computing a large similarity matrix, which is memory-intensive:

"Due to memory constraints we use a micro-batch size of 32 so the pooled contrastive loss is only applied over these 32 examples, not the full batch of 512."

The total batch size is 512, meaning 16 micro-batches of 32. The contrastive loss is computed independently within each micro-batch — a predicted representation is contrasted against the other 31 examples in its micro-batch as negatives. This reduces the effective number of negatives but makes training feasible within GPU memory constraints.

Loss weighting. The instance contrastive loss is not the primary training objective — the modality patch discrimination loss is. The instance contrastive loss is added as an auxiliary objective:

"A scalar multiple controls the contribution of instance contrastive loss to modality patch discrimination loss. For experiments in this paper we scale the instance contrastive loss by 0.1."

The total loss per training step is:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{mod-patch}} + 0.1 \cdot \mathcal{L}_{\text{inst}}$

where $\mathcal{L}_{\text{mod-patch}}$ is summed over all predicted masked tokens from both masked views, and $\mathcal{L}_{\text{inst}}$ is computed once per micro-batch. The 0.1 scaling reflects the paper's finding that the instance loss is a helpful regularizer but should not dominate training — the primary task is local patch prediction, with global alignment as a secondary objective.

Ablation impact. Table 4 shows that adding the instance contrastive loss to the Latent MIM Lite + modality masking + modality patch discrimination baseline improves m-so2sat from 55.3% to 56.8%, m-eurosat from 91.5% to 92.3%, and PASTIS from 48.1 to 49.0 mIoU. In the more comprehensive ablation (Table 6), removing the instance contrastive loss from the final recipe ("No Inst. Contrastive Loss" row) reduces the average rank from 2.9 to 4.7, suggesting it provides a meaningful but moderate improvement to overall performance.

---

Full Training Recipe and Hyperparameters

The paper provides a detailed training recipe that serves as the reference for reproducibility.

Optimization. The training uses AdamW (Adam with decoupled weight decay):

• Base learning rate: $1 \times 10^{-4}$
• Weight decay: $0.02$
• Batch size: $512$ (total)
• Micro-batch size: $32$ (for instance contrastive loss computation)
• Learning rate schedule: linear warmup for $8000$ steps from 0 to the base learning rate, followed by cosine annealing that decays the learning rate by a factor of $0.1$ over the remaining steps.
• Total training steps: $667,200$

The cosine annealing schedule reduces the learning rate from the peak value of $1 \times 10^{-4}$ to a final value of $1 \times 10^{-5}$ (10% of peak) over the course of training, with the decay following a half-cosine curve. The 8,000-step linear warmup prevents the model from experiencing large gradients at initialization when the random weights and the frozen random projections are poorly aligned.

Data loading and augmentation. Each training step involves random choices that determine the input format:

• Effective patch size: sampled uniformly from $\{1, 2, \dots, 8\}$. Rather than changing the patch embedding weights (as in standard FlexiViT), the input image is resized to simulate the chosen patch size. A patch size of 1 means each pixel becomes a token; a patch size of 8 means 8×8 = 64 pixels are averaged or downsampled into a single token.
• Spatial crop: a random square crop from the input, with side length in tokens sampled uniformly from $\{1, 2, \dots, 12\}$. The smallest possible input is a single token covering a 1×1 pixel region (patch size 1, crop size 1); the largest is 96×96 pixels (patch size 8, crop size 12 = 96).
• Temporal sampling: between 3 and 12 timesteps from the available monthly observations in the one-year window. The paper doesn't specify whether the selection is random or contiguous.
• Modality availability: not all modalities are available for all samples. The data loading handles missing modalities gracefully — they are simply not included in the input.

This aggressive data augmentation (variable spatial extent, variable resolution, variable temporal coverage, variable modality availability) serves as a strong regularizer and teaches the model to be robust to the wide variety of input configurations it will encounter at inference time.

Token throughput. The paper reports that during training the model processes "around 100 billion tokens." This is the total number of input tokens seen across all training steps, counting each spatial-temporal-modality token separately. For context, a single training example at the maximum configuration (12 timesteps, all modalities present, 12×12 spatial crop, all bandsets) could contain tens of thousands of tokens, though the average is much lower due to masking and the variable crop sizes.

Hardware. Pretraining was conducted on NVIDIA H100 GPUs (for Nano, Tiny, and Base variants) and B200 GPUs (for Large). The GPU-hour counts are reported in Table 5:

• Nano: 1,149 GPU-hours on H100
• Tiny: 1,149 GPU-hours on H100
• Base: 2,989 GPU-hours on H100
• Large: 5,240 GPU-hours on B200

The total energy consumption across all pretraining runs was 4,307 kWh, equivalent to about 5 months of average US household electricity use.

Ablation training protocol. The ablations in Tables 4 and 6 use a reduced training budget. Table 4 models are trained for 140,000 steps (approximately 21% of the full 667,200 steps), while Table 6 models are trained for 300,000 steps (approximately 45% of full training). The paper notes that improvements observed during development on a representative subset of evaluation tasks ("in-loop evals") carried over to the full evaluation, validating the use of shorter training runs for ablations.

Why these choices. Several design decisions warrant explanation:

• Variable patch size and crop: This is perhaps the most distinctive training choice. By forcing the model to process inputs at radically different spatial scales — from single-token to 144-token grids — the model cannot overfit to a particular spatial context. This is especially important for Earth observation, where the same model might be applied to very different spatial footprints (a single field vs. a whole county) and must produce consistent representations regardless.
• Cosine annealing to 10% of peak LR: The paper does not reduce the learning rate all the way to zero. The 0.1 decay factor means training ends at $1 \times 10^{-5}$. This is a common practice that provides a small but non-zero learning rate at the end, allowing the model to continue learning rather than stalling completely.
• Micro-batch contrastive loss: The constraint of only contrasting within micro-batches of 32 (rather than the full batch of 512) is a compromise between memory constraints and contrastive loss quality. More negatives would be better, but 31 negatives per positive is sufficient to provide a meaningful contrastive signal.
• Small weight on instance loss (0.1): The paper found that the instance contrastive loss is a useful auxiliary objective but can interfere with the primary patch discrimination task if weighted too heavily. The 0.1 scaling ensures the model focuses primarily on per-patch reconstruction.

---

Design Choices: Why This Approach Over Alternatives

The paper makes several non-obvious design choices that collectively define the OlmoEarth approach. Understanding the rationale for each helps clarify why the model works and where its limitations might lie.

Why frozen random projection over a learned target encoder? This is the central choice. The alternatives are: (a) pixel-space reconstruction (MAE), which is stable but yields weaker representations; (b) a learned target encoder updated via exponential moving average (Latent MIM, I-JEPA), which yields stronger representations but is unstable and prone to collapse; and (c) a frozen tokenizer trained via vector quantization (TerraMind's approach), which requires a separate training stage for the tokenizer. Option (a) was ruled out by the paper's own ablation showing MAE significantly underperforms (Table 6). Option (b) was ruled out empirically by the collapse observed in the "Full Latent MIM" baseline (Table 4). Option (c) is viable — TerraMind uses it — but adds complexity: training a quantized autoencoder is itself a non-trivial undertaking and the codebook size and training procedure become additional hyperparameters to tune. The frozen random projection requires no training, no hyperparameters (beyond initialization scheme), and is guaranteed to be stable. The paper's ablation evidence suggests it works well enough for EO data that the additional complexity of learned alternatives is not justified.

Why resizing images instead of FlexiViT's pseudo-inverse? FlexiViT resizes the patch embedding weights to handle different patch sizes, which requires computing a pseudo-inverse and can be numerically sensitive. The paper's approach — keep weights fixed and resize the input — is simpler and claimed to be "probably basically equivalent." The tradeoff is that resizing the input changes the effective spatial resolution (information is lost when downsampling to simulate a larger patch size), while the FlexiViT approach preserves the original image resolution and changes the model's receptive field. For Earth observation data, where spatial resolution is somewhat flexible and resampling to 10m has already been performed, the simpler approach is well-motivated.

Why mask entire bandsets instead of just high masking ratios? The alternative (used by Galileo) is to randomly mask tokens at a very high ratio (90%). The problem with high random masking is that it treats all tokens identically: the model might see a few tokens from every modality rather than being forced to reason across modalities. By masking entire bandsets, OlmoEarth forces the model to reconstruct one type of data entirely from other types — simulating the realistic scenario where some sensor data is unavailable (cloud cover, sensor failure, temporal gaps). This cross-modal reconstruction task requires genuine understanding of the relationships between modalities.

Why never encode map data? Models like Galileo and TerraMind allow map data as encoder input, which means the encoder learns representations of both observations and maps. At inference time, when maps may be unavailable or outdated, the encoder may underperform because it expects map inputs it no longer receives. OlmoEarth's encoder never sees maps during pretraining, so it never develops a dependence on them. The maps still provide a training signal — the decoder must predict them — but the encoder's representations are based purely on satellite observations. The paper's evaluation results, where OlmoEarth outperforms Galileo and TerraMind on most tasks (Tables 2 and 3), provide empirical support for this design.

Why contrast within bandsets only? The alternative is the standard patch discrimination loss that contrasts against all target tokens. The paper argues this leads to "easy" cross-modality negatives dominating the loss. By restricting negatives to the same bandset, the loss becomes harder and more semantically meaningful. A potential downside — not discussed in the paper — is that within-bandset negatives are fewer than global negatives, which could make the contrastive task easier in some cases (fewer distractors). The empirical results suggest this is not a problem, likely because within-bandset negatives are genuinely challenging to distinguish.

Why instance contrastive loss as an auxiliary objective? The alternative is to rely purely on the decoder's per-token predictions to produce good global representations (as MAE does, where the <CLS> token is trained only through the pixel reconstruction objective). The paper argues this is insufficient because tokens from different modalities may not inhabit a shared space, making pooling unreliable. The instance contrastive loss explicitly addresses this by forcing differently-masked views of the same sample to similar global representations. A downside — the micro-batch limitation means the contrastive signal is weaker than it would be with full-batch negatives, which may partially explain why the instance loss provides only moderate improvements in ablations (Table 6: removing it reduces average rank from 2.9 to 4.7).

4. Key Insights and Innovations

Innovation 1: The Frozen Random Projection as a Collapse-Free Training Stabilizer

The paper's most distinctive contribution is not a new architecture or loss function, but a diagnostic insight with a surgical fix: the representation collapse that plagues latent-space self-supervised learning in Earth observation arises specifically from the co-adaptation between the target encoder and the prediction model, and replacing the learned target encoder with a frozen random linear projection eliminates this failure mode entirely while preserving — and in fact improving — downstream performance.

What the field did before. The dominant assumption in latent-space self-supervised learning — from I-JEPA (Assran et al., 2023) through Latent MIM (Wei et al., 2024) to contemporary EO models like Anysat and Galileo — has been that the target encoder must be learned, typically via exponential moving average of the main model's weights, to produce informative latent targets. The theoretical motivation is clear: a learned target can adapt to the data distribution, compressing away irrelevant pixel-level detail while preserving semantic structure. The practical consequence, however, has been fragility: the feedback loop between target and predictor creates a dynamical system that can degenerate to trivial equilibria. The field's response has been to manage this instability through careful hyperparameter tuning, architectural constraints, or alternative approaches like TerraMind's quantized autoencoder tokenizer — each of which adds complexity and cost.

What makes OlmoEarth's approach a conceptual shift. The paper reframes the problem: the target encoder does not need to be optimal; it needs to be stable and sufficiently informative. The empirical evidence in Table 4 is stark — the "Full Latent MIM" baseline not only underperforms but collapses during training, achieving 7.9 mIoU on PASTIS versus 35.2 for the frozen-projection variant (+27.3 mIoU, essentially the difference between a broken model and a functional one). This is not a marginal improvement from better hyperparameters; it is a qualitative change in training dynamics. The frozen random projection — justified by theoretical work on random projections preserving pairwise distances (Blum, 2005; Bingham and Mannila, 2001) — acts as a deterministic, non-adaptive transformation of the input. There are no parameters to update, no moving averages, no feedback loop. Collapse becomes mathematically impossible because the target is fixed.

Why this matters beyond OlmoEarth. This finding challenges a core assumption in self-supervised learning: that learned targets are necessary for strong representations. The paper demonstrates that for Earth observation data — and potentially for other structured, multimodal domains — a fixed random projection is not merely a "good enough" approximation but can produce better downstream features than a learned target encoder, because the stability gains outweigh the loss of target optimality. This is a fundamental insight about the tradeoff between target quality and training stability that has implications for other domains where latent-space methods suffer from collapse (e.g., video understanding, medical imaging, robotics). The paper is explicit that this approach may be "too simplistic in more diverse domains like natural image processing," but the conditional claim — that it works for EO because the data is sufficiently structured that random projections capture meaningful features — is a precise, testable hypothesis that opens new research directions.

Evidence anchor. Table 4, rows 1-2: standard Latent MIM (collapsed) vs. Latent MIM Lite (+10 pp on m-so2sat, +18.8 pp on m-eurosat, +27.3 mIoU on PASTIS). Table 6, "Final Recipe" vs. "MAE" baseline showing Latent MIM Lite substantially outperforms pixel-space reconstruction across 10 tasks (average rank 2.9 vs. 5.1).

---

Innovation 2: Modality-Aware Masking as a Cross-Modal Reasoning Curriculum

The paper introduces a masking strategy that fundamentally reframes the self-supervised pretraining task from "fill in randomly masked patches using nearby context" to "reconstruct entirely missing sensor modalities from partial observations of other modalities." This is more than an engineering trick; it is a conceptual redefinition of what the model is being trained to do — and it addresses a specific, identified failure mode of standard random masking in multimodal EO data.

What the field did before. Standard masked modeling for images (MAE, BERT) randomly masks a fraction of input tokens and trains the model to reconstruct them from the remaining visible context. For EO models like Galileo, the adaptation has been to increase the masking ratio to extreme levels (90%) to prevent the task from being solved by spatial or temporal interpolation from correlated neighbors. This is a quantitative fix — "make the task harder by hiding more" — that treats the problem as one of difficulty scaling rather than task design.

What makes OlmoEarth's approach a conceptual shift. The paper diagnoses the real problem: in multimodal, multi-temporal EO data, tokens are redundant across space, time, and sensor modalities. Random masking at any ratio leaves correlated tokens visible (different bandset from the same sensor, adjacent timestep, similar land cover at a nearby location), allowing the model to solve reconstruction without cross-modal reasoning. OlmoEarth's solution is not to increase the masking ratio but to change what is masked: by sometimes hiding entire bandsets entirely ("decode only"), the model is forced to reconstruct one type of data (e.g., Sentinel-2 optical bands) from completely different types (e.g., Sentinel-1 radar, Landsat-8 thermal, or temporal context). This is no longer an inpainting task; it is a cross-modal translation task that requires the model to understand the physical relationships between different sensors — that certain surface materials produce specific radar backscatter signatures and specific optical reflectance spectra simultaneously, and that inferring one from the other requires semantic understanding of land cover, vegetation structure, and surface properties.

The constraint on map modalities deepens this insight. By never allowing map-derived data as encoder input (only as decoder targets), the model cannot develop a dependence on high-quality labels that might be unavailable or outdated at inference time. This is a deliberate architectural choice that anticipates deployment constraints — a form of test-time robustness by construction rather than by post-hoc regularization. Other models (Galileo, TerraMind) treat maps as valid inputs, creating a potential train-test mismatch when maps are stale or unavailable.

Evidence anchor. Table 4 shows the incremental benefit of modality masking on top of Latent MIM Lite: PASTIS mIoU jumps from 35.2 to 46.6 (+11.4 points). This is the single largest incremental gain in the development trajectory, suggesting that the masking strategy is as important as the frozen projection for overall performance. The qualitative examples in Figures 2 and 3 reinforce this: the model must learn to translate between fundamentally different data types, a harder but more transferable skill than within-modality interpolation.

---

Innovation 3: Bandset-Restricted Contrastive Loss as a Hard-Negative Mining Strategy

The modality patch discrimination loss is a seemingly minor modification — restrict the contrastive comparison so that a predicted token is only compared against target tokens from the same bandset — but it encodes a significant insight about where the useful training signal lives in multimodal contrastive learning. The paper recognizes that in standard patch discrimination, cross-modality negatives dominate the loss because they are numerous and trivially distinguishable, diluting the gradient signal for the hard within-modality discrimination that actually transfers to downstream tasks.

What the field did before. Standard patch discrimination (Wei et al., 2024) contrasts each predicted token against all target tokens in the input, regardless of modality. This works for natural images because all patches come from the same RGB distribution. For multimodal EO, it means a predicted Sentinel-2 token is contrasted against Sentinel-1, Landsat-8, and map tokens — most of which are trivially distinguishable based on modality-specific features. The model's gradient signal is dominated by learning to reject these easy negatives.

What makes this a conceptual contribution. The paper identifies a signal-to-noise problem in contrastive loss design for multimodal data. The field's default assumption — that more negatives are always better — is shown to be false when the negatives come from fundamentally different data distributions. By restricting negatives to within-bandset comparisons, the loss becomes harder (fine-grained discrimination between patches of the same sensor type) and the gradient signal becomes more concentrated on semantically meaningful features. This is a form of automatic hard-negative mining: the within-bandset tokens are the hardest negatives because they share the same modality distribution, so the model must learn subtle differences (wheat vs. barley in Sentinel-2 imagery) rather than coarse differences (Sentinel-2 vs. Sentinel-1).

The paper's ablation quantifies this: adding modality patch discrimination on top of Latent MIM Lite + modality masking improves m-so2sat from 53.6% to 55.3% and PASTIS from 46.6 to 48.1 mIoU (Table 4). These are incremental gains — the modality masking provides the larger jump — but they represent a real improvement in representation quality from a principled loss modification. The insight generalizes beyond EO: any domain where contrastive learning operates over heterogeneous data sources (multimodal medical imaging, multi-sensor robotics, audio-visual learning) may benefit from restricting the contrastive comparison to semantically coherent subsets rather than contrasting across arbitrarily different modalities.

Evidence anchor. Table 4, rows 3-4: "Modality Masking" baseline vs. "+ Modality Patch Disc." Table 6 confirms the pattern across a broader set of 10 tasks.

---

Innovation 4: Instance Contrastive Loss as Cross-Modal Representation Alignment

Standard self-supervised objectives for masked modeling (MAE, Latent MIM) operate on per-token predictions — they ensure that local patches are well-represented but provide no explicit incentive for tokens from different modalities to inhabit a shared representational space. OlmoEarth's instance contrastive loss addresses a previously unarticulated failure mode: naive average-pooling of multimodal tokens produces incoherent global representations if tokens from different modalities cluster in different regions of the embedding space.

What the field did before. Most EO foundation models either use a dedicated <CLASS> token trained implicitly through the reconstruction objective (expecting it to capture global information as a byproduct of local prediction) or simply pool tokens and hope for the best. The problem is that tokens from Sentinel-1 SAR and Sentinel-2 optical data carry fundamentally different physical information and could, without explicit encouragement, be embedded in entirely different subspaces. Averaging them would produce a vector in an intermediate, potentially semantically empty region — what the paper implies when it notes that "tokens from different modalities may look very different from each other."

What makes this a conceptual contribution. The paper identifies that global representation quality is not a free byproduct of local patch prediction in multimodal settings, and introduces a lightweight auxiliary objective that explicitly aligns cross-modal representations. By generating two differently masked views of the same input and contrasting their global pooled representations against other samples in the batch, the loss forces the encoder to produce tokens that, when averaged, yield a consistent sample-level identity regardless of which modalities happened to be visible. This simultaneously achieves two goals: (1) it encourages tokens from different modalities to be comparable (so pooling is meaningful), and (2) it enforces sample-level discriminability (so the pooled vector captures what makes this geographic location and time period distinctive from others).

The micro-batch limitation (contrasting only within groups of 32 rather than the full batch of 512) weakens the contrastive signal, which likely explains why the ablation gains are moderate (Table 6: removing instance contrastive loss reduces average rank from 2.9 to 4.7). But the conceptual contribution stands independent of the magnitude: the paper identifies a specific representational requirement that masked modeling alone does not guarantee, and provides a principled solution.

Evidence anchor. Table 4, rows 4-5: "+ Contrastive Loss" provides consistent but modest gains across all three validation tasks. The design choice of scaling the instance loss by 0.1 relative to the patch discrimination loss reflects the auxiliary nature of this objective — it refines representations rather than driving them.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on 18 research benchmarks and 19 datasets from 7 partner organizations. Research benchmarks include all seven Sentinel-2 and Landsat tasks from GEO-Bench (m-bigearthnet, m-so2sat, m-brick-kiln, m-forestnet, m-eurosat, m-cashewplant, m-SA-crop-type), plus BreizhCrops, CropHarvest, PASTIS, MADOS, and Sen1Floods11. Partner tasks span classification, segmentation, object detection, and regression across domains including land cover mapping (AWF), ecosystem classification (GEA North Africa), mangrove detection (Global Mangrove Watch), crop type mapping (Nandi), forest loss driver classification (Amazon Conservation), live fuel moisture content regression (NASA JPL), marine infrastructure detection, vessel detection/type/length estimation (Skylight), and solar farm detection. The paper notes that not all models support all modalities, so some evaluation cells are left blank — for instance, models that do not support time series data receive each time step as a separate input, and models incompatible with specific sensor types (Sentinel-1 only, Sentinel-2 only, Landsat-8 only) cannot be evaluated on those tasks. Dataset-specific limitations are documented in Appendix A, including the observation that m-brick-kiln negatives make the classification task "too easy" (many negatives contain only dark pixels), that Sen1Floods11 accuracy is not well-correlated with other benchmarks, and that single-image inputs in GEO-Bench are artificial for labels that do not change rapidly over time.

• Base model(s). OlmoEarth is trained in four ViT encoder sizes: Nano (4 layers, 128-dim hidden, 8 heads, 1.4M parameters), Tiny (12 layers, 192-dim hidden, 3 heads, 6.2M parameters), Base (12 layers, 768-dim hidden, 12 heads, 90M parameters), and Large (24 layers, 1024-dim hidden, 16 heads, 300M parameters). All variants share a decoder with 4 layers matching the encoder's hidden dimension and head count. The paper compares against 12 other foundation models imported into the same evaluation framework: Anysat ViT Base (Astruc et al., 2024), Clay ViT Large (Clay Foundation), CopernicusFM ViT Base, CROMA ViT Base and Large (Fuller et al., 2024), DINOv3 ViT Base/Large/Huge+/7B (Siméoni et al., 2025), DINOv3 Sat ViT Large/7B, Galileo ViT Nano/Tiny/Base (Tseng et al., 2025), Panopticon ViT Base (Waldmann and Shah, 2025), Presto ViT Nano (Tseng et al., 2023), Prithvi v2 ViT Large/Huge (Szwarcman et al., 2024), Satlas Swin Base (Bastani et al., 2023), TerraMind ViT Base/Large (Jakubik et al., 2025), and TESSERA (Feng et al., 2025). The choice of PaLM 2-S* in the original reference example is not relevant here; OlmoEarth's own architecture family is the focus.

• Metrics. Metric choice varies by task type. For single time-step classification (m-bigearthnet, m-so2sat, m-brick-kiln, m-forestnet, m-eurosat, Sen1Floods11, AWF, Nandi), the paper reports either micro-F1 (μF1), accuracy (Acc.), or both depending on the benchmark convention. For multi-temporal classification and segmentation (BreizhCrops, CropHarvest, m-cashewplant, m-SA-crop-type, PASTIS, MADOS, partner tasks with segmentation), the primary metric is mean Intersection over Union (mIoU). For regression tasks (Live Fuel Moisture Content, Vessel Length), the metric is L1 error. For object detection (Vessel Detection, Marine Infrastructure), the metric is F1 score. The paper follows standard evaluation practices from Reed et al. (2023), Fuller et al. (2024), and Tseng et al. (2025) for kNN and linear probing. For fine-tuning, validation set performance is used to select checkpoints and hyperparameters, with test set results reported for the best validation configuration.

• Baselines. The 12 external foundation models serve as baselines, each evaluated using the same training recipes and hyperparameter sweeps as OlmoEarth. The paper states that "we import other top performing foundation models into our evaluation framework and evaluate them as well so they are directly comparable," and that "we use the same training recipes for each foundation model but sweep a variety of hyperparameters to find the best performance for each model on each task." This is significant because it means the reported numbers for competing models may differ from their original papers — they reflect performance under a standardized evaluation protocol rather than the models' originally reported settings. The paper also includes a "Random Init" baseline in Table 3, which is an OlmoEarth ViT Base with randomly initialized weights (no pretraining), fine-tuned using the same recipe. This serves as a lower bound to quantify the value added by pretraining.

• Generation budget / compute accounting. The paper measures computational efficiency in terms of multiply-accumulate operations (MACs) to encode one example, as shown in the Pareto frontier plot (Figure 1). The x-axis of this plot is MACs, and the y-axis is average performance across 13 embedding tasks. The paper explicitly states that this average covers "specifically the Sentinel-2 versions of: m-bigearthnet, m-so2sat, m-brick-kiln, m-eurosat, BreizhCrops, CropHarvest-Togo, CropHarvest-PRC, m-cashewplant, m-SA-crop-type, PASTIS, MADOS, AWF, Nandi." Input size varies by task, so the MAC count is computed per-example with the actual input dimensions used for each evaluation. For pretraining compute, the paper reports GPU-hours (Table 5): Nano uses 1,149 H100-hours, Tiny uses 1,149 H100-hours, Base uses 2,989 H100-hours, Large uses 5,240 B200-hours. For fine-tuning compute, separate GPU-hour counts are reported per model size, with total fine-tuning costs ranging from 39 kWh (Large, research benchmarks only) to 685 kWh (Base, all tasks).

• Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection in the standard sense. Instead, it uses a fixed train/validation/test split for each benchmark as provided by the dataset authors. For kNN and linear probing evaluations, the paper sweeps over data normalization strategies (pretraining statistics vs. evaluation set statistics) and feature pooling methods (mean vs. max across time), and for linear probing additionally sweeps learning rates over {1×10⁻⁴, 5×10⁻⁴, 1×10⁻³, 5×10⁻³, 1×10⁻², 5×10⁻², 1×10⁻¹, 5×10⁻¹}, reporting test set results for the configuration with the highest validation set performance. For fine-tuning on research benchmarks, learning rates are swept over {1×10⁻⁴, 5×10⁻⁴, 1×10⁻³}. For fine-tuning on partner tasks, a learning rate of 1×10⁻⁴ is used for all tasks except Nandi, where some models exhibit unstable learning and the rate is swept over {1×10⁻⁴, 1×10⁻⁵}. No confidence intervals, standard errors, or statistical significance tests are reported for any result — the paper presents single-point estimates throughout. The ablation models in Table 4 are evaluated on validation sets only (not test sets), and the paper states that "during development we ran a subset of our evaluations in our 'in-loop evals' but saw that improvements on a representative subset carried over to the full evaluation."

---

Main Quantitative Results

Embedding-Based Evaluation (kNN and Linear Probing)

Table 2 presents the kNN and linear probing results across 24 tasks. OlmoEarth achieves the best performance on 15 of 24 tasks, with the ViT Large variant being best on 7 tasks, ViT Base on 6 tasks, ViT Tiny on 2 tasks, and ViT Nano on 0 tasks (though Nano is competitive, achieving 96.2% on m-brick-kiln and 83.7% on CropHarvest-Togo with Sentinel-2 data, which are near-best results).

Key head-to-head results (OlmoEarth ViT Base vs. strongest competitor for each task):

• m-bigearthnet (μF1, S2): Panopticon ViT Base achieves 64.9, CopernicusFM achieves 64.6, TerraMind ViT Base achieves 63.9. OlmoEarth ViT Base achieves 62.4. OlmoEarth is competitive but not best — it ranks 6th among the evaluated models on this task. The gap between OlmoEarth and the leader is 2.5 μF1 points.

• m-so2sat (Acc., S2): OlmoEarth ViT Large achieves 68.2%, OlmoEarth ViT Base achieves 67.7%. The next best models are Panopticon ViT Base at 60.5% and Galileo ViT Base at 55.7%. The gap between OlmoEarth and the second-best model is 7.2 percentage points — a substantial margin that suggests OlmoEarth's multimodal training provides strong transfer to this Sentinel-2 classification task. Notably, the purely visual models (DINOv3 family) peak at 50.1% (DINOv3 Sat ViT 7B), roughly 18 points below OlmoEarth.

• m-brick-kiln (Acc., S2): OlmoEarth ViT Nano achieves 96.2%, surpassing all other models. Panopticon achieves 92.9%, CROMA ViT Base achieves 92.0%, TerraMind ViT Large achieves 92.2%. The paper notes in Appendix A that this benchmark has an issue — "many negatives seem to have only dark pixels, making it easy to distinguish them" — which may explain the high absolute scores and compressed range.

• m-eurosat (Acc., S2): OlmoEarth ViT Large achieves 96.3%, OlmoEarth ViT Base achieves 94.7%. Panopticon achieves 95.2%, CopernicusFM achieves 84.7%. OlmoEarth is best, but Panopticon is close (0.5-1.6 points behind depending on variant). DINOv3 Sat ViT 7B achieves 91.3%, suggesting this task benefits from specialized EO pretraining over general vision pretraining.

• CropHarvest-Togo (Acc., S2): OlmoEarth ViT Base achieves 87.3%, the highest result. Galileo ViT Tiny achieves 80.3%, Galileo ViT Base achieves 80.1%, CROMA ViT Base achieves 80.7%. OlmoEarth outperforms the next best by 6.6 percentage points.

• CropHarvest-PRC (Acc., S1+S2): OlmoEarth ViT Tiny achieves 82.4%, OlmoEarth ViT Base achieves 82.0%. TerraMind ViT Base achieves 77.5%, Galileo ViT Base achieves 77.5%, Panopticon achieves 76.5%. The gap is 4.9 points.

• PASTIS (mIoU, S2): OlmoEarth ViT Large achieves 51.8, OlmoEarth ViT Base achieves 50.6. TerraMind ViT Large achieves 41.3, TerraMind ViT Base achieves 40.9, Anysat achieves 41.9, CROMA ViT Large achieves 42.7. OlmoEarth outperforms the next best by 7.9 mIoU points. The paper notes that OlmoEarth Large is "significantly worse" than OlmoEarth Base on embedding-based pixel time series tasks — PASTIS with S1 (29.7 vs. 30.6) — speculating that "this may reflect that we explore the training recipe for the Base model more than Large."

• MADOS (mIoU, S2): OlmoEarth ViT Large achieves 66.4, OlmoEarth ViT Base achieves 67.2. Panopticon achieves 66.1, TerraMind ViT Large achieves 67.5, CROMA ViT Large achieves 66.4. OlmoEarth and TerraMind are essentially tied at the top. Appendix A notes that MADOS "includes a lot of rare classes that greatly affect mIoU in the test set, making metrics highly variable across runs of the same model with different seeds."

• AWF classification (Acc.): On Landsat-8 (L8), OlmoEarth ViT Base achieves 77.0% vs. Panopticon at 66.0% and Galileo at 66.5%. On Sentinel-2 (S2), OlmoEarth ViT Base achieves 77.5% vs. Galileo at 72.5%. On Sentinel-1 (S1), OlmoEarth ViT Base achieves 68.5% vs. CopernicusFM at 59.0%. OlmoEarth is best on all three sensor configurations for this task.

• Nandi classification (Acc.): On Landsat-8, OlmoEarth ViT Base achieves 67.9% vs. Panopticon at 60.4% and Satlas at 61.5%. On Sentinel-2, OlmoEarth ViT Base achieves 74.7% vs. CROMA ViT Large at 68.2% and Galileo at 67.3%. On Sentinel-1, OlmoEarth ViT Base achieves 26.5% vs. CROMA ViT Large at 26.1%. The S1 results are notably low across all models, suggesting this is a fundamentally hard single-sensor configuration for this task.

Cross-model patterns in Table 2:

• Panopticon is the strongest competitor for embedding tasks, achieving best or near-best performance on m-bigearthnet (64.9), m-so2sat (60.5), m-brick-kiln (92.9), m-eurosat (95.2), and several AWF configurations.
• DINOv3 models (general vision pretraining) perform competitively on visually-oriented classification tasks (m-brick-kiln: 88.2-91.4%, m-eurosat: 84.0-91.3%) but lag substantially on tasks requiring temporal understanding — BreizhCrops is 31.3% for all DINOv3 variants (they do not support time series, so each time step is input separately and results are likely degraded).
• Galileo shows strong performance on agriculture-related benchmarks (BreizhCrops, CropHarvest), consistent with its training on crop type data.
• Scale does not consistently help within model families: CROMA ViT Large sometimes underperforms CROMA ViT Base (m-bigearthnet: 59.2 vs. 61.3; m-so2sat: 48.2 vs. 51.3). Similarly, TerraMind ViT Large and Base are often within 1 point of each other. OlmoEarth ViT Large underperforms ViT Base on several tasks (PASTIS S1: 29.7 vs. 30.6; CropHarvest-Togo: 67.6 vs. 80.1; AWF S2: 73.0 vs. 77.5), which the paper attributes to less hyperparameter exploration for the Large variant. This pattern — larger models not reliably outperforming smaller ones — appears across multiple model families and suggests that scaling EO foundation models is non-trivial.

Full Fine-Tuning Results

Table 3 presents the fine-tuning results across 29 tasks, with OlmoEarth achieving the best performance on 19 of 29 tasks. The fine-tuning evaluation adds 10 partner tasks not present in the embedding evaluation (GEA North Africa, Forest Loss Driver, Live Fuel Moisture Content, Mangrove, Marine Infrastructure, Vessel Detection/Type/Length, Solar Farm Detection), and for these tasks only fine-tuning results are reported.

Research benchmark fine-tuning (Table 3, left columns):

• m-bigearthnet (μF1, S2): Satlas Swin Base achieves 72.7, TerraMind ViT Base achieves 72.6, OlmoEarth ViT Large achieves 72.4, OlmoEarth ViT Base achieves 72.0. The top four models are within 0.7 μF1 of each other — essentially tied given the single-point estimates without confidence intervals. TerraMind ViT Large achieves 74.0, which is the highest overall, but OlmoEarth is competitive.

• m-so2sat (Acc., S2): OlmoEarth ViT Base achieves 68.6%, CopernicusFM ViT Base achieves 66.8%, TerraMind ViT Base achieves 66.1%, Panopticon ViT Base achieves 65.4%. OlmoEarth leads by 1.8 points.

• m-brick-kiln (Acc., S2): Multiple models achieve near-ceiling performance — Panopticon: 99.0%, DINOv3 Sat ViT Large: 98.9%, CROMA ViT Base: 98.7%, Anysat: 98.7%, Clay: 98.7%, Satlas: 98.7%, OlmoEarth ViT Base: 98.6%. The compressed range (98.1-99.0%) suggests this benchmark may be saturated for fine-tuned models, consistent with the Appendix A observation about trivial negatives.

• m-eurosat (Acc., S2): OlmoEarth ViT Base achieves 98.7%, Panopticon achieves 98.2%, CopernicusFM achieves 98.5%, Galileo achieves 97.8%. OlmoEarth leads but the margins are small (0.2-0.9 points).

• m-cashewplant (mIoU, S2): OlmoEarth ViT Large achieves 80.6, DINOv3 Sat ViT Large achieves 80.6, TerraMind ViT Large achieves 81.3, Prithvi v2 ViT Huge achieves 81.1. OlmoEarth is competitive but TerraMind and Prithvi v2 are slightly ahead. The paper notes that patch size sensitivity affects this benchmark (Appendix E, Table 8), with OlmoEarth Base improving from 27.7 to 32.3 mIoU in linear probing and from 71.9 to 79.8 in fine-tuning as patch size increases from 4 to 16.

• m-SA-crop-type (mIoU, S2): TerraMind ViT Large achieves 41.1, OlmoEarth ViT Large achieves 40.8, Prithvi v2 ViT Huge achieves 38.8, Galileo ViT Base achieves 35.7. The top models are clustered within 2.3 points.

• PASTIS (mIoU, S2): OlmoEarth ViT Large achieves 66.3, leading the field. TerraMind ViT Large achieves 60.9, Galileo ViT Base achieves 61.2, Anysat achieves 60.9. OlmoEarth's margin is 5.4 points. The Random Init baseline achieves 43.9, confirming that pretraining provides substantial value (22.4 mIoU gain).

• MADOS (mIoU, S2): OlmoEarth ViT Large achieves 81.8, the highest result. TerraMind ViT Base achieves 73.2, Panopticon achieves 72.8, Galileo ViT Base achieves 71.9. OlmoEarth's margin is 8.6 points over the second-best model.

• Sen1Floods11 (Acc., S1): All models cluster between 77.0% and 79.8%. OlmoEarth ViT Large achieves 79.8%, TerraMind ViT Base achieves 79.5%, CROMA ViT Large achieves 79.4%. The narrow range (2.8 points across all models) supports the paper's observation that "all of the remote sensing models we tested get between 78-80% accuracy, and we find that the accuracy is not well correlated with other benchmarks." The Random Init baseline achieves 77.0%, suggesting pretraining adds at most ~3 points on this task.

Partner task fine-tuning (Table 3, right columns): This is where OlmoEarth shows its strongest relative performance. The partner tasks are the paper's primary claim to real-world relevance, so they deserve careful examination.

• AWF (Acc.): OlmoEarth ViT Base achieves 87.0% (S2) and 86.0% (S2+S1), the highest results. TerraMind ViT Base achieves 84.0% (S2) and 82.0% (S2+S1), Galileo ViT Base achieves 81.0% (S2) and 81.5% (S2+S1). OlmoEarth leads by 3-4 points. On the Landsat-only configuration (L8), the paper reports OlmoEarth ViT Tiny at 82.5% and OlmoEarth ViT Nano at 82.5%, though competing models' L8 results are sparse (only Anysat and Clay report L8 AWF results in the fine-tuning table).

• GEA North Africa (Acc., S2): Galileo ViT Base achieves 62.9%, OlmoEarth ViT Base achieves 62.4%, CopernicusFM achieves 58.8%, Panopticon achieves 54.3%. OlmoEarth is second, 0.5 points behind Galileo. The paper states this is a level-3 IUCN ecosystem classification with 110 categories, making it a challenging fine-grained task.

• Forest Loss Driver (Acc., S2): OlmoEarth ViT Tiny achieves 97.7%, OlmoEarth ViT Base achieves 97.1%, TerraMind ViT Base achieves 96.4%, Panopticon achieves 96.4%. The range is narrow (95.1-97.7%), but OlmoEarth leads. This task involves distinguishing 10 deforestation driver classes from before/after forest loss image pairs, which tests the model's ability to detect change and classify its cause.

• Live Fuel Moisture Content (L1, S2): OlmoEarth ViT Base achieves 18.5 (lower is better for L1 error), OlmoEarth ViT Large achieves 19.9, Galileo ViT Base achieves 20.1, TerraMind ViT Base achieves 24.3. OlmoEarth leads by 1.6 L1 points over Galileo. The AEF Frozen + Decoder baseline achieves 23.1 (Table 7), and the Random Init baseline achieves 20.9, indicating pretraining provides a meaningful but not massive improvement on this regression task. The paper notes this task uses the Globe-LFMC-2.0 dataset with 41,214 examples — one of the larger partner datasets.

• Mangrove (Acc., S2 and S2+S1): OlmoEarth ViT Base achieves 97.6% (S2) and 97.9% (S2+S1), consistent with TerraMind ViT Base at 97.7% (S2) and 96.8% (S2+S1). Galileo achieves 97.3-97.5%. The models are essentially tied — all above 96.3% — suggesting this binary/ternary classification problem is relatively easy for modern foundation models or that the dataset has limited discriminative difficulty. The mangrove case study reports that fine-tuning OlmoEarth on Global Mangrove Watch data achieved 98.1% F1, up from their existing random forest baseline of 95.3%.

• Marine Infrastructure (F1, S2 and S2+S1): OlmoEarth ViT Base achieves 86.3% (S2) and 89.6% (S2+S1), Panopticon achieves 86.4% (S2) and 88.6% (S2+S1), CROMA ViT Base achieves 86.1% (S2). DINOv3 Sat ViT Large achieves only 35.8% (S2+S1) — a dramatic underperformance suggesting these models require multi-temporal context that single-image general vision models lack. The multimodal configuration (S2+S1) consistently improves over S2-only across all models (e.g., OlmoEarth: 86.3% → 89.6%).

• Nandi (Acc.): OlmoEarth ViT Base achieves 75.4% (L8), 79.2% (S1), and 78.8% (S2). The AEF Frozen + Decoder baseline achieves 66.0% (Table 7), and the AEF kNN baseline achieves 55.6%. Full fine-tuning provides substantial gains: OlmoEarth Base achieves 82.2% with full fine-tuning on this task (Table 7), compared to 62.9% for frozen embeddings + decoder. This is consistent with the paper's argument that full fine-tuning is essential for maximizing performance on specific downstream tasks.

• Vessel Detection (F1): On Landsat-8, OlmoEarth ViT Base achieves 74.6% vs. Panopticon at 74.9% and Satlas at 77.6%. On Sentinel-1, Satlas achieves 76.8% vs. OlmoEarth ViT Base at 75.0%. On Sentinel-2, OlmoEarth ViT Base achieves 85.4% vs. TerraMind ViT Base at 83.5% and Galileo ViT Base at 83.1%. OlmoEarth is best on Sentinel-2, competitive on Landsat-8, and slightly behind Satlas on Sentinel-1. The paper notes that for these tasks the input is a single image, making them more similar to standard computer vision object detection.

• Vessel Type (F1, S2): Galileo ViT Base achieves 81.9%, CROMA ViT Base achieves 86.1%, TerraMind ViT Base achieves 84.0%, Panopticon achieves 86.4%. OlmoEarth ViT Base achieves 86.3%. The models in the 81.9-86.4% range are tightly clustered. The DINOv3 Sat ViT Large achieves only 35.8%, again highlighting the gap between general vision models and EO-specific models for tasks requiring domain-specific features.

• Vessel Length (L1, S2): OlmoEarth ViT Base achieves 15.4, Panopticon achieves 17.7, CopernicusFM achieves 16.7, Galileo ViT Base achieves 16.4, Prithvi v2 ViT Huge achieves 17.4. OlmoEarth leads by 1.0 L1 point.

• Solar Farm Detection (mIoU, S2 and S1+S2): OlmoEarth ViT Large achieves 84.2% (S2) — though this cell is empty in the table, the pattern across the row suggests 84.2% may refer to the OlmoEarth ViT Large S2 result, but the table has --- for the Large variant on S2 — the Base variant achieves 85.4% (S2) and 86.7% (S1+S2). TerraMind ViT Base achieves 83.5% (S2) and 82.1% (S1+S2), Galileo ViT Base achieves 83.1% (S2) and 85.1% (S1+S2). The AEF Frozen + Decoder baseline achieves 77.5% (Table 7), and the AEF full fine-tuning is "Not Possible," while OlmoEarth full fine-tuning achieves 86.7% (S1+S2) vs. 84.8% for frozen OlmoEarth + decoder.

Random Init baseline (Row 1 of Table 3 fine-tuning section). The randomly initialized OlmoEarth ViT Base provides a lower bound on what the architecture can achieve without pretraining. Key comparisons:

• m-bigearthnet: 61.0 (Random Init) vs. 72.0 (OlmoEarth Base) → +11.0 points from pretraining
• m-so2sat: 48.9 vs. 68.6 → +19.7 points
• m-eurosat: 80.3 vs. 98.7 → +18.4 points
• PASTIS S2: 43.9 vs. 64.3 → +20.4 mIoU
• Sen1Floods11: 77.0 vs. 79.8 → +2.8 points
• LFMC L1: 20.9 vs. 18.5 → +2.4 points (lower is better)
• Mangrove S2: 96.3 vs. 97.6 → +1.3 points

Pretraining provides the largest gains on tasks with limited training data and complex semantic content (PASTIS, m-so2sat, m-eurosat), and the smallest gains on tasks that are either saturated (Sen1Floods11 ~78-80% for all models, Mangrove ~97% for all models) or where visual features alone may be sufficient (Sen1Floods11 water segmentation). The Solar Farm Random Init result (74.1% mIoU S2, 70.3% S1+S2) vs. pretrained (85.4%, 86.7%) shows pretraining provides +11.3 and +16.4 mIoU points respectively.

Pareto Efficiency Analysis (Figure 1)

Figure 1 plots average performance across 13 embedding tasks vs. multiply-accumulate operations (MACs) to encode one example, aggregated across tasks where input size varies. The paper claims that "OlmoEarth defines a Pareto optimum of performance vs. computational efficiency."

From the plot (described in text but not numerically tabulated in the paper), the pattern is:

• OlmoEarth ViT Nano (1.4M parameters, lowest MACs): achieves the highest performance per MAC among small models, outperforming Galileo ViT Nano and Presto ViT Nano which operate in similar MAC ranges but with lower average accuracy.
• OlmoEarth ViT Tiny (6.2M parameters): outperforms Galileo ViT Tiny at similar MAC count, and is competitive with larger models like CROMA and CopernicusFM ViT Base despite using fewer MACs.
• OlmoEarth ViT Base (90M parameters): achieves the highest average performance in the ~90M parameter class, above Panopticon ViT Base, TerraMind ViT Base, and Galileo ViT Base, at comparable MACs.
• DINOv3 models occupy an interesting position: the ViT 7B variant achieves high performance (competitive with Galileo/CROMA Base) but at massive computational cost, placing it far from the Pareto frontier. DINOv3 Sat variants shift leftward (lower MACs, similar or better performance), but do not surpass OlmoEarth on average performance at any scale.
• Clay ViT Large and Satlas Swin Base fall inside the frontier — lower performance than OlmoEarth at comparable or higher MAC counts.

The key takeaway is that OlmoEarth achieves best-in-class performance at each model size tier, with the gap being most pronounced at the Nano and Tiny scales where efficient representation matters most for practical deployment. The paper does not provide the underlying numerical values for this plot, making it difficult to assess the precise magnitude of the efficiency advantage.

---

Ablation Studies and Robustness Checks

Full Latent MIM vs. Latent MIM Lite (Table 4, rows 1-2): The standard Latent MIM baseline (learned target encoder) collapsed during training and achieved only 32.2% on m-so2sat, 68.4% on m-eurosat, and 7.9 mIoU on PASTIS. Switching to Latent MIM Lite (frozen random projection) raised these to 42.2% (+10.0 pp), 87.2% (+18.8 pp), and 35.2 mIoU (+27.3). The PASTIS improvement is the most dramatic — from near-random (7.9 mIoU for 19 classes) to a functional model — confirming that the baseline failure was collapse, not just underperformance. The paper notes that these ablation models are trained for only 140,000 steps (21% of full training), so absolute numbers are lower than the final model but relative comparisons remain informative.

Adding modality-aware masking (Table 4, row 3): On top of Latent MIM Lite, adding modality masking (the ability to mask entire bandsets as "decode only") improves m-so2sat from 42.2% to 53.6% (+11.4 pp), m-eurosat from 87.2% to 90.2% (+3.0 pp), and PASTIS from 35.2 to 46.6 mIoU (+11.4). This is the single largest incremental improvement in the development trajectory for classification tasks, suggesting that the cross-modal reconstruction requirement fundamentally changes what the model learns.

Adding modality patch discrimination loss (Table 4, row 4): Further adding the bandset-restricted contrastive loss improves m-so2sat from 53.6% to 55.3% (+1.7 pp), m-eurosat from 90.2% to 91.5% (+1.3 pp), and PASTIS from 46.6 to 48.1 (+1.5 mIoU). These are incremental but consistent gains.

Adding instance contrastive loss (Table 4, row 5): Adding the SimCLR-style global contrastive loss improves m-so2sat from 55.3% to 56.8% (+1.5 pp), m-eurosat from 91.5% to 92.3% (+0.8 pp), and PASTIS from 48.1 to 49.0 (+0.9 mIoU). The gains are modest, consistent with the loss being a secondary objective scaled by 0.1.

Adding maps data (Table 4, row 6): Incorporating the six supervised map modalities as decoder-only targets improves m-so2sat from 56.8% to 62.4% (+5.6 pp), m-eurosat from 92.3% to 92.9% (+0.6 pp), and PASTIS from 49.0 to 50.7 (+1.7 mIoU). The largest impact is on m-so2sat, an urban land cover classification task where map-derived information (OpenStreetMap roads and buildings, WorldCover land cover) provides strong supervisory signal that transfers to better observational representations even though the encoder never sees map data directly. The small improvement on m-eurosat (already near-ceiling at 92.3%) and moderate improvement on PASTIS suggest the benefit of maps is task-dependent.

Additional ablations on final recipe (Table 6): These ablation models are trained for 300,000 steps (45% of full training) and evaluated across 10 tasks with a larger set of modifications. The "Final Recipe" row achieves the best average rank of 2.9 and the second-best average score of 66.6 (behind "Patch Disc Loss" at 66.2, which uses standard cross-modality patch discrimination instead of the modality-aware variant — this is a notable anomaly where the ablated variant outscores the final recipe on average, though the average rank is worse at 3.0 vs. 2.9).

MAE baseline (Table 6, row 1): A pixel-space masked autoencoder (predicting raw pixel values with Smooth L1 loss) achieves an average rank of 5.1 across 10 tasks and an average score of 63.2, compared to 66.6 for the final Latent MIM Lite recipe (rank 2.9). The gaps are largest on PASTIS S2 (46.6 vs. 52.2 mIoU) and MADOS S2 (68.7 vs. 71.7 mIoU), suggesting pixel reconstruction is particularly weak for dense prediction tasks.

Sentinel-2 only data (Table 6, row 2): Training only on Sentinel-2 observations (removing Sentinel-1, Landsat-8, and all map modalities) reduces the average score from 66.6 to 46.4 and the average rank from 2.9 to last place. The model cannot be evaluated on Landsat-8 tasks (m-forestnet) or Sentinel-1 tasks (PASTIS S1, Sen1Floods11). This demonstrates that multimodal pretraining is essential for OlmoEarth's performance — the model benefits substantially from cross-modal learning even when evaluated on a single modality.

No maps (Table 6, row 3): Removing map modalities while keeping all observation modalities reduces the average score from 66.6 to 64.9 and worsens the average rank from 2.9 to 4.7. The impact is measurable but modest — maps help but are not the dominant source of OlmoEarth's performance. This aligns with Table 4 where adding maps provided the final incremental gain (not the foundational one).

No agricultural maps (Table 6, row 4): Removing only the agricultural map modalities (WorldCereal and Cropland Data Layer) while keeping other maps (OSM, WorldCover, SRTM, Canopy Height) achieves an average score of 66.1 and rank of 3.6. This is intermediate between "No Maps" (64.9, rank 4.7) and "Final Recipe" (66.6, rank 2.9), suggesting agricultural maps contribute to performance but are not uniquely critical.

Random masking (Table 6, row 5): Replacing modality-aware masking with standard random masking (all bandsets always "encode and decode") reduces the average score from 66.6 to 65.4 and worsens the rank from 2.9 to 4.7. The largest impact is on PASTIS S2 (51.1 vs. 52.2) and PASTIS S1 (24.7 vs. 29.4), consistent with the idea that forcing cross-modal reconstruction benefits structured prediction tasks more than simple classification.

No instance contrastive loss (Table 6, row 6): Removing the auxiliary contrastive loss reduces the average score from 66.6 to 65.9 and worsens the rank from 2.9 to 4.7. The impact is diffuse — no single task is dramatically affected, but overall performance degrades slightly.

Standard patch discrimination loss (Table 6, row 7): Using standard patch discrimination (contrasting against all target tokens regardless of modality) instead of modality-aware patch discrimination achieves an average score of 66.2 (slightly HIGHER than the final recipe's 66.6? — no, 66.2 is lower) and a rank of 3.0 vs. 2.9. The differences are small: m-so2sat is 62.1 vs. 65.9 (final recipe is better), m-eurosat is 94.0 vs. 94.6 (final recipe is better), PASTIS S2 is 50.0 vs. 52.2 (final recipe is better), but m-brick-kiln is 96.3 vs. 94.2 (standard patch disc is better) and MADOS is 74.1 vs. 71.7 (standard patch disc is better). The mixed results suggest the modality-aware loss provides a small net benefit that is task-dependent.

Patch size sensitivity on m-cashewplant (Appendix E, Table 8): For OlmoEarth ViT Base, linear probing mIoU improves from 27.7 (patch size 4) to 27.9 (patch size 8) to 32.3 (patch size 16). Fine-tuning mIoU improves from 71.9 to 76.2 to 79.8. Galileo shows the same trend: 24.3 → 25.6 → 28.9 for LP, 73.0 → 76.9 → 78.8 for FT. The paper hypothesizes this is due to "spatially coarse labels in the dataset, which are polygons instead of pixels" — larger patch sizes naturally produce coarser predictions that better match the label granularity. This is an unusual finding since "a smaller patch size typically improves performance" (citing Figure 4 of Tseng et al., 2025), and it highlights the importance of matching model resolution to label characteristics.

AlphaEarth Foundations comparison (Appendix D, Table 7): This ablation compares OlmoEarth against AEF embeddings across five tasks using three training strategies: kNN, frozen backbone + decoder, and full fine-tuning. Key findings:

• kNN: OlmoEarth outperforms AEF on Nandi (66.2 vs. 55.6) and AWF (82 vs. 81), while AEF outperforms OlmoEarth on Ecosystem classification (60.6 vs. 59.3). This is a fair embedding-to-embedding comparison since both use only the precomputed features.
• Frozen + Decoder: OlmoEarth achieves 62.9 (Nandi), 84.0 (AWF), 61.1 (Ecosystem), 19.9 L1 (LFMC), 84.8 mIoU (Solar Farm). AEF achieves 66.0, 75.9, 61.2, 23.1, 77.5. OlmoEarth is better on 3 of 5 tasks.
• Full Fine-tuning: AEF cannot be fine-tuned (embeddings only). OlmoEarth achieves 82.2 (Nandi), 86.0 (AWF), 62.4 (Ecosystem), 17.9 L1 (LFMC), 86.7 mIoU (Solar Farm). The gains over Frozen + Decoder are substantial: +19.3 pp on Nandi, +2.0 pp on AWF, +1.3 pp on Ecosystem, +2.0 L1 on LFMC, +1.9 mIoU on Solar Farm. This directly supports the paper's claim that "full fine-tuning enables even better results" and that precomputed embeddings are limiting.

Negative result: OlmoEarth Large does not reliably outperform OlmoEarth Base. The paper notes this explicitly: "OlmoEarth Large does not always outperform OlmoEarth Base, and for embedding-based pixel time series tasks it is significantly worse." In Table 2, OlmoEarth Large is worse than Base on CropHarvest-Togo S1 (67.6 vs. 80.1, a 12.5 pp gap), CropHarvest-Togo S1+S2 (78.1 vs. 82.0, a 3.9 pp gap), AWF S2 (73.0 vs. 77.5), and PASTIS S1 (29.7 vs. 30.6). In Table 3, OlmoEarth Large is worse than Base on m-forestnet (52.7 vs. 51.2? No, Large is 52.7, Base is 51.2 — Large is actually better here), but the broader pattern is inconsistent scaling. The paper attributes this to "this may reflect that we explore the training recipe for the Base model more than Large," and notes that similar patterns appear in TerraMind and CROMA (where Base models sometimes outperform Large), so this "may reflect the challenges of scaling Earth observation models." This is an important finding: model scaling in EO does not follow the relatively smooth scaling laws observed in language and vision, and larger models require careful tuning to realize their potential advantage.

Negative result: ReST^EM-style revision training degrades performance (Appendix K reference in prior sections). The paper notes in a prior section that optimizing the revision model with RL-style training caused performance to degrade, but this result appears in the context of a different model family (revision models for language) and is not an OlmoEarth-specific result. There is no equivalent negative result for OlmoEarth training — the paper does not report any attempt to use RL-based fine-tuning or iterative self-training for OlmoEarth.

---

Critical Assessment

Claim 1: "OlmoEarth achieves state-of-the-art performance compared to 12 other foundation models"

What the experiments show. Table 2 demonstrates OlmoEarth is best on 15 of 24 embedding tasks, and Table 3 demonstrates best on 19 of 29 fine-tuning tasks. This is a legitimate claim under the paper's evaluation protocol. However, "best" is defined as the highest single-point estimate without confidence intervals or statistical tests, and the margin of superiority varies dramatically across tasks. On m-brick-kiln fine-tuning, OlmoEarth achieves 98.6% while Panopticon achieves 99.0% — OlmoEarth is not best but the difference is 0.4 percentage points, which is almost certainly within noise. On PASTIS S2 fine-tuning, OlmoEarth achieves 66.3 mIoU vs. the next-best at 60.9 (TerraMind Large) — a 5.4 point gap that is likely meaningful. The paper does not provide any formal ranking aggregation, statistical significance assessment, or correction for multiple comparisons across 24+29=53 tests, which makes the binary "best on N tasks" framing potentially misleading for the many tasks where the top 3-4 models are within 1-2 points of each other.

What the experiments do not show. The paper evaluates only the models that the authors chose to import into their framework. There is no guarantee this is the complete set of state-of-the-art EO foundation models, and models released after the paper's submission are obviously not included. More importantly, the paper uses "the same training recipes for each foundation model but sweep a variety of hyperparameters to find the best performance for each model on each task." This is a double-edged sword: it makes comparison fairer by controlling for training protocol, but it also means the reported numbers for competing models reflect the paper's training recipe, not the recipe those models' authors would have chosen. If a model was designed to work best with a different optimizer, learning rate schedule, or data augmentation, the paper's uniform protocol may disadvantage it. The paper does not discuss this tradeoff.

A particular concern: for fine-tuning partner tasks, the paper uses a learning rate of 1×10⁻⁴ for all models on all tasks except Nandi. The competing models were likely designed with different optimal learning rates, and sweeping only over {1×10⁻⁴, 5×10⁻⁴, 1×10⁻³} for research benchmarks (and not at all for most partner tasks) may not find each model's optimal configuration. This is a reasonable practical constraint but undermines the claim that the comparison identifies which model is inherently "best."

Claim 2: "Latent MIM Lite eliminates training instability while maintaining representational power"

What the experiments show. Table 4 clearly demonstrates that the baseline Latent MIM (learned target encoder) collapsed during training, achieving 7.9 mIoU on PASTIS vs. 35.2 for Latent MIM Lite. The claim that Latent MIM Lite is stable is supported — there is no evidence of collapse in any of the ablation or final models that use the frozen random projection. The claim that it "maintains representational power" is supported by the consistently strong downstream results across 53 tasks, the Pareto frontier position in Figure 1, and the ablation showing substantial improvements over the MAE baseline (Table 6).

What the experiments do not show. The paper demonstrates stability for one training run per variant. True stability — in the sense of reliable replication across random seeds, across different pretraining datasets, and across different hardware configurations — would require multiple training runs with different randomness, which the paper does not perform (or at least does not report). The paper also does not systematically characterize why the learned target encoder collapses. Is it a property of the data, the model architecture, the optimizer, the learning rate? Does collapse happen gradually (performance degrades over training) or suddenly (a phase transition at some step)? The paper asserts that collapse occurs and that the frozen projection fixes it, but provides no diagnostic analysis (e.g., representation rank plots, gradient norm trajectories, target encoder parameter trajectories) that would help other researchers understand or predict when collapse will occur.

The paper also does not compare against TerraMind's alternative stability solution — a frozen quantized autoencoder tokenizer — which is conceptually the closest prior work to Latent MIM Lite. TerraMind is included in the evaluation tables, but there is no ablation or analysis comparing the two stabilization mechanisms. A head-to-head comparison of frozen random projection vs. frozen VQ tokenizer on the same data and architecture would significantly strengthen the claim that Latent MIM Lite is the preferred approach.

Claim 3: "Modality-aware masking and bandset-restricted contrastive loss are key innovations that improve performance"

What the experiments show. Table 4 shows clear incremental gains as each component is added: modality masking adds +11.4 mIoU on PASTIS, modality patch discrimination adds +1.5, contrastive loss adds +0.9, maps add +1.7. Table 6 shows that removing each component from the final recipe degrades performance (average rank worsens from 2.9 to 3.0-5.1 depending on the ablation). The components contribute, and the ablation experiments are well-designed to isolate their effects.

What the experiments do not show. The ablation experiments in Table 4 are trained for only 140,000 steps (21% of full training). It is unclear whether the relative importance of each component changes at full training duration — some components might provide early-training benefits that diminish, while others might become more important later. Table 6 uses 300,000 steps (45% of full training), which is better but still not the full budget. The paper's claim that "improvements on a representative subset carried over to the full evaluation" is reasonable but unverified without full-training ablations for at least the most important components.

Additionally, the ablation experiments test adding/removing components but do not explore interactions between components. For instance, does modality-aware masking provide the same benefit with standard patch discrimination loss as with modality patch discrimination? Does the instance contrastive loss matter more or less when maps are included? The paper's additive/removal ablation design cannot detect such interactions, and there may be configurations where a component only helps (or only hurts) in the presence of another component.

The standard patch discrimination vs. modality patch discrimination comparison (Table 6) is particularly interesting because the results are mixed — the standard variant is actually better on m-brick-kiln (96.3 vs. 94.2) and MADOS (74.1 vs. 71.7). This suggests the bandset restriction is not universally beneficial and may help on some tasks while hurting on others, which the paper does not discuss.

Claim 4: "OlmoEarth Platform enables non-profits and NGOs to use foundation models for real-world tasks"

What the experiments show. Section 5.1 describes two case studies: Global Mangrove Watch achieved 98.1% F1 using OlmoEarth fine-tuning (up from 95.3% with their existing random forest), and Global Ecosystem Atlas labeled more than 15,000 data points using the platform over three months. Appendix B lists 7 partner organizations who provided data for evaluation. These are concrete demonstrations of real-world use.

What the experiments do not show. The case studies are preliminary and limited in scope. The mangrove result is a single F1 score without details on the training setup, test set construction, or comparison to alternative approaches the organization might have taken (e.g., fine-tuning a different foundation model, improving their random forest, using a non-ML approach). The ecosystem mapping result is described in terms of labeling throughput, not model performance or downstream map quality. Neither case study provides the kind of rigorous comparison (controlled experiment, multiple baselines, statistical analysis) that would establish the platform's superiority over alternatives for these specific use cases.

More fundamentally, the paper does not evaluate whether the platform actually solves the accessibility problem it identifies. The claim is that non-profits "lack GPU infrastructure or deep learning expertise," but the paper provides no evidence that the platform's target users can successfully use it without assistance. The case studies involve organizations that partnered directly with the authors — this is not the same as an independent non-profit discovering the platform, onboarding themselves, and successfully deploying a model. A user study or deployment report with partners who used the platform independently would substantiate this claim.

Claim 5: "OlmoEarth defines a Pareto optimum of performance vs. computational efficiency"

What the experiments show. Figure 1 plots average performance on 13 embedding tasks against MACs for encoding, and OlmoEarth models occupy the upper-left frontier (best performance per MAC across the evaluated model set). At each model size (Nano, Tiny, Base), OlmoEarth achieves higher average performance than comparably-sized or comparably-expensive alternatives.

What the experiments do not show. The Pareto plot aggregates performance across 13 tasks by simple averaging. Tasks have different difficulty levels, different metric scales (accuracy vs. μF1 vs. mIoU), and different practical importance. Averaging them equally assumes all tasks are equally informative about model quality, which is unlikely to be true. The paper explicitly notes that some benchmarks have limitations (m-brick-kiln has "too easy" negatives, Sen1Floods11 has near-identical performance across all models, MADOS has high variance), but all 13 tasks contribute equally to the average. A weighted average based on task difficulty or practical relevance might produce different Pareto frontiers.

The MAC count is computed per-example with actual input dimensions, which is appropriate. However, the input size varies by task (as the paper notes), and models that are more efficient at some input sizes might be less efficient at others. The Pareto plot aggregates across these varying sizes, obscuring task-specific efficiency tradeoffs. A model that is efficient on small-input tasks (classification) but inefficient on large-input tasks (segmentation) could still appear on the Pareto frontier.

Finally, the Pareto frontier depends entirely on which models are included in the comparison. The paper includes 12 other foundation models, but there may be additional models (released concurrently or after the paper's experiments were conducted) that would shift the frontier. The paper does not discuss how the frontier might change with future models, which limits the durability of this finding.

Missing Experiments That Would Strengthen the Paper

1. Multi-seed training runs. The stability claim would be substantially strengthened by demonstrating that the same training recipe converges reliably across multiple random seeds. A single training run per model size cannot distinguish between genuine stability and a lucky initialization. The fact that Table 4's baseline Latent MIM collapsed could itself be partly a result of unlucky initialization — multiple seeded runs would clarify whether collapse is deterministic or stochastic.

2. Learning rate and batch size scaling for the Large model. The paper notes that OlmoEarth Large underperforms Base on several tasks and attributes this to less hyperparameter exploration. A learning rate sweep for Large would clarify whether the underperformance is a fundamental scaling problem or a tuning artifact. The current evidence is insufficient to determine whether EO foundation models genuinely exhibit diminishing returns to scale, or whether the paper simply did not find the right hyperparameters for the larger model.

3. Data quantity and diversity ablations. The paper uses 285,288 pretraining samples. How does performance scale with dataset size? Would half the data produce similar results? Would double the data produce better results? The paper includes an ablation with "Only S2 Data" (Table 6) showing dramatic degradation, but does not vary the total quantity of multimodal data. Understanding data scaling behavior is important for practitioners deciding how much data to curate.

4. Ablation of modality count. Does each additional modality provide diminishing returns? The paper includes ablation of "No Maps" vs. "No Agricultural Maps" vs. full maps, but does not systematically remove individual observation modalities (Sentinel-1 only, Sentinel-2 only, Landsat-8 only, S1+S2, etc.). The "Only S2 Data" ablation removes everything including maps, so we cannot determine whether the degradation comes from losing Sentinel-1, Landsat-8, maps, or all three.

5. Temporal sequence length and sampling strategy ablation. The paper uses up to 12 monthly timesteps and samples 3-12 during training. How does performance vary with the number of timesteps available at inference? Does monthly sampling matter, or would quarterly be sufficient? The paper does not ablate the temporal dimension, which is surprising given that many of the partner tasks (crop type mapping, deforestation monitoring) are inherently temporal.

6. Direct comparison with TerraMind's VQ tokenizer approach. Since TerraMind is the closest prior work that also addresses training stability through a frozen tokenizer, a controlled comparison — same data, same architecture, same training budget, comparing frozen random projection vs. frozen VQ tokenizer vs. learned target encoder — would be highly informative. The current evaluation compares full models with all their architectural differences, making it impossible to isolate the effect of the stabilization mechanism.

7. Fine-tuning hyperparameter sweep for all models on all tasks. The paper uses a learning rate of 1×10⁻⁴ for all models on partner tasks (except Nandi). If competing models would benefit from different learning rates — which is highly likely given architectural differences — the current protocol may systematically disadvantage them. A broader sweep for at least the top-performing competing models would make the comparison fairer and the conclusions more robust.

8. Statistical significance and confidence intervals. Every number in Tables 2 and 3 is a single-point estimate. For tasks where the top 3 models are separated by less than 1-2 points (m-bigearthnet fine-tuning: 72.7 vs. 72.6 vs. 72.4; m-brick-kiln embedding: 96.2 vs. 92.9 vs. 92.2), it is impossible to determine whether the observed ranking is real or noise. Reporting standard deviations across multiple fine-tuning runs (even just 3 seeds per task for the top few models) would substantially increase confidence in the rankings.

Where the Claims Hold Conditionally

The claim that OlmoEarth is "state-of-the-art" holds under the specific evaluation protocol used in the paper — same training recipes for all models, hyperparameter sweeps within prescribed ranges, no statistical testing — but may not hold under alternative protocols, and the single-point estimates mask substantial uncertainty on many tasks.

The claim that Latent MIM Lite eliminates training instability holds for the specific architecture, data, and hyperparameters used in the paper, but has not been demonstrated to generalize to other architectures (non-ViT encoders), other data distributions (non-EO modalities), or other hyperparameter configurations (different learning rates, batch sizes, optimizers).

The claim that modality-aware masking and bandset-restricted loss improve performance holds for EO data where multimodal redundancy is high, but the paper explicitly notes these innovations may not transfer to domains with less structured multimodal relationships. The ablation results suggest the benefits are real but the magnitude is moderate for the loss modification specifically.

The claim about real-world impact through the platform is supported by existence proofs (case studies) but not by systematic evaluation — the platform may enable non-profits to use foundation models, but the paper does not provide evidence that this actually happens at scale, independently, or more effectively than alternative approaches.

The claim about Pareto optimality is a snapshot of the model landscape at evaluation time and depends on the specific set of models compared, the specific set of tasks used in the average, and the specific MAC counting methodology. The paper's inclusion of 12 competing models makes this one of the more comprehensive comparisons in the EO foundation model literature, but the frontier is inherently dynamic and the aggregation obscures task-specific tradeoffs.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Headline Efficiency Gains

The assumption or constraint. The entire compute-optimal framework described in the prior sections depends on knowing, in advance, how difficult each prompt is for the base model. The paper acknowledges this cost directly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

Specifically, the method for estimating difficulty — whether oracle (ground-truth pass@1 over 2,048 samples) or predicted (PRM final-answer score averaged over 2,048 samples) — requires generating and scoring 2,048 complete solutions per prompt before any strategy selection occurs. This is 4× to 8× more compute than the largest test-time budgets studied (256–512 generations), meaning the difficulty estimation step alone can dominate the total inference cost.

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment where difficulty is not known in advance, the total cost is difficulty estimation + strategy execution, and the former can be larger than the latter. For example, a prompt that the system ultimately solves with 16 generations of beam search (the compute-optimal allocation) might require 2,048 generations just to determine that beam search with 16 generations is the right choice — yielding a total cost of 2,064 generations, which is substantially worse than simply running best-of-256. The headline efficiency numbers are therefore an upper bound on what is achievable only if difficulty can be estimated cheaply.

What evidence exists in the paper. The paper does not measure the amortized cost of difficulty estimation in any experiment. The compute-optimal scaling curves (Figures 4 and 8) plot performance against the strategy execution budget only, with difficulty treated as known. The authors acknowledge this gap explicitly (Section 3.2) and frame it as an exploration-exploitation tradeoff, but provide no empirical quantification of how the efficiency gains change when difficulty estimation compute is included. There is no ablation studying how performance varies with the number of samples used for difficulty estimation — 2,048 is used throughout with no sensitivity analysis.

Mitigation status. The paper acknowledges this as "a key avenue for future work" and suggests training models to predict difficulty directly from question text, or using adaptive schemes that interleave difficulty estimation with problem-solving. Neither approach is developed or evaluated. Until this gap is closed, the 4× figure should be understood as a measure of potential efficiency given perfect difficulty information, not realized deployment gain.

---

Hard Problems Remain Fundamentally Beyond the Reach of Test-Time Compute

The assumption or constraint. The paper's entire framework operates on the premise that the base model can produce correct solutions at some non-trivial rate. When this fails — when the model's pass@1 on a problem class is effectively zero — no amount of test-time compute helps. This is not a bug in the method but a fundamental bound: search and revision can only amplify existing capability, not create it from nothing.

The consequence. Across all methods studied — search against the PRM, iterative revisions, and their compute-optimal combinations — the hardest difficulty quintile (bin 5) shows near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets up to 256. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line for revisions is essentially flat near 0–5%, consistently below the 14× larger model's greedy performance at all values of R. For PRM search, bin 5 shows a relative disadvantage of −52.9% at R ≫ 1 compared to the larger model. This means that for genuinely challenging problems — those requiring reasoning capabilities the base model simply does not possess — the compute-optimal framework offers no path forward.

What evidence exists in the paper. The difficulty-bin breakdowns across Figures 3, 7, and 9 consistently show bin 5 performance that is flat across all budgets and strategies. The authors are transparent about this in the FLOPs-matched analysis (Section 7), stating in the takeaway box that "on the hardest questions, the improvement from test-time compute is minimal and scaling pretraining is more effective." Table 2 in the prior sections provides additional evidence in the Earth observation domain: on tasks where no model achieves strong performance (e.g., Nandi S1 classification where all models score 20–27%), the ranking between models is essentially noise.

Mitigation status. The paper does not attempt to solve this problem. It accurately characterizes the limitation and draws the appropriate conclusion — that for problems outside the base model's capability range, scaling pretraining remains the only viable path. This is a principled limitation, not a fixable flaw, but it means the approach provides no benefit for the specific use cases (novel reasoning, out-of-distribution generalization) where foundation models are most needed. Future work combining test-time compute with retrieval-augmented generation or tool use might extend capability boundaries, but the paper does not explore this direction.

---

The Method Has Been Validated on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the paper — the search algorithm comparison, the revision model analysis, the compute-optimal policy derivation, the FLOPs-matched pretraining comparison — uses a single benchmark (MATH, Hendrycks et al., 2021) and a single model family (PaLM 2-S*, Codey variant, Anil et al., 2023). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified.

The consequence. Several aspects of the paper's findings could be model- or domain-specific, and the paper provides no evidence for generalization:

• PRM over-optimization behavior (Section 5.3): The specific budget thresholds at which beam search degrades easy-problem performance depend on the PRM's calibration, which in turn depends on the base model's output distribution. A model with better-calibrated confidence or different error patterns might exhibit different over-optimization curves, shifting the optimal strategy per difficulty bin.
• Revision model effectiveness (Section 6): The revision model's ability to learn from incorrect in-context examples — and the specific 38% correct-to-incorrect reversion rate — depends on the base model's in-context learning capabilities. Model families with stronger or weaker in-context learning would likely show different revision dynamics.
• Difficulty-dependent scaling curves: The finding that easy problems benefit from revisions while medium problems benefit from search is a function of what the base model finds easy vs. medium. A different model might have a different capability profile, changing which strategies are optimal at which difficulty levels. The five-quintile difficulty bin boundaries are entirely specific to PaLM 2-S*'s pass@1 distribution on MATH.

What evidence exists in the paper. None. There are no experiments on other benchmarks (e.g., GSM8K for math reasoning, HumanEval for code generation, logical reasoning datasets), and no experiments with other model families (e.g., LLaMA, GPT, Mistral). The paper's claims about general principles — that difficulty-conditioned allocation is key, that verifier over-optimization limits search, that revisions and search complement each other — are supported only for this specific model-benchmark pair.

Mitigation status. The paper does not address this limitation. The cross-validation protocol (Section 3.2) ensures the compute-optimal policy does not overfit to the single MATH test set of 500 questions, but it does nothing to establish generalization to other benchmarks or model families. A reader deploying these methods on a different model or task domain would need to re-derive the optimal strategies from scratch, which requires the expensive difficulty estimation step the paper acknowledges is a bottleneck. The paper's compute-optimal policy lookup tables are specific to PaLM 2-S* on MATH and do not transfer.

---

Sequential Revision Strategies Introduce Latency Penalties Not Captured by the Generation Budget Model

The assumption or constraint. The paper measures test-time compute exclusively in "generations" — the number of complete solutions sampled. This is a reasonable proxy for total FLOPs but abstracts away a critical practical distinction: parallel generation strategies (best-of-N, parallel chains) can execute simultaneously on sufficient hardware, while sequential strategies (revision chains, sequential revision within a chain) are inherently serial — each step depends on the output of the previous step.

The consequence. The compute-optimal policies derived in Sections 5 and 6 favor sequential-heavy strategies in several regimes:

• On easy problems, purely sequential revisions dominate (Figure 7, right, bin 1–2), meaning the model generates one revision, then conditions on it for the next revision, then the next — a chain of 32 or 64 serial forward passes.
• On hard problems, the optimal strategy uses a balanced sequential-to-parallel ratio, meaning some serialism remains even when parallelism is introduced.

A strategy that spends 128 generations as 32 parallel chains of length 4 takes roughly equal wall-clock time to a fully parallel 128-sample best-of-N on hardware with sufficient parallelism. But a strategy that spends 128 generations as a single chain of 128 sequential revisions takes ~128× longer in wall-clock time — each revision must wait for the prior one to complete before generation can begin. For latency-sensitive applications (interactive assistants, real-time decision-making, any user-facing system), this serial dependency may make the compute-optimal policy impractical regardless of its accuracy advantage.

What evidence exists in the paper. None. The paper does not discuss latency, wall-clock time, or throughput in any context. The generation budget model treats all generations as interchangeable units of compute, ignoring the serial vs. parallel distinction entirely. This is a reasonable abstraction for FLOPs-based analysis and batch inference where total throughput matters more than per-query latency, but it omits a dimension that is often decisive in deployment decisions.

Mitigation status. Not addressed. The paper does not acknowledge latency as a relevant constraint, nor does it discuss how the compute-optimal policies would change if a latency budget (maximum number of sequential steps) were imposed alongside the generation budget. A practitioner deploying the methods would need to independently evaluate whether the accuracy gains from sequential revisions justify the latency cost for their specific use case, with no guidance from the paper.

---

The 14× Larger Model Baseline Is Not Compute-Optimal, Weakening the Pretraining-Vs-Inference Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) where model size increases but data quantity stays constant. The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This departs from the Chinchilla scaling laws (Hoffmann et al., 2022), which prescribe scaling data and parameters equally for compute-optimal pretraining. A Chinchilla-optimal model trained with 14× more total FLOPs would allocate some of that budget to more training data, likely outperforming a parameter-only-scaled model at the same total FLOPs.

The consequence. The reported advantages of test-time compute over pretraining in the FLOPs-matched comparison — such as +27.8% relative improvement on easy questions at R ≪ 1 for revisions — are measured against a pretraining baseline that may be weaker than what compute-optimal pretraining would produce. If a properly Chinchilla-scaled larger model were used as the comparison point, the test-time compute advantage would likely shrink or potentially reverse on some difficulty tiers. The paper's key finding — that "a smaller model augmented with compute-optimal test-time strategies can outperform a ~14× larger pretrained model" — is therefore qualified by the fact that the larger model may not be the best possible model at its compute budget.

Additionally, the 14× larger model uses only greedy decoding in the comparison — no majority voting, no best-of-N, no search, no test-time compute of any kind. Giving the larger model even a modest test-time compute budget (say, best-of-8 with the same PRM) would create a substantially stronger baseline. The paper's framing as "test-time compute vs. pretraining" is not entirely fair, since the larger model is denied the inference-time strategies that the smaller model receives.

What evidence exists in the paper. The paper acknowledges the parameter-only scaling choice explicitly (Section 7) but does not provide any sensitivity analysis — there is no comparison against a Chinchilla-optimal scaled model, nor any ablation where the larger model receives test-time compute. The claim is presented with this caveat in the text, but the headline numbers (the bar charts in Figure 1, the percentages quoted in the abstract and introduction) do not carry the qualification.

Mitigation status. The paper frames this as future work, stating it "leaves the analysis of compute-optimal scaling of pretraining compute... to future work." This is a reasonable scope limitation for a paper introducing a new test-time compute framework, and the authors are transparent about it. However, a reader interpreting the headline finding that test-time compute "can outperform a ~14× larger model" should understand that this has been demonstrated only against a specific, potentially suboptimal, pretraining baseline — not against the best possible use of the same total pretraining FLOPs.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the Earth observation foundation model landscape not by introducing a radically new architecture or objective, but by identifying and surgically fixing the specific failure mode that had made latent-space self-supervised learning unreliable for multimodal EO data. The contribution is best understood as a diagnosis-driven simplification rather than a paradigm shift: the field had been adding complexity to manage training instability (TerraMind's quantized autoencoder tokenizers, Galileo's careful hyperparameter engineering, various exponential moving average schemes), while the actual problem — co-adaptation between target and predictor networks — could be eliminated entirely by removing the learned target encoder and substituting a frozen random projection. This is a conceptual reframing of the stability problem from "we need better ways to learn stable targets" to "we don't need learned targets at all."

The magnitude of this shift is substantial but bounded. It is substantial because the failure mode was genuinely blocking progress: Table 4 shows standard Latent MIM achieving 7.9 mIoU on PASTIS — a collapsed model that is essentially useless — while the frozen-projection variant achieves 35.2, and the full recipe reaches 50.7. This is not a marginal improvement; it is the difference between a non-functional training run and a state-of-the-art model. Prior work had documented these instabilities (Mo et al., 2024; Assran et al., 2023) but had not isolated the cause so cleanly nor provided a fix this simple. The fact that the fix is a deletion (removing the learned target encoder) rather than an addition (another regularizer, another normalization layer, another training stage) makes it particularly impactful: it reduces both conceptual and engineering complexity.

It is bounded because the finding may be domain-specific. The paper is explicit: "While it's possible this approach is too simplistic in more diverse domains like natural image processing, empirical results show a clear benefit in our domain of Earth observation data." For EO — where the data is highly structured, modalities are limited in diversity, and spatial and temporal redundancy is high — random projections preserve sufficient structure to serve as meaningful targets. For natural images with arbitrary content and high diversity, random projections may not capture enough semantic information, and learned targets may be genuinely necessary. The paper does not claim universality; it claims domain-specific effectiveness with a clear hypothesis about why.

The work reconciles a contradiction in the EO foundation model literature that had been simmering but not explicitly articulated. Several leading models — Galileo, Anysat, Panopticon — had adopted latent-space objectives and reported strong results, yet replication attempts frequently encountered instability. The paper's own experience captures this: "When attempting to replicate existing work we frequently saw training instability, representation collapse, and models underperforming their stated potential." The resolution is that these models can work well — the paper's evaluation shows Galileo and Panopticon achieving competitive results on many tasks — but their training is fragile in ways that are not captured by the final reported numbers. A model that achieves state-of-the-art results when training succeeds but collapses 30% of the time is not practically equivalent to one that trains reliably every time. OlmoEarth's contribution is to make latent-space training reliable without sacrificing the representational quality that motivated the shift away from pixel-space reconstruction in the first place. This converts latent-space EO pretraining from a high-risk/high-reward research activity into a reproducible engineering procedure.

The paper also reshapes the evaluation landscape for EO foundation models. By running 12 competing models through the same training recipes, hyperparameter sweeps, and diverse downstream tasks (24 embedding evaluations, 29 fine-tuning evaluations), the paper provides the most comprehensive and directly comparable assessment of EO foundation models to date. Prior work had evaluated on different benchmarks, used different training protocols, and reported different metrics, making model comparison essentially impossible. The paper's Tables 2 and 3 — while not without limitations (single-point estimates without confidence intervals, uniform training recipes that may disadvantage some models) — establish a de facto standard that future work can build on and improve. A researcher developing a new EO foundation model now has a clear target: outperform OlmoEarth on this specific, reproducible evaluation suite. This is a practical infrastructure contribution that may prove as impactful as the technical innovations.

The paper also validates that supervised maps can be integrated into self-supervised pretraining as decoder-only targets, with the encoder never seeing map data. This is a design choice that several prior models (Galileo, TerraMind) did not make — they allowed maps as encoder inputs — and the paper's strong empirical results (Table 2, 3) combined with the ablation showing maps add measurable value (Table 4, +5.6 pp on m-so2sat, +1.7 mIoU on PASTIS) demonstrate that this cleaner separation works. This has implications beyond EO: any domain where some modalities are reliably available at training time but potentially missing or stale at inference time (medical imaging with multiple scan types, robotics with privileged simulator information) could adopt the same encode-only vs. decode-only modality partitioning.

Finally, the paper establishes that model scale does not monotonically improve performance for EO foundation models under current training paradigms. OlmoEarth Large underperforms OlmoEarth Base on multiple tasks (CropHarvest-Togo S1: 67.6 vs. 80.1; AWF S2: 73.0 vs. 77.5; PASTIS S1: 29.7 vs. 30.6), and similar patterns appear in CROMA (Base often outperforms Large) and TerraMind. The paper attributes this to less hyperparameter exploration for larger models, but the pattern across multiple model families suggests a deeper issue: scaling laws that are well-characterized for language and vision may not transfer cleanly to multimodal EO data, where the effective "data diversity" is limited by the physical constraints of Earth's surface and where larger models may overfit to spurious correlations in the pretraining distribution. This finding — that bigger is not always better — should shift research attention from scaling model size toward improving training stability, data quality, and evaluation robustness, where the paper demonstrates clearer gains.

Follow-Up Research This Work Enables

1. Systematic characterization of when random projection targets are sufficient vs. when learned targets are necessary. The paper demonstrates that frozen random projections work well for multimodal EO data but explicitly hedges on generalization: "it's possible this approach is too simplistic in more diverse domains like natural image processing." A controlled study across domains of varying diversity — satellite imagery (low diversity, this paper), aerial photography (medium diversity), natural images (high diversity), medical imaging (structured but potentially diverse) — using identical architecture and training protocols, comparing frozen random projection targets against learned target encoders (EMA-based, VQ-based), would establish the boundary conditions for Latent MIM Lite. The key measurement: at what level of data diversity does the frozen projection's representational quality degrade below the learned target encoder's quality, and does collapse risk increase or decrease with diversity? The paper's Table 6 provides a starting point: the MAE baseline (pixel reconstruction, no latent space) achieves meaningful but weaker performance (average rank 5.1 vs. 2.9), suggesting that even in a worst-case scenario where latent modeling fails, pixel-space reconstruction is a functional fallback — but a more precise characterization would tell practitioners whether Latent MIM Lite is safe to use as a default or requires domain-specific validation.

2. Diagnostic toolkit for predicting and detecting representation collapse in latent-space SSL. The paper reports that "Full Latent MIM collapsed during training" but provides no diagnostic analysis — no representation rank plots, no gradient norm trajectories, no target encoder parameter statistics, no mutual information estimates between views. This leaves other researchers in the same position the authors were in: training runs that may or may not collapse, with no early warning signal. A follow-up study instrumenting the training process of both the collapsing (standard Latent MIM) and stable (Latent MIM Lite) variants would identify leading indicators of collapse — metrics that degrade detectably before performance collapses, enabling early stopping or intervention. Candidate diagnostics: the effective rank of the target encoder's output representations (collapse → rank drops to 1), the gradient norm ratio between predictor and target encoder (collapse → ratio diverges), the variance of the target representations across different inputs (collapse → variance shrinks), or the nearest-neighbor consistency of the target space (collapse → all targets converge to a single point). The paper's own training infrastructure already generates the necessary data; the follow-up would mine it for diagnostic signals. The practical payoff: a "collapse detector" that could be added to any latent-space SSL training loop, warning practitioners to intervene (reduce learning rate, increase EMA decay, switch to frozen targets) before the run is wasted.

3. Ablation of the minimum viable pretraining data for multimodal EO models. The paper uses 285,288 samples drawn from 120 OpenStreetMap categories across the globe, with 9 modalities (3 observation, 6 map) and up to 12 monthly timesteps. This is a large, carefully curated dataset, but the paper provides no data scaling analysis. A systematic study varying (a) the number of pretraining locations, (b) the number of modalities, (c) the temporal coverage, and (d) the geographic diversity (e.g., continents held out) would answer practical questions for organizations wanting to train their own models: How many samples are needed for a given target performance? Does adding a new modality help more than adding more locations? How important is temporal coverage vs. spatial coverage? The paper's ablation of "Only S2 Data" (Table 6) shows catastrophic degradation (average rank last), but this removes all other modalities simultaneously and doesn't isolate individual contributions. A factorial ablation design — removing one modality at a time, varying sample count independently of modality count — would produce actionable data scaling curves analogous to the Chinchilla laws for language, but specific to multimodal EO. The paper's training infrastructure and evaluation suite are designed for exactly this kind of experiment; the 140,000-step ablation protocol provides a template for running many configurations efficiently.

4. Direct comparison of stabilization mechanisms on identical data and architecture. The paper compares full models (OlmoEarth vs. TerraMind vs. Galileo) with all their architectural and data differences, making it impossible to attribute performance differences to any single design choice. A controlled experiment training the same ViT architecture on the same pretraining data with three different stabilization mechanisms — (a) OlmoEarth's frozen random projection, (b) TerraMind's frozen VQ tokenizer, and (c) a learned target encoder with EMA (standard Latent MIM) — would isolate the effect of the stabilization strategy on both downstream performance and training stability. The key measurements: final downstream performance (replicating Table 4's three-task validation set), training stability metrics (number of runs that collapse out of N seeds), and computational overhead (VQ tokenizer training adds a separate pretraining stage; EMA adds per-step overhead; frozen projection adds no overhead). The paper already has all three approaches implemented or available through imported models — TerraMind is in the evaluation tables, standard Latent MIM is in the ablation tables — so the remaining work is to control for architecture and data differences. The result would definitively answer whether the frozen random projection is the best stabilization approach or merely a simpler one that happens to work well with OlmoEarth's specific architecture and data.

5. Dynamic masking strategy that adapts bandset selection based on difficulty or training progress. The paper's modality-aware masking assigns bandsets to encode/decode categories randomly, independent of the sample's content or the model's current capabilities. An adaptive masking strategy could: (a) estimate which modalities are most informative for reconstructing which other modalities, and bias masking toward harder cross-modal pairs; (b) start training with easier masking (more bandsets shared between encode and decode) and progressively increase difficulty; or (c) use the model's own reconstruction loss as a signal to identify which cross-modal relationships it has already learned and which it still struggles with, directing masking toward the latter. The paper's framework provides natural scaffolding: the four-category bandset assignment and the per-bandset loss computation make it straightforward to track which modality reconstructions have high vs. low loss, and to adjust masking probabilities accordingly. A curriculum learning approach — starting with standard random masking (all bandsets encode-and-decode) and gradually introducing decode-only bandsets as training progresses — would be a simple first experiment. The paper's observation that standard random masking is "too easy" (Section 2.3) and the ablation showing that modality masking provides the single largest performance jump (Table 4, +11.4 mIoU on PASTIS) suggest that the difficulty of the masking task matters enormously, and that optimizing this difficulty dynamically could yield further gains.

6. Extending the evaluation framework to include calibration, robustness, and out-of-distribution generalization. The paper's evaluation — 24 embedding tasks and 29 fine-tuning tasks — is comprehensive in coverage but focuses exclusively on in-distribution accuracy (or mIoU, μF1, etc.). Real-world EO deployments care about additional properties: calibration (do the model's confidence scores reflect its actual accuracy?), robustness to input perturbations (cloud cover, sensor degradation, missing timesteps, misregistration), and out-of-distribution generalization (does a model trained on global data perform well on a specific region with unique land cover types not seen in pretraining?). The partner tasks provide natural testbeds: the AWF Kenya dataset tests generalization to a specific African region that may be underrepresented in pretraining (land cover types like "lava forest" are unusual), and the Skylight vessel detection tasks test robustness to different sensor configurations and maritime environments. A follow-up evaluation adding these dimensions — specifically, reporting expected calibration error on classification tasks, measuring performance degradation under simulated cloud cover or missing timesteps, and reporting per-continent or per-biome breakdowns — would transform the paper's evaluation suite from a leaderboard into a diagnostic tool that reveals not just which model is best, but why and under what conditions. The paper's inclusion of 12 competing models and standardized training recipes makes this extension efficient: the embeddings and fine-tuned models already exist; the additional evaluation just needs to be run.

Practical Applications and Downstream Use Cases

1. On-demand fine-tuning for environmental monitoring by non-ML-expert organizations. The OlmoEarth Platform, combined with the pretrained models, enables organizations like Global Mangrove Watch and Global Ecosystem Atlas to go from labeled data to deployed models without GPU infrastructure or deep learning expertise. The concrete value demonstrated in the paper: Global Mangrove Watch improved from 95.3% F1 (random forest) to 98.1% F1 (fine-tuned OlmoEarth) on mangrove detection, and can now generate maps on a monthly cadence rather than yearly — a 12× increase in temporal resolution that directly enables faster detection of mangrove loss. The platform processed 15,000+ labeled data points for Global Ecosystem Atlas over three months, suggesting throughput of ~5,000 labels/month in an active project. For an organization considering adoption, the paper provides concrete numbers: fine-tuning OlmoEarth Base on all partner tasks consumed 685 kWh total (Table 5), equivalent to roughly $70–140 in cloud GPU costs at typical H100 pricing, and the platform abstracts away the infrastructure management. The Solar Farm Detection task shows what's possible: fine-tuned OlmoEarth achieves 86.7% mIoU on S1+S2 input for detecting solar farm polygons, compared to 77.5% for frozen AlphaEarth embeddings — a 9.2 mIoU improvement that, at continent scale, translates to substantially more accurate renewable energy deployment tracking. The key barrier addressed is not model capability (multiple models in Table 3 achieve near-ceiling performance on many tasks) but accessibility: the platform makes fine-tuning and inference operational for organizations that would otherwise be limited to precomputed embeddings or non-ML approaches.

2. Efficient on-device or edge deployment via the Nano and Tiny model variants. The paper's ViT Nano (1.4M parameters) and ViT Tiny (6.2M parameters) achieve performance that often matches or exceeds much larger models from competitors. In Table 2, OlmoEarth ViT Nano achieves 96.2% on m-brick-kiln (best overall), 83.7% on CropHarvest-Togo S2 (best among models that support the task), and 89.9% on m-eurosat (competitive with models 10–200× larger). In Table 3, OlmoEarth ViT Tiny achieves 97.7% on Forest Loss Driver classification, 85.5% on AWF land cover with S1+S2 input, and 85.2% mIoU on Solar Farm detection — all within a few points of the much larger ViT Base. Figure 1 positions these small models on the Pareto frontier: they achieve the highest performance per MAC of any evaluated model. The practical implication: organizations deploying models on edge devices (drones, field tablets, low-power servers in remote locations) or running inference at continental scale with cost constraints can use OlmoEarth Nano or Tiny without sacrificing substantial accuracy relative to models 20–200× larger. The paper's variable patch size training (1 to 8) means these models can process inputs at the resolution appropriate for the task and hardware, rather than being locked to a single computationally expensive resolution. A specific deployment scenario: a conservation NGO monitoring deforestation in the Amazon using Sentinel-2 imagery on a field laptop could run OlmoEarth Nano at patch size 8 (fast, lower resolution) for initial screening and switch to OlmoEarth Base at patch size 4 (slower, higher resolution) for flagged regions, all using the same pretrained model family.

3. Multimodal data fusion without hand-designed feature engineering. Traditional remote sensing pipelines often require domain experts to design features that combine information from different sensors — NDVI from Sentinel-2, backscatter ratios from Sentinel-1, thermal anomalies from Landsat-8 — before feeding them into a task-specific model. OlmoEarth learns these cross-modal relationships automatically through the modality-aware masking pretraining: the model must reconstruct Sentinel-2 optical bands from Sentinel-1 SAR data, or predict land cover maps from Landsat-8 thermal imagery, forcing it to learn the physical relationships between sensors. The downstream results demonstrate that this translates to real benefits: on the Marine Infrastructure detection task, using S2+S1 input improves OlmoEarth ViT Base from 86.3% F1 (S2 only) to 89.6% F1 (S2+S1 combined), a 3.3-point gain. On Solar Farm detection, S1+S2 achieves 86.7% mIoU vs. 85.4% for S2 only, a smaller but consistent gain. On AWF land cover classification, S2+S1 achieves 86.0% vs. 87.0% for S2 only — here S1 doesn't help, but the model gracefully handles the extra modality without degradation. The practical benefit: organizations with access to multiple data sources can feed all available modalities into the same model and let it learn which combinations are informative for which tasks, rather than making those decisions manually through feature engineering. The paper's design choice to never encode map modalities ensures the model doesn't depend on data that may be unavailable at inference time, making this fusion robust to missing sensors.

When to Prefer This Method

The paper itself does not explicitly frame a decision rule for when to use OlmoEarth versus specific named alternatives. However, the comprehensive evaluation against 12 competing models, combined with the architectural design choices and the documented limitations, supports the following practical guidance drawn directly from the paper's evidence:

• Use OlmoEarth when you need reliably stable training of a multimodal EO model and do not have the resources to debug representation collapse. The paper's primary contribution is eliminating the training instability that plagued prior latent-space approaches (documented collapse in Table 4). If your use case involves pretraining from scratch on custom EO data, Latent MIM Lite with frozen random projections provides a collapse-free training recipe that does not require the tuning or auxiliary training stages of alternatives like TerraMind's VQ tokenizer.

• Use OlmoEarth when you need a small, efficient model for deployment-constrained settings. The ViT Nano (1.4M parameters) and ViT Tiny (6.2M parameters) achieve performance competitive with models 10–200× larger on many tasks (Figure 1, Tables 2-3), making them the best option when inference cost, memory, or latency are binding constraints. No other evaluated model matches OlmoEarth's performance at these small scales.

• Use OlmoEarth Nano when your task involves classifying crops. The paper notes that Presto (Tseng et al., 2023) was designed specifically for crop type classification with single-pixel time series, and Table 2 shows Presto achieving competitive performance on CropHarvest and BreizhCrops tasks. However, OlmoEarth ViT Nano outperforms Presto on CropHarvest-PRC (S2: 79.5 vs. 74.1; S1+S2: 74.3 vs. 76.6) and on CropHarvest-Togo S2 (83.7 vs. 81.4), while also supporting a broader range of tasks and sensor configurations. Presto is specialized; OlmoEarth is general but matches or exceeds the specialist on its home turf.

• Prefer Panopticon or TerraMind when the specific task is among those where they lead in Tables 2-3 and you are working with frozen embeddings rather than fine-tuning. Panopticon achieves the best embedding performance on m-bigearthnet (64.9 μF1 vs. 62.4 for OlmoEarth Base) and m-brick-kiln fine-tuning (99.0% vs. 98.6%). TerraMind ViT Large leads on m-bigearthnet fine-tuning (74.0 μF1 vs. 72.4 for OlmoEarth Large) and m-cashewplant fine-tuning (81.3 mIoU vs. 80.6). These margins are small (0.4–1.6 points) and fall within reasonable noise without confidence intervals, but for applications where these specific tasks are the primary concern, the paper's own numbers suggest the alternatives may be marginally better.

• Prefer full fine-tuning of OlmoEarth over frozen embeddings (including AlphaEarth) when task-specific performance matters and you have even a modest labeled dataset. Table 7 quantifies the gap: on Nandi classification, kNN with OlmoEarth embeddings achieves 66.2%, frozen + decoder achieves 62.9%, and full fine-tuning achieves 82.2% — a 15.6–19.3 point improvement from fine-tuning. On Solar Farm detection, frozen embeddings achieve 84.8% mIoU vs. 86.7% with fine-tuning. The gains from fine-tuning are task-dependent but consistently positive, and the paper's fine-tuning recipe (Section 3.4) is straightforward. If your organization cannot fine-tune at all (e.g., you only have access to precomputed embeddings), AlphaEarth embeddings are competitive with frozen OlmoEarth embeddings (Table 7) and may be preferable if the precomputed global coverage eliminates your need for inference infrastructure — but you sacrifice the ability to improve through fine-tuning.