GE-Act 2.0

Learning visual prediction and action from complementary data从互补数据中学习视觉预测与动作生成

Manipulation data provides complementary kinds of supervision. Video shows how a scene changes, often without recording the actions that caused those changes. Robot trajectories contain actions but may lack language instructions or depict unsuccessful attempts. GE-Act 2.0 uses both through separate pretraining: our single-step visual planner learns to predict future states from video, while the inverse dynamics model learns to recover actions from observed transitions. Both models are pretrained from scratch on manipulation data.机器人操作数据提供了互补的监督信息。视频记录场景如何变化，却往往没有记录引发变化的动作；机器人轨迹包含动作，但可能缺少语言指令，也可能来自失败尝试。GE-Act 2.0 通过独立预训练利用这两类数据：单步视觉规划器从视频中学习预测未来状态，逆动力学模型则从观测到的状态转移中学习恢复动作。两个模型均在操作数据上从零开始预训练。

The two components are then connected through predicted visual futures. Knowledge-aligned selective optimization (KASO) selects predictions compatible with the recorded actions, allowing joint training to build on what each component has learned. We scale this co-training stage and measure the capabilities present before downstream adaptation.随后，我们通过预测的未来视觉状态连接这两个模块。知识对齐选择性优化（KASO）筛选与记录动作相容的预测，使联合训练能够延续各模块在独立预训练中获得的能力。我们扩展这一协同训练阶段的数据规模，并评估模型在下游适配之前已具备的能力。

Scaling Zero-Shot Manipulation Skills零样本操作技能随数据规模提升

We evaluate pretrained checkpoints directly on G1-OP and G2-90D, which contribute more than 50% and less than 2% of the co-training data, respectively. Evaluation follows a zero-shot, out-of-distribution (OOD) protocol, without task- or embodiment-specific fine-tuning, using new object instances, scenes, backgrounds and lighting conditions not seen during pretraining or co-training.我们将预训练检查点直接部署到 G1-OP 和 G2-90D 上进行评估，二者分别占协同训练数据的 50% 以上和不足 2%。评估遵循零样本、分布外协议，不做任务或本体专属微调，使用预训练和协同训练中均未出现的新物体实例、场景、背景和光照条件。

We scale manipulation data through four nested pools of 300, 1,200, 5,000 and 30,000 hours, using the same pretrained components and training recipe. Each checkpoint is evaluated on 100 tasks across 20 skill groups on both embodiments, allowing us to examine how performance scales with shared training data on both a well-represented and a sparsely represented robot. The recordings show the 30,000-hour model; the task wall traces performance across all four scales.我们使用四个逐级包含的数据集，将操作数据规模扩展至 300、1,200、5,000 和 30,000 小时，各组实验采用相同的预训练模块和训练方案。每个检查点均在两个本体上评估20 类技能中的 100 项任务，从而考察在数据充足与数据稀缺的机器人上，性能如何随共享训练数据规模变化。下方视频展示 30,000 小时模型的执行表现，任务墙展示四个数据规模下的成功率。

Following Language Instructions遵循语言指令

Under the same zero-shot OOD protocol, we evaluate GE-Act 2.0’s instruction grounding across object identity, colour, size, shape, position and order. The demonstrations highlight precise object selection in cluttered scenes, adherence to instructions that conflict with learned action biases, and generalization to new tasks through instruction composition.在同一零样本、分布外评估协议下，我们从物体类别、颜色、尺寸、形状、位置和顺序六个维度评估 GE-Act 2.0 的指令指代解析能力。下方演示重点展示在杂乱场景中精确选取物体、遵循与已学动作偏好相冲突的指令，以及通过指令组合泛化到新任务。

{{ m.chipEmb }}{{ m.chipSpeed }}{{ m.chipOod }} {{ v.pendingLabel }}

{{ v.caption }}

{{ m.tag }}

{{ m.chipEmb }}{{ m.chipSpeed }}{{ m.chipOod }} {{ v.pendingLabel }}

{{ m.cueText }}

{{ m.title }} {{ m.text }}

GE-Act 2.0 ArchitectureGE-Act 2.0 架构

GE-Act 2.0 turns an instruction into action through three components in sequence. The control-oriented autoencoder (CoAE) maps camera observations into a compact latent space. The single-step visual planner (SVP) predicts future states in that space, conditioned on the instruction and current observation. An inverse dynamics model (IDM) reads the predicted future together with the current latents and proprioception to produce an action chunk.GE-Act 2.0 通过三个依次连接的模块将指令转化为动作。面向控制的自编码器（CoAE）将相机观测映射到紧凑的潜在空间；单步视觉规划器（SVP）以当前观测和指令为条件，在该空间中预测未来状态；逆动力学模型（IDM）结合预测的未来、当前视觉潜变量与本体感知信息，生成动作片段。

The future sequence covers two timescales: dense visual states over the immediate action horizon and sparse states extending to the end of the task. The IDM uses the full sequence for context, but the robot executes only the dense action chunk. Sparse far-horizon action predictions provide auxiliary supervision during training. This keeps the immediate control output connected to the visual outcome of the broader task.未来状态序列覆盖两种时间尺度：近期动作执行范围内的密集视觉状态，以及延伸至任务结束的稀疏视觉状态。IDM 使用完整序列提供的上下文，但机器人仅执行密集动作片段。远期稀疏动作预测在训练中提供辅助监督，将即时控制与整个任务的视觉结果联系起来。

A Control-Oriented Latent Space面向控制的潜在空间

A useful visual representation must preserve the information required to predict both future states and actions. CoAE compresses each 256 × 384 camera frame into just 24 latent tokens, using 64× spatial downsampling. This reduces the sequence length processed by the visual planner while retaining a pixel decoder for inspecting predicted futures.有效的视觉表征需要保留预测未来状态和动作所需的信息。CoAE 使用 64 倍空间下采样，将每张 256 × 384 的相机图像压缩为仅 24 个潜在 token，减少视觉规划器处理的序列长度。同时保留像素解码器，使预测的未来状态能够还原为图像并被检查。

Reconstruction alone does not explicitly organize the representation around manipulation. CoAE therefore adds three alignment heads alongside its reconstruction objective. They match frozen features from SigLIP 2 for semantics, V-JEPA 2.1 for motion and DINOv3 for dense visual structure. These complementary targets encourage the compressed latents to retain information that visual planning and action recovery need.仅依靠重建目标，无法显式地使表征围绕操作任务组织信息。因此，CoAE 在重建目标之外加入三个对齐头，分别匹配冻结的 SigLIP 2 语义特征、V-JEPA 2.1 运动特征和 DINOv3 密集视觉结构特征。这些互补目标促使压缩后的潜变量保留视觉规划与动作恢复所需的信息。

Single-Step Generation Enables Separate Pretraining单步生成支持独立预训练

The connection between visual prediction and inverse dynamics must remain trainable. A conventional multi-step generator produces a completed future after a sequence of denoising passes. Backpropagating an action loss through that future requires differentiating through the generation chain. Coupling action prediction to intermediate generator features can avoid this chain, but ties action learning to the generator’s internal representation.视觉预测与逆动力学之间的连接需要支持梯度传播。传统多步生成器经过一系列去噪步骤后，才能得到完整的未来状态；若要通过该预测反向传播动作损失，就需要对整条生成链求导。让动作预测依赖生成器的中间特征可以绕过这条链，但也使动作学习与生成器的内部表征绑定。

SVP instead generates the complete future latent sequence in one differentiable MeanFlow pass. The IDM can first learn from actual observed transitions as a standalone model, then connect to completed predictions through the same input interface. During co-training, the action loss passes through the IDM into the planner, allowing both components to adapt to the connection.SVP 在一次可微的 MeanFlow 前向计算中生成完整的未来潜变量序列。IDM 因此可以先作为独立模型，从真实观测到的状态转移中学习，再通过相同的输入接口接入完整的预测结果。在协同训练中，动作损失经过 IDM 传回视觉规划器，使两个模块共同适应这一连接。

Aligning Visual Futures with Action Supervision将未来视觉预测与动作监督对齐

A correct visual prediction is not automatically compatible with a recorded action. Given the same scene and instruction, a robot may complete a task in several valid ways. A generated future may depict one completion while the demonstration records another. Training the IDM on that future with the recorded action asks it to learn from a mismatched pair.正确的视觉预测并不一定与记录的动作相容。面对相同场景和指令，机器人可能以多种有效方式完成任务；生成的未来状态可能对应其中一种，而示范记录的是另一种。将这样的未来预测与记录动作配对训练，会让 IDM 接收到不匹配的监督。

We term this supervision mismatch the validity gap. Improving visual plausibility alone does not ensure that an independently generated future describes the behavior in the recording. Knowledge-aligned selective optimization (KASO) addresses the connection directly: it generates several candidate futures and selects those whose implied actions are compatible with the recorded behavior.我们将这种监督不匹配称为“有效性缺口”。仅提高视觉预测的真实性，并不能保证独立生成的未来状态与记录中的行为一致。知识对齐选择性优化（KASO）直接处理这一问题：生成多个候选未来，并选择其中所隐含动作与记录行为相容的候选。

GE-Act 2.0 shows that scaling manipulation data produces broad improvements in zero-shot, out-of-distribution performance across manipulation skills, with gains extending to an embodiment sparsely represented in the shared training data. The model also demonstrates fine-grained instruction grounding and follows explicit commands even when they conflict with an ongoing action or a familiar scene association. Together, these findings establish GE-Act 2.0 as a scalable approach to learning transferable manipulation capabilities through world-action pretraining.GE-Act 2.0 表明，扩大操作数据规模能够在各类操作技能上带来广泛的零样本、分布外性能提升，且这种提升同样延伸到在共享训练数据中占比很少的本体。模型还表现出细粒度的指令理解能力：即使指令与正在进行的动作或场景中惯常的关联相冲突，也能按照明确指令执行。这些结果共同表明，GE-Act 2.0 是一条通过世界—动作预训练学习可迁移操作能力的可扩展路径。