An open, closed-loop serving benchmark for Vision-Language-Action policies and embodied world models at the CoRL 2026 EEAI Workshop. Submissions chart the single-chip latency-throughput Pareto frontier under real-time control deadlines.
GitHub Repository · Methodology Specification (PDF) · Submit a Run
Season 1 opened October 30, 2026 (0 community submissions recorded so far; the reference harness and eight-gate verifier were validated end-to-end on physical NVIDIA L40S and Cloud TPU v6e-1 without seeding organizer scores). Merged pull requests populate this chart automatically; dashed lines illustrate how concurrency sweeps (C = 1..16) trace the latency-throughput frontier against each track's real-time control deadline.
| Track | Target Hardware | Leading Entry | Real-Time SLA |
|---|---|---|---|
| Track 1A: π0.5-LIBERO | NVIDIA L40S (48 GB) · TPU v6e-1 (32 GB) | Open (no submissions yet; submit PR #1) | p99 ≤ 250 ms (≥ 4.0 Chunks/s) |
| Track 1B: GR00T N1.7-3B | NVIDIA L40S (48 GB) · TPU v6e-1 (32 GB) | Open (no submissions yet; submit PR #1) | p99 ≤ 400 ms (≥ 2.5 Chunks/s) |
| Track 2: Cosmos-Predict2.5-2B | NVIDIA L40S (48 GB) · TPU v6e-1 (32 GB) | Open (no submissions yet; submit PR #1) | p99 ≤ 1000 ms (≥ 1.0 Pred/s) |
Because closed-loop deployments trade tail latency against batch throughput, two Pareto curves may cross. Season 1 evaluates each track on the two standardized cloud platforms using three scalar award categories anchored to the track's real-time control deadline.
| Award Category | Scalar Ranking Metric | Admissibility Requirement |
|---|---|---|
Interactive Reactivity (C = 1) |
Highest single-stream control rate 1000 / p99_ms at concurrency C = 1 (lowest C = 1 p99 latency, tie-broken by lowest p50_ms). |
Unimodal latency tail p99 / p50 ≤ 1.5 at C = 1. |
| Real-Time Fleet Throughput | Highest aggregate Actions/s (Track 1) or Predictions/s (Track 2) achieved over canonical points C ∈ {1, 2, 4, 8, 16}. |
Selected point must satisfy the track's gapless real-time p99 SLA. |
| Green Embodied Efficiency | Highest hardware-measured Actions/J (Track 1) or Predictions/J (Track 2) across SLA-compliant points in {1, 2, 4, 8, 16}. |
10 Hz NVML board TBP power on NVIDIA L40S at constant cap (TPU reported as N/A). |
Exact ties are broken by lower-step Real-Time Hypervolume over SLA-compliant discrete points relative to reference origin (0.5 · XSLA, 0).
Winner recognition. Verified Season 1 category winners receive monetary prizes, an official CoRL 2026 EEAI Workshop Award Certificate, an invitation to apply for Google Cloud credits, permanent verified leaderboard attribution, and an invitation to co-author the post-season EmbodiedX Season 1 Systems Report. Early submissions merged before the November 10 freeze are recognized live on stage at CoRL on November 12 at Hilton Austin, Room 408 (2:00–6:00 PM CST).
EmbodiedX separates the two action-policy workloads (Track 1A and Track 1B) from world-model video prediction (Track 2) because their per-action compute budgets and latency deadlines differ substantially. A 20 Hz servo step corresponds to 50 ms (20 Chunks/s single-tick reaction).
| Track | Checkpoint and Input Suite | Locked Horizon | Gapless SLA and Closed Gate |
|---|---|---|---|
| Track 1A: π0.5-LIBERO | lerobot/pi05_libero_finetuned_v044 (10 ODE steps, dual 224×224 RGB on LIBERO traces) |
H = 10, Keval = 5 |
p99 ≤ 250 ms (≥ 4.0 Chunks/s); CosSim ≥ 0.995 + CVaR-1% |
| Track 1B: GR00T N1.7-3B | nvidia/GR00T-N1.7-LIBERO (libero_10; 16 DiT steps, dual 256×256 RGB on LIBERO traces) |
H = 16, Keval = 8 |
p99 ≤ 400 ms (≥ 2.5 Chunks/s); CosSim ≥ 0.995 + CVaR-1% |
| Track 2: Cosmos-Predict2.5-2B | nvidia/Cosmos-Predict2.5-2B (robot/action-cond; 10 steps, CFG = 1.0, 256×256 rollout traces) |
Keval = 16 frames |
p99 ≤ 1000 ms (≥ 1.0 Pred/s); SSIM ≥ 0.88, PSNR ≥ 26.0 dB, CVaR-1% SSIM ≥ 0.80, CVaR-1% PSNR ≥ 22.0 dB |
Every submission runs on a single physical chip (num_devices == 1, TP = PP = EP = 1) at stock vendor clock and power limits. Season 1 standardizes on one NVIDIA GPU and one Google Cloud TPU so entries compete strictly on kernel, compiler, and serving systems engineering rather than hardware budget, and so organizers can re-run every finalist on identical cloud silicon.
10 Hz NVML board energy telemetry. Available on demand through Shadeform (~$0.88–$1.15/hr), AWS EC2 g6e.xlarge, or the NVIDIA Academic Grant Program.TorchTPU stacks. Available on demand through Google Cloud (~$1.375/hr) or free via a renewable 30-day Google TPU Research Cloud (TRC) academic allocation.Onboard edge modules are reserved for future seasons so Season 1 remains strictly reproducible on standardized cloud hardware.
Keval so action throughput cannot be inflated by lengthening open-loop chunks. Closed keeps checkpoint architecture, weights, and solver step count fixed under BF16/FP16/FP8 numerics verified by inline Golden-IO parity (CosSim plus CVaR-1% tail floor for Track 1; SSIM ≥ 0.88, PSNR ≥ 26.0 dB, CVaR-1% SSIM ≥ 0.80, and CVaR-1% PSNR ≥ 22.0 dB for Track 2). Open allows INT4 quantization, pruning, step distillation, or token merging only when accompanied by a signed Stage-2 quality receipt: latency-coupled LIBERO-RT success within -5.0 pp of reference for Track 1, or SSIM ≥ 0.88, PSNR ≥ 26.0 dB, LPIPS ≤ 0.12, FVD ≤ 120, InvDyn ≥ 0.99, and ranking Kendall τ ≥ 0.80 for Track 2.C ∈ {1, 2, 4, 8, 16} starting at C = 1 (allowing only monotonic tail truncation on device OOM, with zero off-grid concurrencies and zero curve interpolation). Each point requires N = 3 trials within ≤ 8% inter-run p99 spread and p99 / p50 ≤ 1.5 at C = 1, measured after real-model warmup (--warmup 30 to 50) at stock vendor TDP/clocks with zero host-CPU neural offload and per-frame cryptographic pixel micro-jitter against trace caching.Actions/J and Predictions/J rankings strictly require power_provenance == "MEASURED" from active hardware sensors (10 Hz NVML total board power on NVIDIA L40S) held at a constant power limit across all concurrencies in the bundle. Assumed-wattage flags (--assume-power-watts) are rejected; Cloud TPU v6e-1 entries report power as N/A where guest VMs do not expose board wattage.bundle_summary.json and submission_meta.json carrying recipe_commit_sha256, a SHA-256 digest over a canonical reveal_manifest.json pinning the immutable OCI container image (@sha256), checkpoint shard digests, lockfile hash, and exact launch command. Submitted curves render live on the public leaderboard while custom kernels and serving recipes remain private until the contest freeze.± 8% agreement against Phase 1 serves strictly as an anti-fraud disqualification gate. Confirmed Season 1 winners publish reproducible open recipes (Verified Open Recipe); continuous post-CoRL entries may carry either open or Proprietary / Audit-Only attribution.git clone https://github.com/efficient-embodied-ai/embodied-efficiency-bench.git
cd embodied-efficiency-bench && pip install -e ".[vla,gpu]"
eeb data prepare-traces --track vla --split mini --out ./data/traces
eeb data generate-golden --model pi05 --traces ./data/traces --num-samples 100 --out ./data/pi05_golden.npz
eeb serve --model pi05 --checkpoint lerobot/pi05_libero_finetuned_v044 --port 8080 &
eeb measure --track track1a --division closed --hardware nvidia_l40s_48gb \
--server http://127.0.0.1:8080 --traces ./data/traces --golden ./data/pi05_golden.npz \
--sweep 1,2,4,8,16 --warmup 30 --calls 320 --runs 3 --strict \
--out ./runs/my_pi05_l40s
eeb prepare-commit-bundle --run-dir ./runs/my_pi05_l40s \
--output ./submissions/track1_vla/my_lab_pi05_l40s
eeb verify-bundle ./submissions/track1_vla/my_lab_pi05_l40s --strictOpen a Phase-1 pull request placing bundle_summary.json and submission_meta.json (containing your recipe_commit_sha256 digest) under submissions/<track_dir>/<submission_id>/ in efficient-embodied-ai/embodied-efficiency-bench. Full packaging details, Cloud TPU v6e-1 (tpu_v6e_1_32gb) single-chip environment flags, and coding-agent harnesses are documented in submissions/README.md and the EmbodiedX Methodology Specification (PDF).