A globally coordinated benchmark for real-world embodied bimanual manipulation
Bimanual manipulation benchmarks today are fragmented. Some evaluate algorithms only in simulation. Others test on a single robot in a single lab. Neither answers the question that matters most: do these methods generalize across platforms, environments, and operators? The EBiM Competition introduces a globally coordinated benchmark that combines an open simulation phase with cross-continent real-robot validation on identical Mobile FR3 Duo platforms. By validating top simulation methods on real hardware in Hamburg, Munich, Pittsburgh, and Shanghai, we build the first reproducible, cross-laboratory benchmark for embodied bimanual manipulation.
Phases I and II make up the Competition. Phase III is the workshop where results are presented.
Release an open simulation benchmark focusing on unstructured manipulation scenarios: cable routing, deformable material handling (thermal pad placement), and assisted living and feeding tasks. Global teams participate through an open call on a shared, reproducible leaderboard.
Teams advancing from simulation validate on real Mobile FR3 Duo robots at partner labs in Hamburg, Munich, Pittsburgh, and Shanghai. After simulation results are announced (Aug 8), qualifying teams get a hands-on bench-testing window on the real platform (Aug 10–19). Repository submissions are open now, since the developer-preview release; organizer-run simulation evaluation may begin at any time. From Aug 20–31, all real-robot bench testing and evaluation are run by the organizers — ensuring cross-site comparability.
Competition results, invited talks, and a panel discussion are presented at a community workshop (venue and date TBD). The workshop releases the benchmark protocol, open dataset, and community roadmap for future iterations.
Phase III: Workshop & Roadmap (venue and date TBD) — see the Workshop page for the full program.
A standardized bimanual mobile manipulation platform by Franka Robotics, deployed across all international testbed sites.
Mobile FR3 Duo extends the FR3 Duo reference platform into mobile environments by integrating it with Tactile Mobile Robot (TMR). It combines dual-arm manipulation, rich perception, and omnidirectional mobility into a single embodied AI system — enabling data collection and policy execution beyond fixed workcells. All competition testbeds in Hamburg, Munich, Pittsburgh, and Shanghai use identical Mobile FR3 Duo platforms to ensure cross-site reproducibility.
Learn more at franka.de/product-prototypes · Franka Robotics GmbH
A multi-axis framework for fair and meaningful evaluation of bimanual manipulation systems.
Current bimanual manipulation benchmarks are fragmented across hardware platforms, task definitions, and evaluation criteria. Our workshop targets the design of a community-wide standard covering task diversity, generalization, robustness, and deployment readiness — evaluated on real robots under standardized conditions.
Three core tasks: cable routing, deformable material handling (thermal pad placement), and assisted living & feeding.
Three layers — Outcome (task points + completion time, ranked); Process (reported, not ranked: SPARC smoothness, torque/energy, re-grasps, handovers, and bimanual coordination via BiCoord SMP/MAD/STI); and Safety (hard-fail gates on force/damage thresholds).
Shared teleoperation, annotation schema, and hardware calibration guidelines for cross-site compatibility.
Platform-neutral APIs and sim-to-real transfer tracks to include diverse robot embodiments.
An open, continuously updated leaderboard of verified real-robot submissions. Each task is ranked independently — no overall aggregate — with per-task tie-breaks (completion time; for Assisted Living, highest completed stage → total → time). Bimanual tasks are tagged by coupling type (PerAct2); simulation results follow the RoboTwin convention.
Curated multi-site demonstration dataset with diverse operators, environments, and objects.
Three core bimanual manipulation tasks — contact-rich assembly, deformable material handling, and assisted living. Tasks run in Isaac Sim, MuJoCo, and Genesis. All tasks are evaluated at Full Autonomy.
Industrial path recognition, cable grasping, routing, and fixation.
Deformable-object manipulation (flexible cable), contact-rich interaction & path recognition, and bimanual re-grasping with dual-arm coordination.
Primary: completion rate — fixtures correctly routed in sequence and validated at Y-checkpoints; only an unbroken correct run from the start counts (a wrong routing voids the rest). Limit: 30 min · Full Autonomy only. Secondary: completion time, re-grasps, handovers. Process: SPARC smoothness, joint torque, bimanual coordination (SMP/STI).
Industrial electronics manufacturing — a Thermal Interface Material (TIM) assembly benchmark.
Deformable-material modeling (soft-body physics), non-linear adhesion and surface friction, high-precision placement of compliant objects, and contact-rich force control with tactile sensing.
Primary: valid-placement IoU — Pick Success × Orientation Success × IoU (0–1); wrong orientation scores 0. Tie-break: completion time. Process/Safety: force-threshold violations.
A four-stage kitchen-to-dining service cycle: Table Setup → Feeding → Bean Recovery → Cleanup.
Two rooms — a Kitchen and a Dining area. Objects start stacked on a plate in the kitchen (plate, cup, bowl with coffee beans, spoon); three of six seats are randomly assigned; a scaled recycling bin and a marked sink region sit in the kitchen. Simulation and real-world setups are identical.
Primary: four stages — Table Setup, Feeding, Bean Recovery, Cleanup — 4 points each (max 16); Bean Recovery scored by recovered-bean ratio. Ranking: highest completed stage → total score → completion time. Safety: peak head/face force (ISO/TS 15066), watchdog interventions.
Reference simulation environments and full scoring scripts will be released as part of the benchmark framework.
Identical Mobile FR3 Duo platforms deployed at four partner labs across Asia, Europe, and North America.
European testbed hosted at a University of Hamburg robotics lab (venue TBA); coordinates data protocols and operator calibration.
Industry testbed providing platform integration and deployment support.
North America testbed at Carnegie Mellon University for cross-continent real-robot validation.
Asia testbed hosted at the Franka Robotics Shanghai branch office for cross-continent real-robot validation.
Join the Shanghai testbed WeChat group →From open simulation to real-robot finals — the end-to-end pipeline and the five pillars that run the benchmark.
How a submission flows from upload through to physical validation.
We invite teams from academia and industry to compete in Phase I (simulation) and qualify for Phase II (real-robot validation).
Because the developer preview shipped later than planned, the submission deadline will be extended; the revised schedule will be announced here soon. Questions about timing? Contact us →
Clone the repo and start building.
The EBiM Benchmark is available now as a developer preview — framework, simulation environments, robot assets, and task environments.
Check STATUS before you build: STATUS.md → lists which task and simulator combinations are usable today. The preview is under active development and may contain bugs, incomplete features, or API changes.
Still coming, released incrementally: the Teleoperation Toolkit (Keyboard / GELLO / Foot Pedal / VR), baseline models, tutorials and documentation, real-world datasets, and additional assets. The benchmark is open source — bug reports and contributions are welcome.
How to submit your team's work.
One issue per team, per task track. Update your submission any time before the deadline by opening a new issue that supersedes the old one.
A public GitHub repository containing
Dockerfile
Encapsulating your work.
README
Explaining how to run it.
Source code is not required. You may optionally link supplementary materials (e.g. Hugging Face model weights or datasets) — if you do, your README must include a clear integration guide.
Submissions are verified against your registration (team name + point-of-contact email); unregistered submissions are not evaluated. Evaluation code in the benchmark repository is a development aid — official scoring follows the rules on this page and is run by the organizers. See the schedule note above for the submission deadline.
Up to $5,250 in prizes per task — cash + purchase voucher, trophy & gift — awarded independently across all three benchmark tasks, backed by AMD and our community partners.
Every prize below is awarded per task, independently for each of the three benchmark tasks — a team can place in every task.
Per task · judged on the physical Mobile FR3 Duo
Per task · awarded in the open simulation phase
AMD development hardware distributed across three regions — U.S., Germany, and Asia — with each region receiving:
Eligibility: Open to all teams participating in the simulation and real-world phases. See Call for Participation for details.
We gratefully acknowledge the partners who make the EBiM Competition possible.
Interested in partnering? Contact us to discuss hardware support, travel grants, or community partnership opportunities.
Phase III of the EBiM initiative is the half-day workshop (venue and date TBD), featuring invited talks, a panel discussion, and competition awards.
