Soccerbot — Unitree G1 Goalkeeper

00 Overview

The task, precisely

Phase 1 — learned pickup. The robot sees a ball on a table of fixed height and picks it up. This is the only part learned from demonstrations, driving a 14-DOF arms-only action space (7 DOF per arm — shoulder pitch/roll/yaw, elbow, wrist roll/pitch/yaw). No waist, legs, or fingers are in the learned policy; the hand is held in a fixed pose and grasping is achieved through arm positioning alone.

Phase 2 — scripted goalkeeper. A fixed number of timesteps after pickup finishes, a hardcoded FSM takes over: turn 180°, watch for a person with the depth-sense camera, shuffle side to side while someone is detected, and once no person has been seen for 6 continuous seconds, throw the ball straight ahead along a fixed trajectory.

The two phases are deliberately separate systems — the scripted goalkeeper has no VLA/model dependency at all, and the pickup policy never touches the throw logic.

Unitree G1 humanoid Single RGB-D camera (pickup vision + human depth) BrainCo Revo 2 hands (no learned grasping) uv workspace, Modal GPU training

data_collection.mp4 teleop capture session

01 Data Collection

Teleop episodes, one pickup at a time

Every training episode starts as a human-teleoperated demonstration of the G1 picking a ball off the table, recorded in LeRobot dataset format alongside synchronized RGB frames and 14-D arm state/action pairs. Getting here took a lot of trial and error on calibration and teleop mimicry (see Challenges) — the payoff was 150 good samples, of which 122 episodes went into the ACT fine-tune below.

150 good samples collected 122 episodes used for fine-tuning LeRobot v3.0 format 14-D state/action, arms-only

rerun vid.mov Rerun viewer walkthrough

02 Inspecting recorded data

Live runs played in Rerun

Recorded teleop and inference runs are logged into Rerun — camera frames, 14-D joint state/action curves, and stage telemetry (turn / avoid / throw) all synced on one timeline. A custom Blueprint and a small Query API script make it fast to scrub through an episode and see exactly what the policy saw and did at each timestep.

Custom Rerun Blueprint Turn / avoid / throw telemetry Query API tooling

5 augmented episodes, back-to-back HF dataset ↗

03 Environment augmentation

Making 122 episodes cover more than 122 rooms

Real teleop demonstrations are all recorded in one physical setup, which leaves a policy brittle to lighting, background, and scene changes it never saw. We use KITE-ML to augment the recorded teleop episodes against environment variation, generating new synthetic-but-grounded versions of each demonstration while preserving the original action labels.

The augmented set is published as g1_final_cleaned-augmented on Hugging Face — the clip on the left plays five augmented episodes straight through.

KITE-ML augmentation Action labels preserved Published to Hugging Face Hub

human_detection.mp4 depth-based presence check

04 Human-aware goalkeeper

See a person, shuffle to stay clear

During the scripted goalkeeper phase, the same RGB-D camera used for pickup switches roles and watches for people using its depth stream. Whenever a person is detected, the robot shuffles side to side — the same goalkeeper motion, repurposed to keep the human out of its throwing lane. Once no one has been detected for 6 continuous seconds, it commits to the fixed throw trajectory.

Depth-based detection Side-to-side shuffle avoidance 6s clear-timeout before throw

05 Models trained

Three policy families, one winner

Pi 0.5, MolmoAct2, and ACT were all fine-tuned on the same 14-D arms-only pickup task and compared head-to-head. ACT-based policies came out on top; the plain ACT checkpoint became the deployed pickup policy, and the augmented dataset above exists specifically to make it more robust before further training.

Pi 0.5

Vision-language-action policy evaluated as a candidate for the learned pickup phase.

CandidateStatus

MolmoAct2

VLM-based action policy evaluated alongside Pi 0.5 and ACT on the same pickup task.

CandidateStatus

Deployed

ACT

Action-chunking transformer, fine-tuned on real G1 teleop demonstrations — selected as the pickup policy.

15kFine-tune steps

122Teleop episodes

↳ We ended up choosing the 15k-step fine-tuned ACT policy, trained on 122 teleop episodes, over Pi 0.5 and MolmoAct2. That same 122-episode set is then run through KITE-ML augmentation (see above) to account for environment changes — new lighting, backgrounds, and scene variation — before further training passes.

06 Live demo

Our actual hackathon demo

live_demo.mp4 original post ↗

07 Challenges & lessons learned

Not everything worked

A humanoid on a soccer pitch surfaces problems that never show up in simulation. Some we solved with a tradeoff, some are still open.

Dexterous hand fine-control

Workaround

An electrical connection issue with the BrainCo Revo 2 hands as provided made fine finger control impossible.

Resolution: couldn't fix the hardware fault, so we designed around it — the hand is held in a fixed pose and pickup is achieved through arm positioning alone. No finger use, no learned grasping.

Teleop calibration & mimicry

Resolved

Setting up teleoperation took a long time — calibrating the robot and the teleop rig so that human operator motion was accurately mirrored by the G1 was a persistent source of bad episodes.

Resolution: after a lot of trial and error, we landed on a working calibration and collected 150 good samples, 122 of which went into the deployed ACT fine-tune.

Picking a pickup policy

Resolved

We fine-tuned across several policy families — Pi 0.5, MolmoAct2, ACT — and had to pick whichever actually performed best on the real robot rather than on paper.

Resolution: ACT, and an ACT variant initialized from Pi 0.5, consistently outperformed the alternatives, so the plain 15k-step ACT checkpoint shipped as the deployed policy.

Manipulation / locomotion / balance clash

Unresolved

The manipulation policy, the locomotion controller, and the deployed goalkeeper behavior fought each other, and the balancer suffered most during programmed action steps. Specifically: when the manipulation policy engages arms_sdk, it somehow disables the waist hinge, and the robot starts bending over.

Resolution: could not fix. We tried damping the waist hinge; the robot still bent over. Root cause remains open.

Uncommanded spasms

Resolved — safety

The G1 would periodically spasm, likely from a combination of policy clash and overheating — a severe safety concern during testing.

Resolution: fixed with a hardware killswitch, close human supervision on every run, and continuous joint diagnostics that track internal temperatures in real time to catch overheating before it triggers a spasm.

08 Architecture

How the pieces fit together

Pulled from the project's GitHub README — the pickup policy and the goalkeeper FSM are intentionally decoupled systems.

┌─────────────────────────┐ 100 real demo │ training/ (Modal) │ episodes ───────▶ │ fine-tune policy │──▶ checkpoint (HF Hub / Modal volume) │ on G1 arms-only 14-D │ └─────────────────────────┘ │ ▼ ┌─────────────────────────┐ ┌──────────────────────────┐ robot sensors ───▶ │ local-vla-inference/ │───▶│ scripted-behavior/ │ (RGB-D camera, │ loads checkpoint, runs │ │ hardcoded FSM: turn, │ arm state) │ policy on real G1 arms │ │ detect, shuffle, throw │ │ for the pickup phase │ │ (no model, no learning) │ └─────────────────────────┘ └──────────────────────────┘

Component	Status	Purpose
`training/`	● implemented	Modal launcher that submits LeRobot fine-tuning jobs (ACT, Pi 0.5, and others) to GPU workers.
`thirdparty/lerobot/`	● vendored	Hugging Face LeRobot — datasets, policies, training loop.
`local-vla-inference/`	● implemented	Real-time control loop: loads the fine-tuned checkpoint, drives the G1's arms via `unitree_sdk2py` for pickup.
`scripted-behavior/`	● implemented	Hardcoded post-pickup FSM — turn 180°, depth-based human detection, shuffle, 6s-timeout throw.

the reel