EXPERIMENT 001 / PIXEL-TO-CONTROL

See pixels.
Take action.

A real-time web game built as a clean benchmark: the agent receives only canvas frames and responds with the same movement, aim, attack and dash controls available to a human.

LIVE SESSIONHuman vs baseline

WASD / ARROW KEYS + MOUSE · TAP TO SHOOT · HOLD 1 SECOND FOR LASER · SPACE TO DASH

MOVE

AIM / FIRE

PLAYER ONE / HUMANHealth 100%

0:0

PLAYER TWO / BASELINEHealth 100%

01Tap: projectile

Click or flick the fire stick for a cancellable moving shot that deals 10 damage.

02Hold: laser

Hold mouse or fire stick for one second to emit a pillar-blocked beam for 25 damage.

03Same analog frame

Keyboard, joystick, and agent all resolve to the same movement, aim, attack and dash action.

Opponent mode INPUT 02

Difficulty DECISIONS ONLY

All levels use identical speed, damage and cooldowns. Only strategy and reaction timing change.

Observation transport SELECTABLE

gRPC runner DISCONNECTED

LOCAL WEB GATEWAY

Agent target: 127.0.0.1:50051

Live telemetry CONTROLLER NEUTRAL

Game clockREAL TIME

Round time0s

Video transportJPEG FRAMES

Video observationsAS FAST AS AVAILABLE

Controller actionsUNCAPPED

Action memoryUNTIL REPLACED

Render rate60 FPS

Tick0

Slow controllerITS PROBLEM

Game behaviorNEVER WAITS

Training recorder IDLE

Optional: save human play as synchronized video + actions

gameplay.webmactions.jsonl · 10 Hzevents.jsonlmanifest.json

The ZIP stays on this device. Action labels use the same six fields as the live agent protocol.

Action contract ANY CONTROLLER / PROTOBUF

message ControllerAction {
  uint64 sequence;
  float move_x; // left stick
  float move_y; // -1.0 … 1.0
  float aim_x;  // right stick
  float aim_y;  // -1.0 … 1.0
  bool attack;  // stick held
  bool dash;
}

✓The gateway and gRPC server bind to localhost. Frames and model weights stay on this computer.

See pixels.Take action.

See pixels.
Take action.