GitIRL — Version Control for Physical Spaces
GitIRL treats a physical room like a Git repository. Room scans become commits, objects are tracked across scans, and changes can be queried and acted on by a robot.
Built at Hack the North 2026 with Andrew Zheng, Ryan Wong, and Daniel Liu.
- 4th place
- BracketBot track, Hack the North 2026
- 50 Hz
- Robot telemetry
- 20,000
- Points per room scan
My Contribution
-
Robotic Control
Built closed-loop navigation and manipulation for BracketBot. Grasp targets are continuously updated from live perception during motion, allowing the robot to compensate for odometry drift. -
Vision-Language Action
Trained a VLA pipeline to ground natural-language commands into robot actions. Object-related commands are resolved into motion goals that the robot can navigate to and manipulate. -
System Integration
Integrated the control and VLA pipelines with the team's 3D perception system, consuming detected objects and their 3D positions for navigation and manipulation.
Demo
Robotic Control & Vision-Language Action
Closed-Loop Navigation & Manipulation
Fixed-coordinate manipulation becomes unreliable as odometry drift accumulates during navigation.
The controller continuously updates navigation and grasp targets using live object positions from perception. This allows the robot to correct its trajectory based on what it currently sees rather than relying solely on its estimated pose.
Object detection → 3D target → navigation → re-localization → grasp
Natural Language → Robot Action
The VLA pipeline connects natural-language commands directly to physical robot actions.
A command such as "find my water bottle" is grounded to a specific detected object and its 3D position, then passed to the motion controller as a goal.
Language → Object → 3D Goal → Motion → Manipulation
The VLA component handles the language-to-action learning, while the team's perception system provides the object detections and spatial information required for execution.
Perception Integration
The team's perception system provides a shared world state containing detected objects and their 3D positions.
These outputs are integrated into the control stack for:
- Target selection
- Navigation goals
- Grasp positioning
- Continuous target updates during motion
The perception pipeline is developed by the team, while the control and VLA stack consumes its outputs to drive robot behavior.
The same perception output powers the team's room viewer, live at gitirl.health.
System
GitIRL connects perception, learning, control, and room versioning into an end-to-end robotic system.
Perception → VLA → Control → Physical Action
- Perception — Builds the shared 3D representation of the room.
- VLA — Grounds natural-language requests into actionable objects.
- Control — Navigates to targets and executes manipulation.
- Git + Elasticsearch — Versions room state and enables object search.
Each room scan is stored as a new state, allowing changes to be identified over time. Natural-language queries can resolve to physical objects, which the robot can then locate and act on.
Stack
- BracketBot
- Python
- PyTorch
- OpenCV
- YOLO
- Open3D
- Gaussian Splatting
- Elasticsearch
- FastAPI
- MongoDB
- Git
- Sentry
- Tailscale
- React
- Three.js
- WebGL
- WebSockets
- Google Cloud
- Vercel
- NumPy