Text-guided visual navigation of a humanoid robot
Give a humanoid robot directions the way you would give them to a person: “go straight, and at the next junction turn right”. TalkToWalk aims to fine-tune a vision-language-action (VLA) model, such as NVIDIA GR00T N1, so that a Unitree G1 can walk such a route through a real building, guided by its camera and the text alone, without collisions.
Go straight. At the next junction, turn right, then right again, go straight, turn right, turn right and stop where you started.
Humanoid robots are increasingly controlled by vision-language-action (VLA) models: networks that look at camera images, read an instruction and output actions. Open foundation models such as GR00T N1 are trained mostly for manipulation with the hands. This thesis adapts one of them to a different job: walking through a specific building along a route that is described only in words.
Input
Camera image
What the robot sees right now.
Input
Route in text
The command it is following, for example “at the next junction, turn right”.
Model
Fine-tuned VLA model
GR00T N1, with an extra output head that recognizes junctions.
Output
Walking commands
Walk forward, turn, stop. The G1’s own controller keeps the robot balanced.
Feedback during fine-tuning
LiDAR distances keep the robot clear of walls, people and other obstacles. Corrections from a person show what a good run looks like.
The design is not final. This is the current plan, and it will change as the first experiments come in.
The route is a short sequence of commands, such as “go straight”, “at the next junction turn right” and “stop”.
At every step the model gets the current camera frame together with the command it is following.
A fine-tuned GR00T N1 turns the image and the command into a walking action. An extra output head answers one question: am I at a junction right now?
At a junction the robot carries out the current command (straight, left or right), and the next command in the sequence becomes the model’s input.
The action goes to the G1’s built-in walking controller through Unitree SDK2 and ROS 2, so the model never has to balance the robot itself.
LiDAR measures the distance to walls, people and other static or moving obstacles. Together with corrections from a person, it gives the feedback for fine-tuning.
A place where the robot has a real decision to make:
Recognizing junctions reliably is the first research milestone, because every command in the route is tied to one.
Most of the work happens in NVIDIA Isaac Sim, with a model of the G1 and of the building’s corridors.
From simulation and from a few routes walked with the real robot under manual or semi-manual control.
Supervised learning, reinforcement learning or a combination of both; not decided yet.
A full lap of the corridor loop in the faculty building, with static and moving obstacles.
Planned submission: spring 2027. This list is updated as the work moves on.
Done
Thesis assigned in October 2026; the core papers on VLA models and humanoid navigation reviewed.
Next
Power and safety routine, the ROS 2 interface, streaming the camera and LiDAR, high-level walking commands.
Planned
The G1 and a corridor environment in NVIDIA Isaac Sim.
Planned
A dataset from simulation and real routes; an extra output head on GR00T N1.
Planned
Fine-tuning GR00T N1 to follow a sequence of commands.
Planned
A full lap of the corridor loop in the faculty building, with static and moving obstacles.
Planned
The written thesis; code and, where possible, data and models published in this repository.
The three papers the thesis builds on, as listed in the thesis assignment.
arXiv:2503.14734
The model to fine-tune. A slow vision-language “System 2” understands the scene and the instruction; a fast diffusion-transformer “System 1” turns that into motor actions.
arXiv:2601.12790
Collision-free local navigation on the same robot, the Unitree G1, with attention guided by predicted waypoints.
PMLR 229, pp. 711–733
Shows that a pre-trained navigation model can be adapted cheaply to new environments and new kinds of goals, including directions such as “turn left”.
This thesis builds on experience from my year project, where I built a reinforcement learning pipeline from scratch.
Year project at FMFI UK, page in Slovak
From a DQN agent for Snake to a full CNN and Monte Carlo tree search self-play pipeline for the Kaggle Hungry Geese competition, benchmarked against a gold-medal public agent. The experience with training loops, self-play and careful evaluation carries straight over to this thesis.
The thesis is still being written. If you build on this repository before it is finished, please cite it as:
@misc{monis2026talktowalk,
author = {Moni{\v{s}}, D{\'a}vid},
title = {{TalkToWalk}: Text-Guided Visual Navigation of a Humanoid Robot},
year = {2026},
howpublished = {\url{https://github.com/DavidMonis/talk-to-walk}},
note = {Bachelor's thesis in preparation, Comenius University Bratislava}
}