The agent wants to reach the flag (+1) without falling into a trap (−1), and every step costs it a little. Each cell shows its value (how much it expects to gain from there: green good, red bad) and its policy (the arrow: what it would do). Paint walls, mud and traps by dragging, and move the sliders to see how its decisions change.
If the agent knows the world (where everything is and how much it slips), it can work out the value of each cell without taking a step. Each sweep applies the Bellman equation: the value of a cell is the best it can get with one action, counting the reward on arrival and the value of where it ends up, discounted by γ. Value spreads out from the goal, one cell further with each sweep, until it stops changing.
Here the agent knows nothing about the world: it walks and learns from what happens to it. It keeps a Q table with what it expects from each action in each cell and, after every step, nudges it a little (α) towards the reward it got plus the best it expects from where it landed. With probability ε it tries a random action (explores); otherwise, the best one it knows (exploits). Without exploring it sticks with the first thing that worked; exploring too much, it never makes use of what it learned.
γ (discount): how much the future matters. With a low γ, a distant goal is worth almost nothing. Slip: the probability that the action fails and goes sideways; on a slippery floor, the agent keeps away from traps even if it takes longer. Step reward: if every step costs a lot, the agent is better off jumping into a nearby trap than taking a long detour.
The agent knows the map: it works out how much each cell is worth with the Bellman equation, one sweep after another, without taking a step.
Press «One sweep» and watch the value leave the goal and spread one cell further with each sweep.