Multimodal first
Multimodality is good. At LMMs-Lab we believe it is a necessary condition for intelligence, and The Harness is designed around that belief. Every modality an environment offers keeps its own form, and each part of the system uses the one that serves it.
The world that is acted on is multimodal
Section titled “The world that is acted on is multimodal”Whatever acts meets more than text. A screen is a picture. A scene has space and sound. A machine has readings. A game changes whether or not anyone describes it.
Language is one of the ways the world shows itself, and a strong one, but it is one among many.
Perception and action are both multimodal
Section titled “Perception and action are both multimodal”To act is to close a loop: perceive, decide, act, and perceive what changed.
What comes in may be seen, read or measured. What goes out may be a sentence, a keystroke or a motion. Intelligence in action is the ability to connect them: to read an instruction and move, to see a change and say what it means.
Designed for it from the start
Section titled “Designed for it from the start”The Harness does not start from one modality and add the others later. Three rules follow from that.
- Each modality keeps its own form. What an environment offers reaches the agent as the environment has it. Nothing is turned into another modality first.
- No modality comes first. Each step uses the one that serves it: it reads where the answer is written, looks where it is drawn, and measures where it is a quantity.
- A new modality needs no new system. An environment says what it offers and what it accepts, and the same control loop runs on it.
What this gives
Section titled “What this gives”One agent works on a desktop, a website, a game and a machine, which share little besides the need to act. Each differs in what it shows, in what it accepts and in its pace, and the agent’s way of working stays the same.
Models perceive and produce more modalities every year. The Harness is already shaped for them: a model that takes in more is given more, with no change to the loop.
Where to go next
Section titled “Where to go next”- Automatic Control: the loop that perception and action close.
- Observations and views: what an environment reports today, and the calls that put a view in front of the model.
- Add an environment: how a new environment says what it offers and accepts.