Designing AI Systems with Long-term Context

No matter how much tooling we wrap around them, LLMs are fundamentally stateless functions, incapable of maintaining memory.

Stateless Intelligence

Let $f$ represent a LLM.

Then suppose you have some prompt $p$ and response $r$:

To have a persistent user experience, and to do tasks that aren’t one shot, you would need some merge function

You can’t just continually have $m(p,r) = \text{concatenate}(p,r)$

because you’ll exceed the context size, and even if it fits in context, LLM quality degrades as non-relevant content accumulates in the context.

The Merge Function

A good merge function $m$ might call $f$, but it is unlikely that $m$ would just be

for some constant prompt $q$. This is because $m$ needs to be clever in deciding what context is relevant, and not all context can fit in $q$.

An example implementation of $m$ could involve calling $f$ multiple times with different segments of the longterm context and combining the outputs. There are a lot of non-trivial choices in designing the merge process that is fundamental and unique to the user experience.

Future-Proofing

Thus, well-designed $m$ functions will stay relevant even as LLMs improve, as human-AI interactions are expected to become increasingly persistent.

Similarly, deciding which parts of $p$ and $r$ to retain and in what format will remain fundamental.

Note: This approach to persistent memory is goign to be central to what we’re building at revvresearch.com.