The importance of looking after your LLM's context window

When you send a chat message to a large language model (LLM), the text you enter gets converted into a list of numbers - each one representing a fragment of text. The LLM then generates a response as a list of numbers, which then get converted back to each number's corresponding text fragment - so you (as a human) are able to read it. Each one of these numbers is called a token, and has a cost associated with it. To send a new message, you have to create a new list of tokens made up of all the previous chat history plus the new message's tokens. This means in order to send each new chat message, you will have to send a larger and larger list.

Your LLM generates a response - one token at a time - by considering all the previous tokens. The number of tokens an LLM can consider at any one time is known as the context window and is finite. This means if your chat history gets too long, your LLM will not be able to consider all the tokens. As well as not being able to consider all the tokens, there are other problems:

• As the context window gets bigger, the attention the LLM pays to each token gets diluted - so it becomes less able to focus on the relevant parts.
• The LLM puts most of its attention towards the start and end of a context window. This means relevant context, as the context window gets larger, might get lost in the middle.

The above scenarios can all contribute to an LLM giving low quality or false output - also known as a hallucination. To solve these problems, you have to essentially try and keep your context window as small as possible at all times. There are various techniques you can use to do this, which I will discuss in future posts. But for now all you need to know is that your LLM works best when it's context window is small - the smaller the better.