There is no such thing as artificial intelligence! What is marketed as AI today is little more than pattern recognition—a process that, in its generalized form, is already hitting the limits of feasibility. The correct term would be "Large Language Model" (LLM), though this actually has more to do with vectors than with language. Current large-scale models are reaching a point where adding more parameters no longer yields better results. Simply using ever-increasing amounts of training data does not help; obtaining better training data is essential. However, that is not the topic I wish to address here.
Initially, the focus was almost exclusively on using ever-increasing amounts of data to train the models; consequently, large models today already exceed 600 billion parameters. However, not all of these are utilized during actual operation—one must distinguish between trained parameters and active ones. You can easily calculate the cost—and question the practicality—of keeping over 600 billion 32-bit floating-point numbers in the memory of expensive accelerators. There are two fundamental ways to scale down the model for deployment:
For better results, it is far more effective to restrict the training data to specific topics from the outset—thereby training specialized LLMs—in cases where generalized models reach their limits. The ability to detect tumors or fold proteins is of little use to an agent intended to assist with programming.
To use an LLM effectively, the most important first step is to define your actual objective. An LLM won’t do the work for you, but—provided you define your goals correctly and select the right model—it can offer support in countless ways. It can broaden your perspective, help you focus on a key point, or allow you to leverage diametrically opposed capabilities. A task or project usually involves multiple steps that may differ significantly in their objectives; you don’t have to stick to a single LLM. Each step may entail different requirements that a different model is likely better suited to handle. There is no single "best" model! While a large, generalized model can certainly play to its strengths during the initial research phase of a new topic, a smaller model equipped with a search module won’t lag far behind when pursuing a clearly defined goal—provided your search query is well-crafted.
When choosing an LLM, it is often not just the goals that are important. What data should the LLM process and under what jurisdiction does it work? Are you even allowed to hand over the data to be processed? Data protection and data security must be taken into account right from the start. How much effort do you have to put into preparing data to comply with such laws and is it even worth it? Should the LLM be able to change data with an agent or should it only have read access and generate commands that can be used after a review by an administrator?
Beyond the hype surrounding massive models that require dedicated data centers, it is certainly possible to run optimized LLMs yourself. Tools such as Ollama, vLLM, or LM Studio, for instance, allow you to run countless models locally with just a few simple steps. However, if you want to avoid dealing with a command line in daily use—or even achieve a user experience comparable to apps from major providers—a bit more setup is required. While it is technically possible to run an LLM on a CPU, processing speeds on a GPU are roughly 10–12 times faster, and GPUs convert less energy into heat (though the amount is still significant). Of course, this only works if the desired models actually fit into the graphics memory (VRAM). Simply calculating which model barely fits into the VRAM isn't enough, however; the data being processed and everything classified as "context" must also fit into that same memory space. Rather than getting bogged down in the details here, I will describe the setup I used to test various Q4-quantized models.
For this example, I am using my laptop with a dedicated Nvidia GeForce RTX 3070 graphics card featuring 8 GB of VRAM—a specification likely met by many desktop graphics cards. A Q4-quantized 7B model (7 billion parameters) requires approximately 4–5 GB, leaving sufficient room in video memory for context and data. If your graphics card has more VRAM, you can run correspondingly larger models.
For my Windows 11 setup, I use Ollama as the backend—running within the user profile—and Qdysseus as the frontend, which runs in a Docker container alongside a few additional tools. I therefore assume a working Docker installation; personally, I use Docker Desktop. You will also need Python and a Git client. You can verify these prerequisites using PowerShell, which we will also use for the installation itself.
python --version
git --version
winget install Ollama.OllamaThe installation takes a little while and is complete once the Ollama icon appears in the taskbar or the program window opens. If you do not wish to store the models in your user profile, you can create an environment variable named OLLAMA_MODELS and set its value to an existing directory path for Ollama to use. Note: An already open PowerShell window will not pick up the variable; please open a new one! You can check whether the variable is set—and what its value is—by running the command echo $Env:OLLAMA_MODELS. To make setting up Odysseus easier, go ahead and download a small LLM (currently 2.2 GB) and launch it:
ollama run Phi3:latestOllama also offers a search page for its model repository. ollama ps shows you running models and ollama list shows installed ones, but simply check ollama /? yourself to see which options are available. Now we can turn to installing Odysseus:
git clone https://github.com/pewdiepie-archdaemon/odysseus.git
cd odysseus
copy .env.example .envOpen the .env file in a text editor of your choice and adjust it to your requirements if necessary. Only then do we start the build process (if you are using Docker Desktop, it must already be running):
docker compose up -d --buildThe process is complex and creates a container from multiple images that incorporate several tools (ntfy, searxng, and chromadb). This means features like notifications, web search, and context expansion using files and presets are built right in, bringing the user experience close to that of apps from major providers. Once the build process is complete, you can access Odysseus in your browser at http://localhost:7000/. However, to log in, you will need the credentials generated during the build; you can find them in the logs:
docker compose logs odysseusYou now have everything you need to run some tests yourself, and gradually you will be able to ask the right questions. Ultimately, it always comes down to being able to formulate another question rather than having a definitive answer, doesn't it?
I know that I know not
(Socrates)
The actual answer is already in the first paragraph: an LLM can compare vectors—a vast number of them, and very quickly. Nothing more, nothing less. It does not process data itself, but its output can control tools that do. Anything for which there is a tool controllable by an algorithm can also be operated by an LLM. It is a matter of interfaces—and that is how so-called AI agents work. So, if Skynet were ever to take over the world, it would be because someone—without considering the consequences—created an interface that should never have existed. The real discussion shouldn't be about what LLMs might become capable of, but rather about what we, as a society, deem appropriate to delegate to such tools. If more and more text, images, and videos are generated by LLMs, what will they be trained on in the future? Ultimately, knowledge, art, and creativity are always born of intellect and emotion—things an LLM cannot comprehend. An LLM compares, sorts, summarizes, or combines content that humans have previously created.
Discuss this topic with me on the Fediverse.