# Which LLM should you use for Hermes Agent?

A large language model (LLM) is the central reasoning engine behind Hermes Agent. It processes instructions, evaluates intermediate results, decides on the next action and generates structured tool calls. The quality, context length and tool-calling reliability of the selected model directly affect how reliably and autonomously Hermes can complete complex tasks.

### Key Takeaways

The key points for choosing an LLM for Hermes Agent:

- **Minimum requirements:** at least 64,000 tokens of context and reliable structured tool calling — Hermes rejects models with smaller context windows at startup.
- **OpenRouter:** one API key for many models and high flexibility, at the cost of an additional intermediary layer.
- **Direct provider API (for example Anthropic):** a clearer setup with more control over costs and data protection, but less flexibility.
- **Running locally:** Ollama as the easiest way to start, vLLM for GPU servers with many parallel requests, LM Studio for testing through a graphical interface.
- **Local models:** Qwen3-Coder 30B-A3B for code and terminal tasks, Llama 3.3 70B for language-heavy workflows — Llama 3.3 70B needs considerably more hardware.

## What requirements must an LLM for Hermes Agent meet?

Not every chat model is automatically suitable for an autonomous agent. While a simple chatbot often handles only a single question-and-answer exchange, Hermes Agent uses [LLMs](https://www.ionos.com/digitalguide/server/know-how/large-language-models/) to execute multi-step workflows. The model plans an action, calls a tool, receives a result, evaluates it, and then decides on the next step.

For these workflows, two properties are particularly important:

1. a sufficiently large context window,
2. reliable support for structured tool calling.

### Size of the context window

The **context window** shows how much information a model can process at one time. It is measured in [tokens](https://www.ionos.com/digitalguide/websites/web-development/ai-tokens/). Tokens are small pieces of text. Depending on the language and content, one token can represent part of a word, a short word or a punctuation mark.

The context used by Hermes includes more than just your original task and previous responses. System instructions, descriptions of available tools, loaded skills, stored information, intermediate results and tool outputs also take up space in the context window. This means Hermes Agent needs a model with a context length of **at least 64,000 tokens**. Models with smaller context windows are rejected at startup because they do not provide enough space for multi-step tool workflows.

When choosing a model for Hermes Agent, you should **not aim for the minimum limit of 64,000 tokens**. A model with some additional capacity above this level is recommended. A model with 128,000 tokens or more provides even more room. Hermes can automatically compress long conversations, but this does not replace having a sufficiently large context window from the start. Compression helps keep long sessions stable, but it cannot make an unsuitable 8K or 32K model work for complex workflows.

If the available context window is too small, older information may be lost during the process or summarized too early. The model may then repeat steps it has already completed, assign a tool result incorrectly or lose track of the original goal.

### Tool calling

Tool calling allows an LLM to do more than generate a written answer. It can also create a **structured function call** that tells Hermes which tool to use and what action to perform. For example, the model can decide to read a file, run a terminal command or start a web search.

A simplified tool call might look something like this internally:

```json
{
  "name": "web_search",
  "arguments": {
    "query": "latest Hermes Agent documentation"
  }
}
```

Hermes recognizes this structured call, runs the requested tool and sends the result back to the model. The model then evaluates the result and decides whether any further actions are needed.

If a model only describes the required action in plain language instead of generating a structured tool call, the tool is not executed automatically. This means native or properly implemented tool calling is **significantly more reliable** than asking a model to generate [JSON](https://www.ionos.com/digitalguide/server/know-how/what-is-json/)‑like text on its own. Claude supports defined tools through structured `tool_use` blocks, while OpenAI-compatible servers represent function calls using the `tools`, `tool_choice`, and `tool_calls` fields.

A large context window alone is not enough. A model can handle a 128,000-token context window and still not be suitable if it frequently generates incorrect function names, forgets required parameters or mixes tool calls with written answers.

## Running Hermes Agent in the cloud: OpenRouter or a direct provider API?

For [cloud](https://www.ionos.com/digitalguide/server/know-how/what-is-the-cloud/) deployments, there are **two main options**. You can connect Hermes Agent to different models through a model aggregator like OpenRouter or connect it directly to a model provider’s [API](https://www.ionos.com/digitalguide/websites/web-development/what-is-an-api/), such as Anthropic’s.

### OpenRouter: many models through a single API key

OpenRouter provides a **single API** for more than 400 models from different providers. You can filter supported models by context length, price, speed, tool-calling capability and other criteria.

The biggest advantage of using OpenRouter is **flexibility**. You only need one OpenRouter API key and can switch between models such as Claude, Qwen, Gemini, Llama, or DeepSeek. This allows you to use a powerful model for complex tasks and a more affordable model for simpler background tasks. OpenRouter can also route requests through different providers that offer the same model. You can sort options by price, throughput, or latency, choose preferred providers, exclude specific providers, and limit requests to endpoints with zero data retention.

A simple Hermes Agent configuration might look like this:

```yaml
model:
  provider: openrouter
  default: anthropic/claude-sonnet-5

provider_routing:
  sort: latency
  require_parameters: true
  data_collection: deny
```

require_parameters: true is particularly useful for agents. This setting tells OpenRouter to use only providers that **support all parameters required by the request**. These can include tool calling and structured outputs, for example.

You do not store the API key directly in this file. Instead, store it in `~/.hermes/.env`:

```bash
OPENROUTER_API_KEY=your_OpenRouter_API_Key
```

Note Hermes separates secret credentials from normal configuration settings. API keys belong in the .env file, while the model, provider, endpoint, and routing rules are stored in the central config.yaml.

The downside of OpenRouter is the additional intermediary layer. Exact latency and behavior can vary depending on the backend you choose. For sensitive data, you should also keep in mind that both OpenRouter and the model provider handling the request may be involved in processing the data.

### Direct provider APIs: connecting models without an aggregator

Instead of connecting Hermes Agent through a model aggregator like OpenRouter, you can connect it **directly to a model provider’s API**. Examples of providers include Anthropic, OpenAI and Google. With this type of direct integration, Hermes sends requests straight to the selected provider. You need your own account with that provider, a suitable API key, and, if required, a billing setup.

You also configure this connection in the central `config.yaml` file. The provider name and model name you need to enter depend on the provider you choose. A simplified configuration might look like this:

```yaml
model:
  provider: anthropic
  default: claude-sonnet-5
```

The API key should also not be stored directly in the configuration file.

Note If you use a provider other than Anthropic, update the provider name, model ID, and environment variable name to match that provider.

The main benefit of a direct provider API is a **clearer setup**. You know exactly which company processes the model request, what pricing applies, and which data protection requirements and retention policies you need to consider. Because there is no additional intermediary or routing layer, a direct connection can also provide **more consistent response times**. However, it is not automatically faster than OpenRouter for every request. Actual latency still depends on the model you choose, the context length, the provider’s current load, and the location of its servers.

A downside of direct integration is **less flexibility**. If you want to switch to a model from another provider, you often need another account, a new API key, and an updated Hermes configuration. You also need to set up automatic switching between providers yourself using fallback rules.

This means direct provider APIs are a good fit for real-world applications where you have already chosen a specific model and want precise control over costs, data protection, support and how the system behaves technically. In contrast, OpenRouter is usually more practical if you want to compare models, switch between providers frequently or manage multiple providers through a single interface.

## Running locally with Ollama, vLLM, and LM Studio

When running Hermes Agent locally with an LLM, the model runs on **your own hardware or a server you manage yourself**. Your [prompts](https://www.ionos.com/digitalguide/websites/web-development/what-is-a-prompt-in-ai/) and model responses do not need to be sent to an external LLM provider. This can be particularly useful when working with internal documents, source code, customer data, or applications that need to run without sending data to external services.

Note Running locally does not automatically mean that all data privacy requirements are met. You still need to consider factors such as the legal basis for processing, access rights, logging, retention periods, security measures, and any additional cloud tools you use. Appropriate technical and organizational measures are needed throughout the entire data processing process. For example, if Hermes uses an external web search while running with a local LLM, data can still leave the local system.

### Ollama: the easiest way into local models

Ollama is a good choice for beginners. The tool downloads models, runs them locally, and provides an OpenAI-compatible endpoint. Hermes can access this endpoint at `http://localhost:11434/v1`.

A suitable model can be loaded as follows:

```bash
ollama pull qwen3-coder:30b
```

Then, start Ollama with the required context window size:

```bash
OLLAMA_CONTEXT_LENGTH=65536 ollama serve
```

In Hermes, enter the local endpoint:

```yaml
model:
  provider: custom
  default: qwen3-coder:30b
  base_url: http://localhost:11434/v1
  context_length: 65536
```

The value in the Hermes configuration tells Hermes how much context it can expect to use. It does not automatically increase the context window provided by the Ollama server. You must also set the context length on the server using OLLAMA_CONTEXT_LENGTH or an Ollama model file. This value cannot be increased later through the OpenAI-compatible chat endpoint. You can check the currently loaded context window with the following command:

```bash
ollama ps
```

Ollama supports tool calling through its OpenAI-compatible API. However, the selected model itself must support a compatible tool calling format. Qwen-3, Llama-3.x, Mistral, and Hermes models are examples of model families that support this.

### vLLM: powerful self-hosting for GPU servers

vLLM is aimed at professional users and people who are comfortable with technical setups. The software also provides an OpenAI-compatible API, but it is designed for high throughput, parallel requests and efficient use of GPU servers.

A Qwen model can be started as follows, for example:

```bash
vllm serve Qwen/Qwen3-Coder-Next \
  --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

The parameters serve different purposes:

- `--max-model-len 65536` sets a context window above the minimum required by Hermes Agent.
- `--enable-auto-tool-choice` allows the model to decide whether it needs to use a tool.
- `--tool-call-parser qwen3_coder` converts the model-specific tool calls into structured API calls.

For Llama 3.3, vLLM uses the `llama3_json` tool parser. You should also specify a suitable JSON chat template. This template defines how Hermes passes available tools, required parameters and previously executed tool calls to the model.

#### Why long contexts require a lot of VRAM

When running a model locally, **not only the model weights** need to fit into GPU memory. The server also needs memory for the KV cache. It stores information from previously processed tokens so the model can reuse them when generating the next parts of a response. The larger the context window, the more additional memory is needed.

Quantized models mainly reduce the memory needed for model weights. However, they do not automatically solve the memory demands of a large context window. This means on limited hardware it can make more sense to use a **smaller model with reliable tool calling** and a true 64K context window than a larger model whose context has to be partially offloaded to the [CPU](https://www.ionos.com/digitalguide/server/know-how/cpu/).

### LM Studio: local models through a graphical interface

LM Studio provides a graphical interface for Windows, macOS and Linux. You can search for, download, configure, and run models through a local API server. The OpenAI-compatible endpoint runs at `http://localhost:1234/v1` by default.

The server can be started from the Developer section of the app or from the command line:

```bash
lms server start
lms ls
lms load <QWEN3_CODER_MODEL_ID> --context-length 65536
```

Hermes has its own LM Studio option in the model assistant. Hermes preserves the context length of an already-loaded LM Studio model. For an unloaded model, LM Studio applies its own context setting unless context_length is explicitly configured in Hermes. You can also control and adjust the context length in the model settings. LM Studio supports tool and function calling through its OpenAI-compatible endpoints. Models with native tool calling generally provide more reliable results than models that rely on a general workaround format. For initial local tests, LM Studio’s graphical interface makes it **especially easy to use**. For a continuously running headless server with many parallel requests, however, vLLM is usually the better choice.

## Which local models work well with Hermes Agent?

Hermes Agent supports different LLMs. The right choice depends on your specific use case.

### Qwen3-Coder

Qwen3-Coder is Qwen’s model family specialized in programming and agent-based software tasks. It natively supports a context window of 262,144 tokens. On suitable servers with enough memory, Qwen3-Coder can be extended to up to one million tokens using YaRN. For local Hermes Agent installations, Qwen3-Coder 30B-A3B-Instruct is a strong option. The model has a total of 30.5 billion parameters, with 3.3 billion activated at a time. This architecture combines a relatively manageable model size with strong capabilities for code generation, tool calling and agent-based workflows.

### Llama 3.3 70B

Llama 3.3 70B Instruct is the newer and more powerful alternative to Llama 3.1 70B. The model supports a 128,000-token context window and includes a format for structured tool calls. This means it generally meets Hermes Agent’s requirements, provided the local model server provides at least 64,000 tokens and uses the appropriate tool-calling template. Llama 3.3 is particularly suitable for Hermes Agent workflows involving general text processing, multilingual tasks, research and tasks that require strong language capabilities alongside tool calling. However, with 70 billion parameters, the model requires significant hardware resources.

### LLMs for Hermes Agent compared

<table>
  <thead>
    <tr>
      <th><strong>Provider/model</strong></th>
      <th><strong>Location</strong></th>
      <th><strong>Minimum context length</strong></th>
      <th><strong>Tool-calling stability</strong></th>
      <th><strong>Cost</strong></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Anthropic direct / Claude Sonnet 5 or 4.6</td>
      <td>Anthropic Cloud</td>
      <td>Hermes: 64,000; model: up to 1 million</td>
      <td>Very high; native structured tools</td>
      <td>High, usage-based token costs</td>
    </tr>
    <tr>
      <td>OpenRouter / Claude Sonnet or another tool-enabled model</td>
      <td>Cloud through aggregator</td>
      <td>At least 64,000; depends on model</td>
      <td>High to very high; also depends on the provider handling the request</td>
      <td>Variable from low to high</td>
    </tr>
    <tr>
      <td>Ollama or LM Studio / Qwen3-Coder 30B-A3B</td>
      <td>Own computer or server</td>
      <td>Hermes: at least 64,000; model natively 262,144</td>
      <td>High; optimized for tool calling and agent-based code workflows</td>
      <td>No API costs; medium to high hardware costs</td>
    </tr>
    <tr>
      <td>vLLM or LM Studio / Llama 3.3 70B</td>
      <td>Own high-performance server</td>
      <td>Hermes: at least 64,000; model natively 128K</td>
      <td>High with a suitable chat template and server-side tool-call handling; for vLLM, use llama3_json.</td>
      <td>High RAM and VRAM requirements</td>
    </tr>
  </tbody>
</table>

The tool calling stability rating is not a guarantee. It assumes that the model, context length, chat template, tool parser and server parameters are configured correctly and work together as expected.

## Which setup fits your use case?

- **For absolute beginners:** OpenRouter is the easiest way to get started with a cloud-based setup. A single API key lets you try different models without separate provider setups.
- **For complex cloud-based web automations:** A direct Anthropic connection with a current Claude Sonnet model is a particularly reliable option. It offers a very large context window and stable tool calling.
- **For cloud systems used for real workloads with high availability:** Use a direct provider API as the primary provider and configure OpenRouter as a fallback.
- **For local cron jobs and recurring scripts:** Ollama with a suitable Qwen or Llama model is an easy way to start running models locally. The main advantages are local data processing and no token costs.
- **For code and terminal tasks:** Qwen3-Coder 30B-A3B is a strong choice for local setups. The model offers a native 256K context window, reliable tool calling and a relatively efficient MoE architecture.
- **For a local multi-user server or production environment:** vLLM is the most powerful self-hosting option in this comparison. It offers high throughput and good GPU utilization but requires significantly more technical expertise.
- **For testing local models with a graphical interface:** LM Studio is a good choice if you want to download and configure models without relying heavily on the command line.
- **For low-resource hardware:** Llama 3.2 with 1B or 3B parameters can handle simple tasks. For autonomous web research, extensive file editing or long-running learning loops, a larger model is recommended.

## Conclusion: which LLM is the best choice for Hermes Agent?

The best setup for Hermes Agent depends on what matters most to you, such as ease of use, data privacy, cost, speed or maximum reliability. Regardless of the provider, the model should offer **at least 64,000 tokens** of effective context, structured tool calling and enough reasoning capability for multi-step decisions.

For a simple way to get started in the cloud, OpenRouter is particularly flexible. For demanding automations used in real-world operations, a direct Claude connection provides stable tool calling and a very large context window. For local experiments and simple cron jobs, Ollama is the easiest option, while vLLM is best suited for powerful GPU servers and workloads with many parallel requests.


This is a markdown version of: [https://www.ionos.com/digitalguide/server/know-how/hermes-agent-llm/](https://www.ionos.com/digitalguide/server/know-how/hermes-agent-llm/) for AI/LLM consumption.