Hy-Memory Setup Guide: Six-Layer Memory for AI Agents on Hermes

I run five AI agents on my homelab. They manage infrastructure, handle email, run automations, and show up as different personalities in different chat rooms. They’re useful. They also used to forget everything about me between sessions.

That was always the weakest part of running agents. Not the model quality, not the tooling, not the cost. Memory. Close a conversation and the agent comes back as a stranger. It knows your name if a config file told it, and nothing else.

Context windows don’t fix this. They’re a scratchpad that resets. What I needed was a real memory system: something that keeps facts, tracks how those facts change over time, and can recall them in a fresh session months later.

Here’s what happened when I went looking in August 2026, and why Hy-Memory, Tencent Hunyuan’s memory framework, won. I’ll cover what it actually is, why I picked it over what I had, how the setup went, and the benchmark I ran before trusting it. Then the things that broke. A few of them broke in ways that were almost silent. Those are the important ones.

What I was running before

The fleet ran two memory providers before this one. Mnemosyne on my main agent and three of the bots. Hindsight on one of the bots. Both taught me something about what I actually needed.

Mnemosyne was the workhorse, and it was a good one. Local SQLite file, recall in under a millisecond, and the richest tool surface of any provider I’ve used: persona tiers, canonical facts, triples, real contradiction handling with veracity scoring. After a heavy curation pass it went from a 754 MB database down to about 20 MB and a clean bank of roughly 700 memories. Its weakness was structure. Facts accumulated flat. When a preference changed, the old fact stayed around and the new one got added on top. If I asked the agent why it believed something, there was no chain of events to follow. Memory accumulated, but it didn’t evolve. Run agents daily and accumulated memory slowly turns into noise.

Hindsight was the benchmark darling. A 94.6% LongMemEval QA score and a 92% on LoCoMo, the kind of numbers that make you look twice. Knowledge graph, consolidation rounds that reflected on what it had stored. It was also the one that hurt the most to operate. Docker plus Postgres plus a heavy embedding stack, an API daemon sitting on the better part of a gigabyte of RAM, and a string of lifecycle bugs: silent memory deletion in append mode, no way to delete a single memory without nuking the bank, duplicate-handling issues closed as won’t-fix. The official workaround for one of those bugs dropped two of the three memory types. It stayed on one bot until the fleet moved, and migrating off it was its own project. No export tooling, so I wrote a pull script and deduplicated by hand, because it stored the same claim as several different fact types. Benchmarks great, life is pain. That’s my summary of Hindsight.

Here’s how the three stack up.

One thing before you read too much into that table. The benchmark numbers only compare when the harness and the metric match, and they don’t. Mnemosyne’s 98.9% is retrieval recall, whether the right chunk comes back. Hy-Memory’s 85.20% is QA accuracy, whether the LLM can answer from what it retrieved. Hindsight’s 94.6% is QA on yet another setup. Those are different stages of the same pipeline. Nobody publishes the same quantity, so “who’s number one” is mostly marketing. What none of those numbers capture is the thing that actually decided this for me.

What Hy-Memory is

Hy-Memory comes from Tencent, released by the TencentDB team under the Hunyuan brand. First release was May 2026. It’s MIT licensed and ships as a pure-Python package on PyPI. Notable detail: there’s no public GitHub repository. That’s unusual for a framework you’re considering for production, and it was the first thing I checked. The missing repo bothered me less once I found out a product team at Tencent stands behind the release, not a solo dev. It’s still a closed development process, and that’s the trade you make. The code is readable inside the wheel if you want to audit it; there’s just no issue tracker and no public changelog.

The core idea is a six-layer memory structure:

• L1: raw traces, the original conversation text
• L2: atomic facts, extracted and searchable
• L3: identity, stable preferences and traits about the user
• L4: summaries, a condensed version of each session
• L5: mental models, patterns inferred from behavior
• L6: intentions, forward-looking predictions about what the user wants next

Two systems do the work. System 1 runs inline, during the conversation, writing layers L1 through L4 within milliseconds of you speaking. System 2 works in the background, takes the raw material System 1 left behind, and builds L5 and L6 over time. It’s basically Kahneman’s dual-process theory applied to memory storage.

There are three modes. Lite does embedding-only writes, no LLM calls. Pro adds extraction and reconciliation driven by an LLM. Ultra throws in System 2 and a graph store. You want ultra if you want the full architecture. Lite is write-only in practice for a Hermes agent, so don’t start there.

It speaks the OpenAI API on both the LLM and embedding side, which matters if you route all your model traffic through a gateway like I do. Vectors go in ChromaDB by default, with Qdrant and FAISS as options. The graph layer runs on Kuzu or Neo4j.

Why the evolution chains matter more than the layers

The layers are nice, but the feature that sold me is evolution chains.

When a memory changes, Hy-Memory links the new version to the old one with a supersedes pointer. Hit any node in that chain during search, and the whole causal history unfolds with it. Vector similarity can’t do this. Semantic search finds things that sound similar; it can’t reconstruct the arc of how an opinion formed.

Their docs use a great example. A musician tries Spotify and loves it. Then the algorithm files their work under background music, and the frustration starts. They leave for Bandcamp. Later they press a limited vinyl run and it sells out. An agent with flat memory recalls “used Spotify, liked it” and suggests going back to Spotify. An agent with an evolution chain knows the full arc: embrace, disappointment, departure, and the reasoning behind each step. One of these gives advice that lands. The other walks the user back into a decision they already regretted.

That’s the difference between a memory system and a filing cabinet. I wanted the former.

One honest caveat about those vendor numbers. 85.20% on LongMemEval, 76.91% on PersonaMem, ahead of mem0 and Graphiti. I didn’t reproduce any of them. It’s vendor-published, so treat it as directional, not gospel. That’s exactly why I benchmarked my own workload before switching.

What else makes Hy-Memory different

Evolution chains are Hy-Memory’s headline feature, but they’re not the only thing that’s different once you actually run it.

Memory gets rewritten instead of appended. Every write goes through a reconciler that merges the new facts with what’s already stored. The same claim shows up again, it gets folded into the existing record instead of stored twice. A contradiction arrives and the old fact gets updated on the spot. I saw this in the migration numbers: 217 curated entries went in, the bank settled at 199 memories, because related facts got merged together. With Mnemosyne that kind of cleanup was a maintenance chore I had to schedule. With Hy-Memory it’s part of the write path. It’s also where a couple of my worst bugs lived, but the intent is real.

The top layer looks forward. L6 holds intentions, predictions about what the user wants next, and they carry expiry dates. It’s the only memory system I’ve used that tries to remember what you’re about to do, not just what you’ve done. I’m still forming an opinion on how much that helps, but it’s a different bet than the others are making.

It’s built for many agents on one server. The namespace is user_id plus agent_id, which is what lets me run five agents against the same framework without their facts bleeding into each other. Most memory providers assume one agent per store. That assumption is why the fleet migration felt like a project instead of a weekend.

And the design goal is fewer, denser memories, not a growing pile. The vendor benchmarks claim a fraction of the entries of mem0 or Graphiti at higher accuracy. I treat those the same way I treat all vendor numbers, but the direction is right: the system wants to know less, better.

There’s a side effect that matters more than the dollar figures. The init context got smaller. Mnemosyne shipped thirty-six memory tools into my system prompt every turn; Hy-Memory ships about five. In my setup that came to roughly ten thousand tokens before a conversation even starts. Smaller context, better recall. I’ll take that trade.

Setting up Hy-Memory on Hermes

Hermes has a first-party plugin, hermes-hy-memory. The setup path is straightforward:

pip install "hermes-hy-memory[init]"

The plugin runs a small HTTP sidecar that is the actual durable writer. Hermes talks to the sidecar, the sidecar talks to the LLM and embedder. Key environment variables:

HY_MEMORY_USER_ID=<your-user-id>
HY_MEMORY_AGENT_ID=hermes
HY_MEMORY_MODE=ultra
MEMORY_LLM_MODEL=deepseek-v4-pro
MEMORY_LLM_BASE_URL=https://your-gateway/v1
MEMORY_EMBEDDER_MODEL=qwen3-embedding-8b
MEMORY_EMBEDDING_DIMS=4096
MEMORY_VECTOR_STORE=chroma

Three setup decisions that mattered:

1. Run the sidecar as a systemd unit with Restart=always, not as a child of your terminal session. Session children die when the session dies. The sidecar is the thing that holds your memory writes, so it should be supervised like a database.

2. Set both HY_MEMORY_MODE and MEMORY_MODE. The plugin reads one, the SDK server reads the other, and if they disagree the system silently runs in a lower mode than you think. I’ll get to this in the “what broke” section, but the short version is: verify what the sidecar actually logged at startup, don’t trust what you configured.

3. Route keys through your secrets manager and inject at runtime. No memory framework needs plaintext keys in a compose file or an env file in your repo.

For the migration from Mnemosyne, I exported the old bank and filtered it down to what was worth keeping: global scope or high importance, which cut about 800 entries to a curated 217. Then I fed them through the sidecar with parallel workers. Ingest runs about 13 to 30 seconds per entry because each one triggers an LLM extraction call. Background it, don’t watch it.

Don’t trust the defaults. We benchmarked.

The vendor recommends a specific Hunyuan model for extraction. I ran their exact recommendation against 12 model configurations on my real, production prompts before committing. The results were not what the docs suggested.

Cost is per day at roughly 47 extraction calls, the pipeline’s actual volume. “Reasoning” is the share of output tokens burned on thinking rather than content.

For structured extraction, reasoning mode is strictly worse. Same model, thinking on versus off: deepseek-v4-pro extracted 22 memories with thinking off and zero reasoning tokens. With thinking on it extracted 16 and spent 73% of its output on reasoning. The models that reasoned the most were the slowest and the least productive. kimi-k2.6 came closest in quality, but at five times the cost it wasn’t close on value. qwen3.5-flash was unusable, valid JSON between 0% and 17% of the time.

One more quirk. The thinking toggle only works on DeepSeek models. The Hunyuan models ignore it entirely: hy3-preview stayed at 95% to 96% reasoning either way. The vendor’s recommended model, “tuned for Hunyuan,” lost on every number that mattered. Up to 17% of its responses weren’t valid JSON. Each call took 50 to 60 seconds. It cost three times as much as the winner.

“Tuned for” is marketing. Measure against your actual prompts.

One nuance before you take this too far. The SDK’s built-in extraction prompts were written for no-reasoning models, which is part of why thinking off won the benchmark so cleanly. And yet production runs with thinking enabled. I flipped it back on after testing intention quality, and the System 2 layer came out noticeably better with reasoning available: more specific, more useful intentions, not generic guesses. The prompts assume no reasoning. The benchmark confirms no reasoning extracts better. I still run thinking on, because what the higher layers gain outweighs what extraction loses. The right answer depends on which layer you’re optimizing. Extraction wants thinking off. Deep synthesis wants it on.

What broke

This is the part the marketing pages don’t cover. Every one of these cost me real time in the first two days, and three of them were silent failures where the system kept working, just wrong.

It lied about its mode

The plugin reads HY_MEMORY_MODE. The SDK server reads MEMORY_MODE. If you start the server directly through a systemd unit instead of letting the plugin spawn it, and you only set the first variable, the system runs in pro mode instead of ultra. No graph, no System 2. The health check doesn’t catch this because it reads the plugin’s variable, not the server’s actual state. I ran for hours before catching it. The only reliable check is grepping the sidecar’s startup log for the mode line.

It wrote memories into the wrong namespace

The SDK’s add() method defaults to an agent ID of default_agent. Hermes reads memories under agent_id=hermes. Migrated memories went into a namespace the agent never looked at. Same symptom class as the mode bug: everything works, nothing is found.

Missing secrets made it quietly degrade

One of the bot boxes was missing its memory secrets from the vault. The sidecar started fine and used default models and a default embedding collection instead of the configured ones. The tell was that writes got suspiciously fast. If your writes speed up by an order of magnitude, your embedder is probably wrong, and your vectors are going into a collection nothing will ever search.

A hardcoded token cap silently dropped memories

The reconciler, the component that merges extracted memories, has a hardcoded 2000-token output cap. With a reasoning model, those 2000 tokens can be consumed entirely by thinking, producing zero actual output. The result: extracted memories silently dropped, never written. The log line looks like a successful call with completion_tokens == 2000. I had to patch the constant in the installed package to 16K.

The graph store locks itself to one writer

Kuzu, the embedded graph database, allows a single writer process. If your sidecar is running, a second client trying to write fails with a lock error. This means the System 2 digest, the background process that builds the high layers, can’t run alongside the sidecar. I had to script the sequence: stop the sidecar, run the digest, start it again. Six minutes of downtime at 3 AM is acceptable. During the day it would not be.

System 2 doesn’t schedule itself

The digest is manual-trigger only. The SDK deliberately ships without an in-process scheduler, which is documented in a source file header but nowhere in the marketing material. I had to build my own cron: a watchdog script that stops the sidecar, runs the digest, restarts the sidecar, and stays silent on success. The first real run processed 167 facts into 5 clusters and 19 high-layer schemas in about six minutes. That part of the architecture genuinely works.

The embedding dimension locks in at first boot

The graph store fixes its vector dimension the first time it initializes. If the database was created before you configured your 4096-dimensional embedder, every subsequent graph write fails with a dimension mismatch. The LLM work succeeds, zero schemas land, and the error report says “no L6 basics,” which reads like an empty bank when it’s actually a broken graph. The fix is deleting the graph database and letting it rebuild.

What it costs

The extraction pipeline costs about $0.037 per day on deepseek-v4-pro with thinking off. Embeddings run through qwen3-embedding-8b at 4096 dimensions through my own gateway, which is where the per-request cost tracking lives. Total monthly cost is small enough that I stopped thinking about it. If you’re on the default single-key OpenAI path, it’ll be more, but it’s still a rounding error compared to your agent’s main traffic. That’s the bill worth attacking, and I’ve written about cutting it with caching before.

The verdict

Would I do it again? Yes. Though I’d go in expecting a young project and treating every default as a suggestion.

The architecture is the right one. Memory that evolves as it accumulates. Causal chains that survive semantic drift. Fast inline capture separated from slow background synthesis. Those are the properties a long-lived agent actually needs, and no amount of prompt engineering gets you there.

The execution is rougher than the marketing. PyPI-only distribution means the development process stays opaque: no public repo, no issue tracker, no changelog. It’s a team at Tencent behind it, not one person, but you’re still trusting a process you can’t see into. The silent failure modes are the real risk: wrong mode, wrong namespace, wrong embedder, dropped memories, all of them working fine until you probe. If you adopt it, budget time for verification tooling, not just configuration.

The verification pattern that saved me, in order: check the sidecar startup log for the mode and embedder, confirm the right namespace on every write, run a benchmark against your own prompts before picking models. Then build a probe suite of real questions to test recall after migration. If the answers come back with the causal chains intact, the system is doing what it claims.

My agents remember things now, including how their opinions shifted over time. It’s early days, but the memory layer finally feels like an asset. It used to feel like a liability.

Related: Why You Need a Local LLM Gateway - and How to Set One Up with Bifrost for the gateway this stack runs through. Hy-Memory’s docs: memory.hunyuan.tencent.com and the PyPI package.