How Haru remembers without delaying a reply

I gave Haru , my open-source voice assistant, a long-term memory with one rule: memory never delays a reply. A voice assistant that forgets you between conversations gets old fast, but one that pauses to remember gets old even faster.

Haru is a Raspberry Pi with a microphone and a speaker on my desk, plus a GPU box that does speech-to-text, the LLM conversation, and text-to-speech. Its memories live in a Relevan search index. This post covers how recall fits in a 250 ms budget, how memories get written, and what I learned along the way. The patterns should carry over to any search-backed store.

The budget: 250 ms out of 1.7 seconds

In a voice assistant, latency is the product. From the moment I stop talking to the moment Haru starts, it takes about 1.7 seconds. Roughly 500 ms of that is voice activity detection deciding I’m done, 330 ms is speech-to-text, and 850 ms is waiting for the LLM’s first sentence.

Memory gets 250 ms of that last stretch, at most. Everything else in the design follows from that number:

• Reading memories happens on the hot path (the work between my last word and Haru’s first), so it has a hard deadline and a local fallback.
• Writing memories happens off the hot path entirely, in a separate process, minutes later.
• The two sides only share a file: an append-only chat log.

The read path: recall with a deadline

1. Preload early. A memory session starts the moment the voice activity detector hears speech, before speech-to-text has even finished. In the background, Haru lists every active memory through Relevan’s documents endpoint into a per-session cache. That’s a plain list call, with no search involved. That cache exists only as the fallback for step 3.
2. Search each turn, with a deadline. recall() sends a Relevan search built from my words and waits at most 250 ms. If the preload is still running, it shares that same budget.
3. Fall back locally. If the search is late or fails, recall ranks the session cache by word overlap with what I said, plus a little weight for importance. A late search still completes into the cache, so later turns benefit from it.
4. Merge. Search hits come first, then local matches fill the list up to 10 memories.
5. Inject. The memories go into the system prompt under a short header: “What you remember from earlier conversations (may be out of date; use it only when it’s relevant, and don’t recite it unprompted)”. Each one carries a [noted YYYY-MM-DD] date.
6. Record. The chat log stores which memory ids came up on each turn, along with the search’s queryId and sessionId. The writer needs those later.

The deadline itself is a few lines of asyncio (lightly reformatted from memory.py):

search = self._spawn(self._search(s, text))
waiting = {search}
if s.loading and not s.loading.done():
    waiting.add(s.loading)  # the preload shares the budget
await asyncio.wait(waiting, timeout=self.budget_s)  # 0.25 s

The search is a normal Relevan search with a filter, so retired memories never come back:

{
  "query": "dog name",
  "size": 10,
  "customerId": "household",
  "filters": [{ "field": "status", "value": "active" }]
}

Every response carries a searchContext.queryId. That ID is what lets the writer tie feedback to this exact search later on. The cache is dropped when the conversation resets, after 15 idle minutes.

The write path: a separate process mines the chat log

The memory writer is its own process. It polls the chat log every 20 seconds and picks up a session once it has been quiet for 2 minutes, or once it has piled up 20 unprocessed turns. For each batch it:

1. Gathers the new turns, plus up to 6 earlier turns as context, explicitly marked “already processed, don’t extract from it”.
2. Gathers up to 40 existing memories related to those turns: everything that came up during them first, then by word overlap and importance.
3. Asks the LLM for JSON operations:

   {
     "add": [{ "text": "...", "subject": "...", "kind": "...", "tags": ["..."], "importance": 3 }],
     "update": [{ "id": "mem-...", "text": "..." }],
     "retire": [{ "id": "mem-...", "reason": "..." }],
     "relevant": ["mem-..."]
   }

4. Validates every operation and writes it to the index.
5. Sends feedback for each memory in relevant, and bumps its use_count and last_used_at.
6. Checkpoints how many log entries of that session are done, so a restart never redoes or skips turns. A failed batch isn’t checkpointed, so it gets retried on the next pass.

Step 5 is where the index learns which memories mattered. For every search in those turns that surfaced a relevant memory, the writer sends an llm_judgment event with the original queryId and the memory’s position:

{
  "actionName": "llm_judgment",
  "queryId": "<from searchContext>",
  "sessionId": "<from searchContext>",
  "objectId": "mem-3f2a9c...",
  "position": 2,
  "customerId": "household"
}

The memory document

Each memory is one short, standalone sentence about the user, plus fields for ranking and lifecycle:

{
  "id": "mem-3f2a9c...",
  "text": "The user's wife's birthday is June 3.",
  "subject": "wife birthday",
  "kind": "event",
  "tags": ["wife", "birthday"],
  "importance": 4,
  "person": "household",
  "status": "active",
  "use_count": 2,
  "created_at": "2026-09-30T19:12:04Z",
  "updated_at": "2026-09-30T19:12:04Z",
  "last_used_at": "2026-10-02T08:40:11Z",
  "source": "<conversation session id>"
}

• kind is one of fact, preference, person, event, plan, project, interest, or finding.
• importance runs from 5 for core identity (a name, family) down to 1 for trivia. Dates, plans, and active projects are 4, preferences and interests are 3, and findings and details are 2.
• status is active or retired. Retiring is a soft delete, and every search filters on status: active.
• person is always household for now. Haru can’t tell voices apart yet, so a memory can’t belong to one specific person.
• use_count and last_used_at go up when a memory actually helped. They’re meant to become ranking signals.

What I learned

Search on content words. Spoken input is long and full of filler. “I am feeling peckish, any snack ideas for me?” scored 0 against my index, while “snack” alone scored 3.5. Recall now strips stop words (including “hey”, “haru”, “please”, and “tell”) and searches what’s left, so that question goes out as feeling peckish any snack ideas.

The better fix probably lives on the search side, where the engine would handle conversational queries itself. That’s something we want to explore at Relevan, so for now the Haru fix is deliberately simple.

Give every memory a real subject. The model kept setting subject to “user”, which is true of every memory and useless for search. The prompt now forbids it, and the writer replaces “user”, “household”, or an empty subject with the first few tags. A subject like “dog Biscuit” or “work schedule” is what a later query actually matches.

Write memories as standalone third-person sentences. “The user’s wife’s birthday is June 3.” reads correctly in a prompt months later, with no conversation around it. It also keeps proper names exactly as heard.

Date anything time-sensitive in the text itself. Projects, interests, and findings go stale. The writer gets the current date and writes “As of September 30, 2026, the user is evaluating…”, and the prompt shows each memory’s date. The model can then judge staleness without any extra machinery.

Lookups reveal interests. “Look up what the Google Developer Program includes” really means “the user is evaluating the Google Developer Program”. Remembering the intent behind a question, plus the key finding Haru reported, turned out to be more useful than the facts I stated outright.

Say what not to remember. The microphone is always on. Without a list of what to skip, the index fills with noise. The extraction prompt explicitly excludes:

• small talk and one-off trivia
• passing states, like “I’m hungry”
• Haru’s own suggestions that I didn’t take up
• garbled transcription
• speech that sounds like a TV, or other people who aren’t talking to Haru

Let the writer update and retire memories. Showing the model the related memories along with their IDs lets it update one (“moved to Denver”) or retire one, so contradictions don’t pile up. Retiring is a status change, so nothing is lost.

Close the loop with feedback, judged after the fact. At recall time, nobody knows whether a memory will help. The writer does, because it sees the whole exchange. Its relevant list becomes search feedback tied to the original queryId and position, which gives the index relevance signal from real use.

Treat the store’s latency as variable, by design. Any remote store has slow moments, and the first request after a long idle period can take seconds. With the 250 ms budget and the local fallback, a slow store only costs that turn some recall quality. Haru still answers on time, from the cache, and the preload logs a warning when it fails.

Keep extraction out of process. Extraction is a slow LLM call that can fail in a lot of ways. Because it runs in a separate process that only reads the chat log, a slow model, a bad JSON reply, or a store outage can’t touch a live conversation.

It also makes the chat log the source of truth. The index can be rebuilt by replaying the log, though today the writer only scans the last few days of files, so a full rebuild means widening that window.

Validate everything the model returns. Local models make the occasional malformed reply. The writer strips code fences, finds the outer JSON object, ignores IDs it wasn’t shown, clamps fields like importance to 1–5, and drops any memory longer than 400 characters.

Known limits

• The preload lists the whole index. Per-turn recall is a Relevan search, but the session preload pages through the documents endpoint and takes every active memory (up to 1,000). The writer lists the index the same way for each batch. That’s fine for one household, but it won’t scale. A better version would preload a bounded slice, like the most important memories, and let the writer use search.
• No speaker identity. Every memory belongs to household until Haru can tell voices apart.
• The writer finds related memories by word overlap. Using the search index there too would catch paraphrases, like “spouse” and “wife”.
• use_count and last_used_at aren’t used in ranking yet. They’re written, and that’s it.
• An “expected queries” field is on hold. Each memory could list the questions it should answer, but that’s worth testing only once the chat log has enough real questions.
• Privacy. Every utterance is transcribed and logged, and the log is what the writer mines. Retention limits on the chat log, and a way for the user to review and delete memories, are on the list.

Try it

Haru is open source at dtaivpp/haru . The memory code is in backend/services/gateway/haru_gateway/: memory.py for the read side, memory_writer.py for the write side, chatlog.py for the log between them, and relevan.py for the API client. The longer design notes are in docs/memory-system.md.

Haru uses three parts of Relevan: search with filters, the documents endpoint, and feedback. If you want your own agent to send feedback the same way, the feedback docs cover the event types and the queryId that ties them to a search.

And if you’d like a search index for your own assistant’s memory, try our free tier, or book a call.