Semantic Deduplication in Agent Long-Term Memory

Vector databases are good at one job: finding text that resembles other text. Traditional retrieval pipelines treat memory as a document store, embedding every interaction and appending it to the index, which works fine when the goal is pulling up a static document but starts to break once an agent accumulates a long running history with a user. Nothing in that setup decides what should be kept, what should be merged, or what should be thrown out.

As an agent keeps talking to the same person over weeks or months, its vector store fills up with duplicate observations, scattered fragments of the same preference stated three different ways, and facts that flatly contradict each other, and every single one of them stays fully retrievable. The agent ends up retrieving whichever one happens to score highest on similarity for a given query, and often that's the stale one.

This produces what's best described as retrieval drift: a memory corpus that grows more cluttered and more internally inconsistent with each new interaction, nudging the agent toward answers that were true once but no longer are, even while the correct, newer fact sits a few rows away in the same database. The Always-On Agents survey names this gap directly. Its review of the field finds that research has focused heavily on how to accumulate and retrieve state, with much less attention paid to governing it, recovering from errors in it, or giving it up, and it points out that "forget," as implemented across most systems, means an edit applied at retrieval time rather than a record actually being erased.

None of this is a knock on vector databases as infrastructure. Storage and retrieval are not memory management. An agent that's supposed to track a person's changing life needs something sitting above the vector store, making decisions about what gets written, what gets replaced, and what gets quietly retired.

What semantic deduplication does inside a memory pipeline

Semantic deduplication makes those decisions. It isn't a cleanup job that runs occasionally to tidy up a messy index. It operates at write time, deciding as each new observation arrives whether it becomes a new durable memory, gets folded into something that already exists, or gets suppressed because it adds nothing new.

The mechanism breaks down into three steps. Second, each candidate fact gets compared, semantically, against memories already in storage. Third, based on how that comparison comes out, the system writes a new record, merges it into an existing one, updates an existing one, or soft-expires something that's now redundant.

The thresholding is where the real engineering lives. The LycheeMemory codebase specifies this precisely: record IDs computed as SHA256(normalized_text) make identical content idempotent at write time. New candidates get compared through approximate nearest-neighbor search against existing records of the same memory type, and anything that clears a cosine similarity of 0.85 gets marked for soft-expiry as a near-duplicate, with any composite record built from the affected sources getting invalidated too.

Soft-expiry, rather than deletion, is the deliberate choice here. A soft-expired record drops out of eligibility for retrieval, so it stops competing with newer, more accurate information, but the record itself stays in the database. That matters because the history of what the agent once believed, and when it stopped believing it, has value, especially for debugging and for the governance questions covered later in this piece.

Mem0, described in Chhikara et al.'s paper (arXiv:2504.19413), shows what this looks like in a working production system. It runs a two-phase pipeline, extraction followed by update, sitting on top of the vector store, and it scopes memories across four dimensions: user_id, agent_id, run_id, and app_id, controlling exactly which memories are visible in which context. Mem0 also separates memory into distinct types: long-term, short-term, semantic, episodic, and procedural.

Timing the deduplication step at scale

Deduplication logic needs somewhere to run, and the naive answer, run it after every single turn of conversation, turns out to be the wrong answer at scale. Calling an LLM to extract, compare, and consolidate memories after every exchange means the cost of maintaining memory grows without bound as a conversation lengthens, which is exactly the wrong direction for a system meant to run continuously in production.

The mechanics of why are straightforward. Each turn fires off a full LLM call that has to reckon with an ever-growing memory store, so the cost per session compounds rather than leveling off the way a well-designed system should.

LycheeMemory V2, described by Dongfang Li and co-authors in a paper from August 2026 (arXiv:2608.12990), diagnoses this cost problem directly and proposes a fix: move from turn-level consolidation to segment-level consolidation. Each finished segment then gets encoded exactly once into a typed, self-contained record, with a bounded amount of reference context carried across segments. The paper reports that this approach reduces the token cost of memory construction. The paper's headline accuracy numbers rest on a single LLM judge without independent corroboration, and the LoCoMo evaluation set used was filtered before scoring, so the comparison isn't a fully controlled one and the reported figure is directionally useful rather than settled.

The accuracy case for segment-level consolidation stands on its own regardless of the benchmark caveats. Grouping exchanges by a detected semantic boundary keeps a coherent event, or a coherent stretch of time, together inside one record instead of splitting it across a fixed-size batch that was never designed to respect where one topic ends and another begins. Segment-level consolidation is built specifically to notice that boundary, which makes it cheaper to run and more faithful to the shape of what actually happened in the conversation.

How atomic memory representation changes what deduplication can detect

The grain at which a memory is stored decides what deduplication logic is able to see. Fine-grained atomic facts, rather than summaries or whole dialogue chunks, give deduplication something precise to compare. Only at that atomic level can a system tell whether two observations say the same thing, contradict each other, or whether one simply contains the other, and only at that level can it act on that judgment automatically rather than guessing.

ROAM, from Zheng and colleagues at Southern University of Science and Technology and collaborating institutions (arXiv:2609.09778, September 2026), builds its whole architecture around that idea. Every incoming atom gets classified against the atoms already stored, using five relations: independent, equivalent, one subsuming the other in either direction (old subsuming new, or new subsuming old), or conflicting. A fixed policy then takes that classification and assigns the memory a structural role, rather than asking a single LLM call to make a compound, murky add-update-delete decision all at once.

Those roles are Primary and Evidence. Evidence holds facts that have been superseded, or that support a Primary fact without competing with it, kept around for provenance but excluded from the retrieval pool. An outdated atom can't crowd out a current one in a search result, because it was never eligible for that search result in the first place, which keeps retrieval results from being polluted by superseded facts.

Compatible details sitting across several atoms, along with changes that happened over time, get synthesized into a compact view attached to the Primary entry for that fact, so what gets retrieved can carry more texture than any single atom would on its own, without losing the precision that comparing atoms one at a time makes possible.

This advantage is visible when comparing against systems that store coarser summaries instead of atoms. But they directly contradict each other as facts, and a system comparing whole chunks has no clean way to catch that. Atomic comparison catches it, because "prefers morning meetings" and "moved all meetings to afternoon" are compared as discrete claims, not folded into surrounding sentences that mask the contradiction.

Conflict is the hardest case any of this has to handle, and ROAM's answer to it is to keep the conflicting record rather than erase it, just stripped of Primary status. Reports describe ROAM's approach as improving answer accuracy substantially across models and evaluation settings compared to coarser-grained alternatives, a figure worth checking against the paper directly rather than treating as fixed.

Whether a knowledge graph is necessary, or whether typed semantic retrieval is enough

Memanto, from Abtahi and co-authors (arXiv:2604.22085, 2026), runs on Moorcheh's Information Theoretic Search engine, a semantic database built without an indexing step, and reports state-of-the-art results among vector-only systems on both LongMemEval and LoCoMo, while needing only one retrieval query per answer, adding no cost at ingestion, and skipping graph schema maintenance altogether. Memanto's own term for the compute, latency, and complexity that memory systems usually pile on is the "Memory Tax," and the claim here is that this typed approach removes that tax by design rather than by tuning it down after the fact.

Mem0 answers the same underlying problem with an actual graph: Mem0g, added in 2025, stores entities as nodes and the relationships between them as edges, giving the system a way to reason about connections a flat comparison would miss.

The fair objection to typed vector-only systems is that they move the hard work of conflict resolution into schema design, up front. Graph-based systems handle relationships between entities that weren't anticipated when the schema was built more gracefully, because the graph can grow new connections without a redesign.

None of this settles into a clean winner. For a team shipping an agent at scale, the cost of running LLM-mediated entity extraction on every single ingestion call, and the opacity of debugging failures in a graph layer, weighs as heavily as any accuracy number on a benchmark table. The choice comes down to what a given team can actually afford to run and maintain per user, not which architecture scores a point higher on a leaderboard.

Semantic similarity retrieval as the wrong organizing principle

Semantic deduplication and similarity-based retrieval solve a specific problem well: keeping a person's facts, preferences, and history coherent across a long relationship with an agent. That problem is not the same as tracking where an agent stands in the middle of a long, multi-step task it's actively executing.

For an agent working through a long-horizon job, booking a multi-leg trip, debugging a deployment, completing a research task with several dependent steps, organizing memory by semantic similarity scatters the execution state the agent needs to track across entries that happen to resemble each other, rather than entries that are causally linked. Deduplication can't fix this: the problem isn't redundancy, but that similarity was never the right axis to organize this kind of information along.

MAGE, described by Chen and colleagues (arXiv:2606.06090), answers this with a hierarchical state tree instead of a flat memory store. When the agent needs to know where it currently stands, it reads the path from the root of the tree down to its current node, rather than assembling an answer out of whatever entries happen to score highest on semantic similarity.

This is a boundary on what deduplication is responsible for, not a contradiction of the case for deduplication. Deduplication keeps the memory substrate, the preferences, facts, and history an agent accumulates about a user, from degrading into noise over time. Execution-state management keeps the agent from losing track of where it is inside a task it's currently running. Production agents built to run long, autonomous, multi-step workflows need both layers; neither one substitutes for the other. The deduplication layer described through the rest of this piece remains the foundation either way, whatever execution-tracking sits above it for a given workload.

Deduplication, compliance, deletion rights, and data governance

Deduplication decisions and data governance obligations are tied together directly. Every time memories get merged, consolidated, or soft-expired, the system needs to keep track of which user each underlying observation came from, because without that trace, a deletion request from one user can't be honored without wiping out memories that belong to someone else.

A vector store with no per-user tagging, no deletion pathway, and no audit trail has no way to meet that requirement, and that's a structural disqualification, not a hypothetical risk, for any agent handling personal data for users in either jurisdiction.

The four visibility scopes built into Mem0's architecture, user_id, agent_id, run_id, and app_id, aren't a convenience feature layered on for flexibility. A memory system without that kind of scoping has no reliable way to erase one user's data without risking damage to records that were merged with, or shared by, someone else.

The hardest governance case comes up when a consolidated memory record was built from observations contributed by more than one user. A system that doesn't preserve where each piece of a merged record came from has no way to make that choice carefully, and defaults to deleting the entire record, which is a blunt outcome but the only safe one available without provenance tracking.

What follows from all of this is a concrete deployment checklist: encryption at rest and in transit, per-user access scoping enforced at write time rather than bolted on as a retrieval filter afterward, automated redaction of personal information before anything gets embedded, retention policies a team can configure, and deletion that actually propagates through every tier of the system, including vector index entries and backup snapshots. Semantic deduplication built without this checklist in mind is a system waiting to fail its first serious deletion request.