You ask your assistant to download a report from a company portal, and it says it can’t reach the site. A month ago the two of you built a helper for exactly that. The helper’s notes talk about “relaying commands to a logged-in browser” and never say “download”, so nothing connects them. Instant Recall is my design for that gap: an index of what the agent already owns, filed by what each thing is for, with a few pointers placed in front of the model before every turn. It’s part of my open Building Jarvis series.
Abstract
Assistants keep plenty and still miss things. What they miss is something the owner owns: a skill, a machine, a decision or a way of solving a problem. It is tied to the request by what it is for, and it often shares no word with it.
A search tool helps only when the agent suspects it owns something. Context pushed into every turn fills up with old documents and breaks the prompt cache.
The paper proposes a design for the finding stage of an agent’s memory. It is an index of what the owner owns, keyed by the situations each thing helps in. Before every turn it pushes a few pointers, placed after the cached part of the prompt, and only pointers the current reader is allowed to see.
This is a design, not a result. Section 9 names the tests that would decide it, run on a replay of the owner’s own corrections against a strong baseline.
Three requests on one morning
The paper opens with three requests. “Download last quarter’s report from the company portal”, and the agent says it can’t reach the portal, although it has the browser helper. “Check the camera on the lab machine”, and it says it has no credentials, although a note about that machine says where the login is kept. “Train it again with the setup we used last time”, and it asks which setup. The answer sits in a session record from three weeks earlier, filed under another project’s name.
Each time I end up saying the same thing: you have this. The paper calls it an ownership miss. The agent owns what the task needs and doesn’t reach for it. Every one costs me attention and teaches me to do the remembering myself.
Normal retrieval returns the stored text closest to the request, so it finds what a request is about. These misses need what a stored thing is for. “Download the report” and “relay commands to a logged-in browser” are linked by a use: when a site blocks a download.
Five places a find can fail
Before a stored item can help, it has to pass five gates. The first gate it fails tells you which fix you need.
- Not indexed. The item lives somewhere the index doesn’t cover, like a skill catalogue or a note about a machine.
- Wrong key. It’s filed under its topic, and the request asks for what it does. “The one we used last time” fails here too.
- Nobody asked. The agent never searched.
- Filtered out. It was found, then removed by a filter, for example a wrong “no longer valid” mark.
- Drowned. It was pushed as the fourteenth of twenty old documents.
How the design works
- Two indexes. A text index covers notes and session records. It mixes word search, meaning search and exact names, with an address map so the agent can “retrieve coarse, address fine”. Next to it sits an asset index: one short entry per thing the owner owns. That covers skills, tools, machines (with where the login is kept, never the login), decisions, past approaches and the latest version of a document.
- Four kinds of key. Names catch “use the portal helper”. Topics catch requests that name the subject. Use keys are short phrases like “when a site blocks a download”. They come from the moment a thing is stored, from the owner’s corrections, and from sessions where an item clearly helped. Co-access links join items that tend to be fetched together, even when they share no words.
- A small push before every turn. Up to five entries that clear a relevance bar, mostly as pointers of about 40 tokens each. If nothing clears the bar, the list is empty, and the paper treats that as a correct answer.
- A guaranteed slot for named things. When the owner names a skill or a machine the agent knows, its entry goes in first, whatever its score. That check lives in code, because a prompt line doesn’t bind reliably.
- Placed where the cache isn’t hurt. The list goes after the cached part of the prompt, next to the owner’s message. A search tool covers everything the push leaves out.
- Only what the reader may see. Every candidate passes one eligibility check, whichever route brought it. That check covers access, quarantine, “please don’t mention this” and whether a fact is still current. A sub-agent with a narrower scope gets nothing it isn’t allowed, not even a count.
- Corrections count at once. Every accepted write goes into a small overlay that is checked on every request. So a correction can be found on the very next message, long before the nightly rebuild, and it survives the swap to the new index.
Why pointers, and why after the cache
The paper works out the price from published cache prices, in multiples of the base input price. One extra search is one extra model call. With a 100,000-token context that call costs about 11,000 base tokens, and only 625 of them are the text it fetched. A search costs a turn, not bytes.
A 40-token pointer costs about 170 base tokens over 30 later calls, and a list of five costs about 850. On those numbers the list pays for itself if it saves one search in more than about one turn in fifteen.
Placement matters more. Put a block that changes every turn inside the cached prefix, a fifth of the way in, and every turn rewrites the rest of the cache: about 92,000 base tokens a turn, the price of eight searches. In the tail, the same block costs only its own tokens.
How we’ll know if it works
Nothing here has been measured yet. The paper fixes the tests before any run. The main one is a replay of the owner’s own “you have this” corrections, each rebuilt as it stood on its date. Seven setups run on the same cases. They start from plain topic search and add one piece at a time, and the full design must beat a strong baseline: every asset’s name and description, retrieved with the same search at the same token and time budget.
Each piece has a bar to clear before it’s adopted. Most comparisons need about 155 paired cases to detect a 10-point gain. Named items and access rules must hold in every case, at least 300 per route. The push has to fit in 50 ms. The owner’s corrections are few and private, so there’s also a synthetic suite anyone can run.
The plan is to build it in TinkerClaw, test it this way, and report the results in a later version.
What changed since the last version
This page used to describe version 7.7 (June 2026), built around a pre-computed concept index with speed and recall figures. Version 8.3 is a full rewrite with a new title. The problem is now the ownership miss, the design is the asset index and the push, and the results are left to the tests above.
Instant Recall is part of Building Jarvis, an open series on persistent agent memory. Follow the work and contribute at github.com/globalcaos/tinkerclaw.
Read the paper
Version 8.7 · 21 October 2026 · 33 pages · the five gates, the two indexes and four keys, the cost model, the eligibility rules and the tests that will decide it
Was this useful?
I’m building these in the open and I’d like your read on them. Has your own assistant ever missed something it already had? What would you want the next paper to dig into? Tell me in the comments below.
Everything on the site, by topic or date: Browse the Tinker Zone