15 July 2026

2 clips · 4 minutes of listening · 1 show

3 shows published 3 episodes in the period, 1.5 hours in all. These are the 4 minutes worth hearing.

AI

Latent Space

Dan Biderman, co-founder and chief executive of Engram.

Episode

The AI Memory Problem: Why Long Context Isn’t Enough — Dan Biderman, Engram Co-founder & CEO

The episode states Engram has just closed a $98 million seed round. He describes growing up in Tel Aviv, serving as an officer in Israeli naval special operations, studying cognitive neuroscience in Israel and taking a PhD in computational neuroscience in New York, then working at Mosaic on LoRA before the labs at Stanford, Cornell and Berkeley his co-founders came from. The company's stated bet is compressing corpora into loadable weight states it calls cartridges.

Hosted by

Latent Space cooking show. The episode description names Allen Park as host, and a second voice, addressed as Sean near the end, is also present.

13 July 2026 · 49m 43s · 2 clips below

1 of 2 explainer

Engram's chief executive says a single Wikipedia article read by a Llama 70B model takes up GPU memory of the same order as the model's entire parameter set, and that his company is trying to remove the prefill step rather than optimise it.

Weights at BF16 run to roughly 140 gigabytes and carry a distorted version of the whole internet, while the serving state for a few tens of kilobytes of text reaches around 80 gigabytes on his figures. Biderman's answer is to pay the compute at training time, load the resulting state and start decoding at once, which follows how data centres now split those two stages onto different cards.

1m 55s · from 22:20
My observations
- Check the arithmetic before repeating it. A few tens of kilobytes is roughly ten thousand tokens, and a 70B model using grouped-query attention holds a cache of a couple of gigabytes at that length rather than eighty. The direction of the point survives, the number does not, and the gap matters to anyone sizing memory demand from it.
- The commercially interesting claim is the prefill one. If a large share of inference compute is models rereading the same corpus, a method that moves that work into a pre-trained state shifts demand away from inference memory bandwidth and towards training compute, which is different silicon on a different purchase cycle.
- He has just raised $98 million on this thesis, so the framing is the pitch. The independent evidence is the disaggregation of prefill and decode already showing up in data centre design, which is happening whether or not his particular method works.
2 of 2 framework

Engram's chief executive picks one query to mark where retrieval fails: which M&A deals did we not complete this year, an answer written in none of the files, which he says costs thousands of dollars a run with a frontier model and compaction.

Nowhere does a document state that a deal fell over, so answering means reading every matter in the book and taking the gist, which any employee in the firm could do unaided. He takes pre-training as the existence proof: the industry learns from the web rather than searching it, because learning a corpus creates associations that reading it back at query time does not.

1m 51s · from 24:47
My observations
- Engram's test asks whether answering requires finding a passage or noticing that a passage is missing, and any deployment sitting on a document estate can be put to it. Retrieval handles the first and cannot handle the second at any index size.
- Negative queries are where the governance risk sits, not just the cost. An exception report, a covenant that was never tested, a valuation that was never refreshed: all of them are absences, and a retrieval-based tool will answer confidently and wrongly rather than return nothing.
- Demand evidence for the cost figure. Thousands of dollars per query is a large claim with no methodology behind it in the clip, but it is falsifiable, so a vendor can be asked to run the query and show the bill.

15 July 2026

AI

Engram's chief executive says a single Wikipedia article read by a Llama 70B model takes up GPU memory of the same order as the model's entire parameter set, and that his company is trying to remove the prefill step rather than optimise it.

Engram's chief executive picks one query to mark where retrieval fails: which M&A deals did we not complete this year, an answer written in none of the files, which he says costs thousands of dollars a run with a frontier model and compaction.