Data compaction
Clewd is a data compaction API for massive sets. It holds each shared pattern a single time and includes it by reference, refactoring continuously as new primitives appear. The way a mind holds a memory.
How it works
A brain does not keep a separate copy of "red" for every red thing it has seen. It holds the primitive once and links to it from every memory. Clewd stores data the same way.
Every object becomes a node that points into a shared graph of primitives. Two objects that look nothing alike often share most of their primitives underneath. The graph is the store; the objects are views onto it.
Compaction grows with the size of the collection, because larger sets repeat more. Clewd is built for data sets where storage and comparison dominate, and where the collection keeps growing.
In three steps
Stream records of any shape into the API.
Each object is reduced to the primitives it shares with everything already stored.
One graph holds it all; every object is reconstructed on read.