Vector databases compress text payloads with generic codecs like LZ4, but not with token-aware schemes. BPE tokenization + entropy coding gives you 5x lossless compression using infrastructure already in every ML stack.
- Published on
Latest
Kumar Shivendu's blog
- Search is everywhere and is one of the hardest problems in CS. Here's why I'm obsessed with it.
- Published on
- Reflections on a year of growth, travel, and pursuing mastery
- Published on
- Published on
- Exploring the JSON embeddings to for matching new products with existing ones
- Published on
Subscribe to the newsletter