Token-Native Storage

Looking for TL;DR? Check key takeaways

Every vector search engine stores two things for each point: the vector and the payload. The AI community has gone all in to compress the vectors with quantization and MRL. However, the text payload gets raw UTF-8, or LZ4 at best, even though the systems that actually consume it (embedders, rerankers, LLMs) operate on tokens, not UTF-8 text. I realised that using OpenAI r50k token IDs over raw text already achieves 2.27× compression, surpassing every popular byte-level compression algorithm, and an entropy-based compressor on top of the token IDs reaches 3.26× (surpassing even a corpus-trained zstd). Effectively, tokenization gives you free compression!

Also, if agents are going to be the biggest users of your system, then your infra should speak their language (tokens), so you're not paying a translation cost on every read/write. This post argues for token-native storage (read/write text as BPE token IDs, not UTF-8 bytes) anywhere models are the primary readers and writers: search/db engines, LLM agent thinking traces / chat history, and maybe even local files. I'll show my point with experiments, talk about limitations, and how the AI/DB community and the AI labs can help get rid of them.

How Token-Native Storage Works

The top path is what databases do today: store UTF-8 bytes, compress with a generic codec. The bottom path is token-native storage: tokenize first, then compress (here entropy-code) the token IDs.

Step 1: Generate Token IDs

BPE (Byte Pair Encoding) tokenization starts with raw bytes and repeatedly merges the most frequent adjacent pair into a new token, building up a vocabulary of common subwords. OpenAI has three popular BPE tokenizers:

1. r50k: used by GPT-2/GPT-3, 50,257 tokens and fits in uint16 (= ~65k)
2. cl100k: used by GPT-3.5/GPT-4, 100,277 tokens and fits in 17 bits (131k) but only 24 bits (3 bytes) is practical due to overhead
3. o200k: used by GPT-4o/GPT-4o mini/o1/o3, 200,019 tokens, same 24-bit (3-byte) practical packing as cl100k.

BPE merges frequent byte sequences into single tokens. "storage" is one token. "Token-native" is three (Token + - + native). On average one BPE token covers 3/4 of a word. An average English word is ~5 characters plus a space, so:

Using UTF-8 text: 6 bytes/word (1 char = 1 byte) × 3/4 word/token = 4.5 bytes/token (raw text)
Using Tokens: 2 bytes/token (r50k uint16 token IDs)
ratio: ~2.25x

This napkin math is what gave me the motivation to run the experiment, and my experiments below prove it in practice.

Another fun fact is that a every word in a sentence comes with a " ", so using uint16 token IDs performs better for any word longer than 1 character. Example: " cat" is 4 raw bytes (space + 3 letters) but just 1 token, 2 bytes (uint16), an easy win :D

Step 2 (Extra): Token ID Compression

Theoretically, step 1 alone already gets you to ~2.25× compression, just raw uint16 packing with no compression algorithm running at all. This step is for squeezing further: gzip and LZ4 see opaque bytes and can't know that [0x01, 0x7F] is token 383. I'm sharing two algorithms for token ID compression but there can be more in future:

1. ANS: An entropy coder working directly on token IDs can give frequent tokens short codes: "the" is 40x more common than "embeddings", so it gets a much shorter code. I used ANS (Asymmetric Numeral Systems) via the constriction library. Here's a great explanation video from 3Blue1Brown (Thanks to my friend Dev for sharing this video)
2. Fix token IDs: ANS gives great compression but comes at some latency cost. While running experiments, I realised a simple but major problem in BPE tokenizer algorithm: BPE assigns token IDs in merge-discovery order, not by frequency. So a common token can have ID 40000 while a rare one gets ID 12. Re-assigning IDs by frequency means the most used tokens get the smallest IDs, so variable-length integer compression algorithms can exploit it better. I used streamvbyte and it almost reaches ANS's compression ratio with ~15-20x faster decode! Also, here's my long term proposal for the AI labs.

import constriction, numpy as np, tiktoken

enc   = tiktoken.get_encoding("r50k_base")
VOCAB = 50_257

# ── one-time setup: train on corpus ──────────────────────────────────────────
# The ANS frequency table is **trained once on a large corpus and then reused everywhere**.
# It is *not* generated separately for each document, which keeps things efficient.
# While generating a custom frequency table per document can compress even further (up to 5x in my tests),
# this approach requiresincluding that table alongside each payload for decoding. This
# would add roughly 900 bytes per 512-token chunk, negating most of the compression gains.
# That's why we re-use it

corpus_ids = enc.encode(corpus_text)
counts = np.bincount(corpus_ids, minlength=VOCAB).astype(np.float64) + 1  # Laplace
probs  = counts / counts.sum()
model  = constriction.stream.model.Categorical(probs, perfect=False)

# ── per-document encode (store this) ─────────────────────────────────────────
def compress(text):
    ids   = enc.encode(text)
    coder = constriction.stream.stack.AnsCoder()
    coder.encode_reverse(np.array(ids, dtype=np.int32), model)
    return len(ids).to_bytes(4, 'big') + coder.get_compressed().tobytes()

# ── per-document decode ───────────────────────────────────────────────────────
def decompress(data):
    n   = int.from_bytes(data[:4], 'big')
    buf = np.frombuffer(data[4:], dtype=np.uint32).copy()
    ids = constriction.stream.stack.AnsCoder(buf).decode(model, n).tolist()
    return enc.decode(ids)

import numpy as np, tiktoken, pyfastpfor

enc   = tiktoken.get_encoding("r50k_base")
VOCAB = 50_257
codec = pyfastpfor.getCodec("streamvbyte")

# ── one-time setup: rank tokens by frequency in the corpus ───────────────────
# BPE hands out IDs in merge-discovery order, not by how often a token is
# actually used, so a common token can sit at ID 40,000 while a rare one sits
# at ID 12. Re-ranking by frequency means the most-used tokens get the
# smallest IDs, which is exactly what a variable-length integer codec like
# streamvbyte rewards.
corpus_ids = np.array(enc.encode(corpus_text), dtype=np.int64)
counts = np.bincount(corpus_ids, minlength=VOCAB)
order  = np.argsort(-counts)                          # token id, most → least frequent
rank_of = np.empty(VOCAB, dtype=np.uint32)
rank_of[order] = np.arange(VOCAB, dtype=np.uint32)    # token id  -> frequency rank
token_of_rank  = order                                # frequency rank -> token id

# ── per-document encode (store this) ─────────────────────────────────────────
def compress(text):
    ids = np.array(enc.encode(text), dtype=np.int64)
    ranks = rank_of[ids]
    out = np.zeros(len(ranks) * 2 + 1024, dtype=np.uint32)
    n = codec.encodeArray(ranks, len(ranks), out, len(out))
    return len(ids).to_bytes(4, 'big') + out[:n].tobytes()

# ── per-document decode ───────────────────────────────────────────────────────
def decompress(data):
    n = int.from_bytes(data[:4], 'big')
    packed = np.frombuffer(data[4:], dtype=np.uint32)
    out = np.zeros(n + 1024, dtype=np.uint32)
    codec.decodeArray(packed, len(packed), out, n)
    return enc.decode(token_of_rank[out[:n]].tolist())

Benchmarks

Storage efficiency

I ran benchmarks on three real, held-out, non-overlapping corpora: English (C4), code (codeparrot-clean, Python), and Hindi (Wikipedia), at 512-token chunks (Standard chunk size). Whenever training was required, it happened only on each domain's train split, and every method below is lossless.

Observations:

1. Raw Token IDs beat the popular byte codecs (LZ4, gzip, zstd) on human languages (English and Hindi) r50k raw hits 2.27x on English (as predicted by napkin math!), o200k reaches 2.55x on Hindi.
   • r50k & cl100k perform poorly on Hindi because they never learnt the Hindi/Devanagari vocabulary. All of them seem to perform poorly on code (maybe due to lots of . in python)
2. tokenizer+ANS achieves the highest compression ratio and surpass almost every algorithm by a wide margin in all 3 datasets! It beats almost everything on code dataset where raw token IDs didn't do well.
3. tokenizer+freq bars sits between raw and ANS for all 3 datasets. It satisfies my goal to buy latency for losing some compression and sorting token IDs by frequency turned out to be a clean way to do it.
4. The fairest byte-codec competitor is zstd --train, since it also learns a vocab from the corpus and achieves similar and sometimes slightly better compression. However, unlike tokenizers, its vocabulary isn't standarized for reuse and encoding latency is much higher. brotli is another interesting algorithm that was trained by Google on a web corpus but is much slower, as we'll see in the next section

Latency

This is the fun part. Now our text compression algorithms will deliver different latency depending on whether the reader/writer is a human or an LLM/agent

Converting to token IDs is a relatively expensive operation. You spend 300-500μs (0.3-0.5ms) on tokenization while LZ4 compression encodes in 12μs. However, when your users are agents, the whole story flips because tokenization is a mandatory step for input/output to/from the model. The difference is only in whether you pay the tokenization/detokenization cost every time or just once:

Observations:

1. When the writer is an LLM, token native storage is 2-20x faster. LLMs produce token IDs by default. So you can just store them raw or optionally apply token ID compression algorithms in 2-18μs (depending on additional token ID compression algo) while converting token IDs to bytes and then storing with even the fastest LZ4 algorithm will take 35μs!
2. When the writer is a human, you can tokenize and store in 300-500μs which is heavy but gives great compression gains. In agentic systems like claude code or even chatgpt, agents do most of the talking/writing while humans give instructions in relatively fewer sentences.
3. brotli, zstd, and zstd --train already have relatively high latency for humans. Now having a detokenize step on top makes even more slower. gzip stays cheap by comparison (56-131μs depending on domain) but its vocab isn't standardized and reusable for agents.

Read is where effect is more pronounced, since it recurs on every read instead of happening once:

Observations:

1. When the readers is agent, tokenization is an expensive but mandatory step. Agents can only understand token IDs so even the fastest decompressing LZ4 algorithm will take 4μs but then the big tokenization step will consume another ~450μs on every single read. But if you store token IDs, it's free or a tiny 1-40μs token ID decompression (10-100x faster!)

2. When reader is human, detokenization slower than LZ4 but comparable to gzip. Detokenization is relatively expensive and takes 25-65μs, but agents read hundreds of chunks before presenting anything to humans. So not having token IDs is a loss in each of those reads. And there will be thousands or millions of read queries on the same row/record from the agent over time. So it's much better to just detokenize once in the end before showing to humans (maybe the frontend should do it)

3. ANS decode (41μs) is ~3-40x slower than reading raw token IDs (2-byte: 1μs, 3-byte: 13μs) which motivated me to propose my design for fixing token IDs which seems to be even faster (10.7μs) than decoding 3-byte token IDs! On English text that costs only 13-20% of ANS's compression ratio, though the gap is much wider on Code/Hindi (up to ~55% for r50k), since +freq's static rank table fits those distributions less tightly than ANS's full probability model does.

Let's do some napkin math to estimate the impact: We start with a ~500μs per read cost but each search query returns 10 chunks for the LLM to read so you lose 5ms of serial CPU time on each such query. Just 11 RPS leads to 30M requests / month each reading 10 chunks = 42 hours wasted / month. Overall, agents already read and write more than humans in agentic systems like claude code or chatgpt. This trend is going to go up.

Also note that every copy of your data enjoy the same 2.27-3.26x reduction, multiplying benefits beyond storage: snapshots, backups, write-ahead log, replicas, and network egress. It means less data moves over the network, more fits in RAM, and reads and indexing get faster.

How can the AI community and the AI labs help?

I have two request from the AI community/labs:

1. Standarization and open sourcing of the tokenizers:

   • Token native storage has great advantages and is production ready today. Most LLMs families except Anthropic Claude and Google Gemini (However Gemma did it) have open-sourced their tokenizers. I request these two major players to do the same.
   • Overall we need a UTF-8 / ASCII like standard for tokenizers. This also avoids any kind of vendor or model version lock in. Agents will soon be the biggest users of web, apis, and all infra. We need to prepare new standards that are optimized for them.
   • Closed source LLM APIs don't accept or return token IDs today. There's also need for standardizing token first input/output LLM APIs, even when your storage tokenizer matches the model's exactly, this only works end to end if you own the inference stack, since hosted chat APIs take text prompts and return text, not raw token arrays. See this follow-up proposing a token-ID-native LLM API.
   • For the organizations training embedding/reranker modelsCohere, BGE, Mixedbread, Jina AI, ZeroEntropy, etc, I request you to please re-use the same tokenizers as LLM BPEs so we can unify search and generation systems to work more efficiently together. (Yes, I know it's not easy to get rid of the WordPiece legacy from base models). This has already started: OpenAI's text-embedding-3 family already uses cl100k directly. Jina's embeddings-v4 moved to a BPE tokenizer too, inheriting from its Qwen2.5-VL backbone.

2. Improving the BPE tokenization algorithm:

   • Request to Anthropic, OpenAI, Google, xAI, Alibaba (Qwen), DeepSeek, and anyone else training the next BPE vocabulary: please re-assign token IDs by frequency rank after building the vocabulary. You can do this on the tokenizer/training corpus or general web. It costs almost nothing but it gives every downstream application some extra compression without any extra effort.
   • Please note that as an user of (or a db/search engine integrating) token native storage, you don't have to wait for this. If you build the remapping on your own corpus, you'll get better token ID compression than a vendor's assigned token IDs anyway :)

Interface

I'm proposing an interface for token-native storage so it's clear what it would look like. First is one-time schema setup: register a field's tokenizer at the collection level.

PUT /collections/my_collection/index
{ "schema": { "text": { "type": "token", "tokenizer": "r50k" } } }

Inserts can take token IDs or plain string (the engine tokenizes it)

PUT /collections/my_collection/points
{
  "points": [{
    "id": 123,
    "vector": [0.12, -0.34, ...],
    "payload": { "text": [1858, 6427, 20272, 318, 257, 6997, ...] } // or a plain string
  }]
}

The engine can decide to store raw token IDs or compress those using ANS or +freq approach I described above based on config. The token ID compression must happen on write and reverse on read, invisibly.

Reads can just specify the field name and if they need field's data as tokens or text.

POST /collections/my_collection/points/search
{ "vector": [...], "limit": 10, "with_payload": { "text": "tokens" } }
// Response:
// {"results": [{"id": 1, "text": [1858, 6427, 20272, 318, 257, 6997, ...]}]}

Default stays "text", so existing clients see no change. An LLM-facing pipeline asks for "tokens" and skips detokenization; anything human-facing asks for "text". This only works end to end if you own the inference stack, since hosted LLM APIs take text prompts, not token arrays (yet).

Limitations

• No shared vocabulary across models yet. Embedding/reranker models use WordPiece or SentencePiece, LLMs use BPE variants (r50k, cl100k, o200k), and different LLM families/providers don't share the vocab either. A token-native payload only pays off end to end for a consumer using the same tokenizer it was stored with, see how the ecosystem could fix this.
• Raw token ID compression efficiency can suffer on unknown languages. For a language/script a tokenizer doesn't have a vocab for, token IDs can lose against raw bytes. r50k never learned to merge Devanagari (Hindi) bytes, so the Hindi word भारत (भ,ा,र,त, each a 3-byte UTF-8, 12 bytes total) falls back to 7 token IDs. Storing them as uint16 makes it slightly worse (14 bytes) so you need token ID compression algorithms to recover the ratio. Raw token IDs are still useful for latency either way, if an LLM is going to read/write them.
• Frequency remapping and ANS are corpus-specific, not free. The remapping itself is cheap but not free to compute, and if there's a big mismatch between the frequency table and your actual corpus, the compression ratio suffers. The core argument, storing raw token IDs instead of UTF-8 bytes, holds regardless of whatever compression you apply on top. See how labs could fix the remap for free, though you don't have to wait for that either.

Key Takeaways

• Compression: Token IDs surpass every popular byte compression algorithm before you even compress them: raw r50k/o200k hit 1.5-2.5x, entropy-coding on top (ANS) gets to 3.26x! And they generalize and perform well across every tokenizer and dataset I tested.
• Latency: When the writer or reader is an LLM, tokenization/detokenization is a mandatory steps. Storing token IDs instead of UTF-8 bytes turns a mandatory ~300-500μs cost into a 2-230x faster read/write, since agents produce and consume tokens natively.
• Token-native storage is production-ready to be adopted in DB/Search engines. But the ecosystem needs to evolve and accept it across the stack for amplifying the impact: models don't share a vocabulary yet, and hosted LLM APIs still only take text input/output for now.

Acknowledgements

The benchmarks use tiktoken, zstandard, lz4, constriction, streamvbyte. Thanks to the concept of napkin math for making me realise this gap.

All benchmark scripts behind this post are on GitHub.

Citation

If you find this useful, please cite:

@misc{kumar2026tokenstorage,
  author       = {Kumar Shivendu},
  title        = {Token-Native Storage: Read and Write in Your Agent's Language},
  year         = {2026},
  url          = {https://www.kshivendu.dev/blog/token-storage},
  note         = {Blog post}
}