Stream tokens, not text

An LLM produces token IDs, one at a time. Servers turn each ID into text, wrap it in JSON, and send it. What if they sent the ID instead and let the client turn it into text? Pick an answer and a connection, then press Play. The same answer streams three ways at once: as text, as token IDs, and as packed IDs. Watch which panel finishes first, and what each one would cost in egress.

The model speed is the token clock: at 1,000 tokens/s a new token exists every 1 ms. A stream cannot finish before the clock does, and it cannot finish before its bytes have fitted through the link. Each panel tells you which of the two held it back. The connection is simulated by the Worker, which delivers each frame only after the frames before it have fitted through a link of that speed. That is a model of a slow connection, not a measurement of a real one. The server replays a saved answer. It is not a live model. Bytes are counted at the application layer, before any HTTP or TLS framing, which adds a few bytes to every flush in every format. The tokenizer is GPT-2's r50k_base (50,257 tokens, every ID fits in 16 bits). The page downloads its 0.5 MB vocabulary once.

What about emoji and other characters a token cuts in half?

A token is a piece of bytes, not a piece of text. Red tokens below are not valid UTF-8 on their own. The first token of 🚀 is f0 9f, and that is half a character. Pick an edge-case answer and press Play to see it. Both sides can handle this. A text server holds bytes back until the character is whole, so its text panel receives fewer frames than there are tokens. A token client feeds every byte, in order, to one streaming UTF-8 decoder and shows the character as soon as its last byte arrives. The page checks each panel against the original text and shows ✓ when it is identical.

This also shows a cost of r50k_base: 猫 is three tokens of one byte each. Newer tokenizers (like o200k) have more whole-character tokens for common scripts. The mid-character problem does not go away with them, only shrinks.

What if the server compresses the text stream?

Measured offline by build_data.py. The server flushes after every token, which is what streaming needs. Sizes in bytes for the whole answer.

gzip and brotli are great when they can see the whole text: brotli at quality 11 squeezes the English answer to 449 bytes. But a stream cannot wait. After every token it must flush, and each flush costs bytes. Packed IDs need no compressor at all, and they were already the smallest streaming format.

What does it cost in egress?

Uses the bytes measured in your last run. Egress is the data your server sends out, billed per GB.

$0.09 per GB is a common cloud list price for outbound traffic. Change it to yours. The saving is about the bytes, so it holds at any price. It does not count replication traffic or server CPU.

How would a client and server handle this?

• Say which tokenizer. IDs mean nothing without the vocabulary. Send the tokenizer name once, here in the X-Tokenizer response header, and let the client cache the vocabulary by that name. Here it is a 0.5 MB file.
• Half characters are free. A token can end in the middle of a character. The emoji 🚀 is several tokens. A text server must hold bytes back until the character is whole (try the Hindi answer: the server skips frames). A token client just feeds the bytes to a streaming UTF-8 decoder.
• Keep a text fallback. A client that does not know the tokenizer asks for mode=text. Same endpoint, same answer.
• Stop sequences and tools. The server can still stop on a string, because it knows the text. Tool-call JSON streams as IDs like anything else, and the client parses it after detokenizing.
• Know the cost. Detokenizing is a table lookup, a few microseconds per token. A lost or repeated packed frame shifts every later ID, so a real protocol needs a length or a sequence number. This demo does not.