Token-Native LLM APIs

Looking for TL;DR? Check key takeaways

The token-native storage post argued for storing text as BPE token IDs instead of UTF-8 bytes: embedders, rerankers, and LLMs all consume tokens, not bytes. But I flagged one limit there, reading a payload back as token IDs only pays off end to end "if you own the inference stack, since hosted LLM APIs take text prompts, not raw token arrays (yet)." This post is about that "yet": what would it take for a hosted LLM API to accept token IDs as input and return them as output, instead of forcing a detokenize-then-retokenize round trip on every call?

What hosted APIs do today

Every hosted chat API is text in, text out, even though the model samples token IDs at every step. The token-ID plumbing already exists one layer down in the serving stacks, it just isn't exposed:

There's one real precedent, and it's easy to miss. OpenAI's legacy Completions prompt field has always accepted "string, array of string, array of number, or array of array of number", and that "array of number" is literally a list of token IDs from the model's own tokenizer. That's documented, shipped behavior, not something I'm proposing. The catch is that it only works with legacy completion models (gpt-3.5-turbo-instruct and older), and OpenAI froze the endpoint back in mid-2023. So the precedent is sitting right there in OpenAI's own API, just stuck on models nobody sends new traffic to.

The pattern also already runs in production, just below the hosted layer. vLLM's LLM class takes prompt_token_ids directly, and with skip_tokenizer_init set it expects token IDs in and hands token IDs back, no text touches the request or response at all. TGI's details mode returns a per-token integer id on output (its input routes still take text). So what I'm proposing below is really a hosted wrapper around something the serving stacks already do.

The closest thing to token-level output is logprobs/top_logprobs. It's easy to read it as token IDs, so here's what it actually contains:

{
  "token": " model",
  "bytes": [32, 109, 111, 100, 101, 108],
  "logprob": -0.234,
  "top_logprobs": [
    { "token": " model", "bytes": [32, 109, 111, 100, 101, 108], "logprob": -0.234 },
    { "token": " system", "bytes": [32, 115, 121, 115, 116, 101, 109], "logprob": -2.891 }
  ]
}

That gives me the decoded token string, its raw UTF-8 bytes, and a log probability, but never the integer vocabulary ID. I can rebuild the text (the API already does that) and rank candidates by probability, but I can't recover which row of the embedding table " model" actually was without re-tokenizing the string myself and hoping my tokenizer matches the server's exactly. Anthropic's Messages API doesn't expose logprobs at all, there's only a community feature request for it so far.

A concrete interface proposal

I solved almost this exact interface problem once already in the storage post, for the read and write paths of a vector DB payload. The same shape carries over with a swap of nouns: prompt for payload, model tokenizer for collection tokenizer.

Input. A chat message's content already accepts a string (text) or an array of content parts (for multimodal messages). The simplest token-ID form reuses the storage post's exact string-or-array trick, dispatched on shape: a string is text, a bare array of integers is token IDs. No new block type needed, and it mirrors OpenAI's legacy Completions prompt, which already unions string with array-of-number.

// Current: content is text
POST /v1/chat/completions
{
  "model": "gpt-5.1",
  "messages": [
    { "role": "user", "content": "Summarize the attached document." }
  ]
}

// Proposed: content is a bare array of token IDs, told apart from text by shape
POST /v1/chat/completions
{
  "model": "gpt-5.1",
  "tokenizer_version": "gpt-5.1-2026-07",
  "messages": [
    { "role": "user", "content": [3838, 42891, 262, 5301, 6190, 13] }
  ]
}

The one case a bare array can't cover is interleaving token IDs with other modalities in the same message (token IDs, then an image). That's exactly what the content-parts array is for, so wrap them in a part alongside the others only when you need that:

"content": [
  { "type": "input_tokens", "token_ids": [3838, 42891, 262, 5301, 6190, 13] },
  { "type": "image_url", "image_url": { "url": "..." } }
]

Output. Mirror the storage post's read side, where with_payload grew a per-field format string ("text" vs "tokens") instead of staying a boolean:

POST /v1/chat/completions
{
  "model": "gpt-5.1",
  "tokenizer_version": "gpt-5.1-2026-07",
  "response_format_tokens": true,
  "messages": [ ... ]
}

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The document proposes...",
        "token_ids": [464, 6190, 26057, 986]
      }
    }
  ]
}

Returning both content and token_ids costs nothing extra, the server has the ID sequence before it decodes to text, decode is the cheap direction. A caller that wants tokens only, to skip shipping text at all, asks for token_ids and omits content, exactly like the storage read path asking for "tokens" only.

Streaming carries the same field per chunk instead of only delta.content:

{ "choices": [{ "delta": { "content": " model", "token_ids": [2746] } }] }

The one field storage never needed is tokenizer_version, because a model's tokenizer changes across versions in a way a database collection's never does. That's not a small detail, and it's most of why this has to be opt-in.

Why it has to be opt-in and versioned

Token IDs only mean something against the exact vocabulary that produced them. Pin the wrong tokenizer version and they decode to garbage. Text doesn't have this problem: a prompt string is portable across every model forever, because text has no version to get wrong.

They're also stuck to one model family. r50k and cl100k IDs mean nothing to a Claude or Llama tokenizer, so the moment you send token IDs you've committed to one provider's one tokenizer version for that request.

So you end up needing a text fallback anyway. In the tokenizer-translation post I found that decoding then re-encoding across tokenizers is lossless and cheap (121.6µs median), but only because it goes through text as the common form. A token-ID API can't take that shortcut when it has to fall back to another provider or model, it still has to decode to text on one side and re-encode on the other.

The payoff

If you're already storing tokens natively, this drops the last detokenize/retokenize pair in the pipeline. Even with token-native storage, every LLM call today forces two conversions that don't change the data at all: the retrieved payload gets turned back into text to build the prompt, and the model's output gets turned back into text before it reaches you, and then you retokenize it to store it natively again.

In the storage post I measured tokenization at ~450µs per read of a 512-token chunk. A token-ID request skips it when you build the prompt and skips it again when you store the output: pull token IDs from storage, send them as the prompt, get token IDs back, store them. Text only shows up at the edges, where a human actually has to read something.

Limitations

• This is a proposal, nothing here ships today. No major hosted chat API accepts or returns token IDs, so the whole interface above is my design. The real precedents I found (OpenAI's legacy Completions prompt, vLLM's prompt_token_ids/skip_tokenizer_init, TGI's per-token id) are either frozen to legacy models or live in the serving stack, never behind a hosted API.
• I couldn't pin down Anthropic's logprobs situation from public sources beyond a community request. If they ship it, the table up top might already be stale by the time you read this.
• Providers might have good reasons not to do this at all. Text is where content-safety filtering, prompt-injection detection, and moderation run today, so a raw token-ID path is a second door into the model that skips whatever of that is string-matching. Token IDs are also an easy way to smuggle things past a filter: you can build a sequence that decodes to something the filter would catch without ever writing that string out. Providers would also have to keep old tokenizer versions around forever, or every token-ID request breaks the day they retrain. And I don't know how token-ID input plays with prefix-based prompt caching. I don't have answers to these, they just seem worth testing.

Key Takeaways

• Hosted chat APIs are text in, text out today, both OpenAI (Chat/Responses) and Anthropic (Messages), even though the model samples token IDs at every step.
• The precedent already exists, just not behind a hosted chat API: OpenAI's legacy Completions prompt accepts token-ID arrays (frozen to legacy models since 2023), and vLLM does token IDs in and out in production.
• My proposed interface reuses the storage post's shapes: a bare array for token-ID input, a format flag for token-ID output, plus a tokenizer_version pin, and it can't be portable across model families the way text is.
• The payoff is for systems already storing tokens: retrieve, prompt, receive, and store as token IDs, with text materializing only at the human-facing edges.

Citation

If you find this useful, please cite:

@misc{kumar2026tokenllmapi,
  author       = {Kumar Shivendu},
  title        = {Token-Native LLM APIs: Input and Output as Token IDs},
  year         = {2026},
  url          = {https://www.kshivendu.dev/blog/token-api},
  note         = {Blog post}
}