FAQs related to GSoC

OpenAI's and Anthropic's chat APIs take text in and give text out, even though the model underneath samples token IDs at every step. There is precedent for token-ID input (OpenAI's legacy Completions prompt field, vLLM's prompt_token_ids), but no major hosted chat API exposes it end to end. This post analyzes the current input/output shapes and proposes a concrete, opt-in token-ID mode using the same string-or-array interface pattern from the token-native storage post.

Token-Native LLM APIs: Input and Output as Token IDs

Cross-tokenizer translation only comes up on a rare path (fallback provider, degraded model, version migration), and decode-then-re-encode already handles it: 100% lossless at 121.6µs median round trip, negligible next to network or inference latency. I also tried to make it faster with a static lookup table and a splice-based translator. It failed on both correctness (16.2% exact match) and speed (0.97x). The naive path remains the right answer, and the deeper case for a shared vocabulary standard is about not having to maintain a tokenizer-coupled storage layer at all.

Translating Between Tokenizers: The Naive Way Is Already Fine

I tried to build fake embeddings that match real ones. Matching all five popular structure metrics was easy — and the result was still a useless fog. The thing that actually worked was copying the geometry locally, which also tells you what a transformer's 24 layers are each doing.

You can fake every statistic of an embedding and still fail

Uniform 4-bit quantization costs an embedding model real retrieval quality. Quantizing only the layers that are not doing real work costs nothing — and a layer-by-layer look at the model tells you which layers those are. Validated on mxbai-embed-large + BEIR scifact.

Dynamic Quantization for Embedding Models: Lossless Retrieval by Crushing Only the Lazy Layers

High-dimensional random vectors are nearly equidistant — the 'nearest neighbor' is barely nearer than nothing, and a search index has nothing to navigate by. I unpack what that means, what 'structure' is the cure for, and what structure actually buys: mostly compute, not recall.

Understanding Random Embeddings

A friend said they expand SPLADE terms into an OpenSearch BM25 text field and it works. We tested it — 0.36 NDCG@10, worse than plain BM25. Here is why, and how to mostly fix it.

SPLADE Into a BM25 Field: The Tokenizer Trap

Inference-free SPLADE nearly matches full SPLADE quality at 13× lower query latency, no GPU needed at query time. Benchmarked against full SPLADE and BM25 on Qdrant and pyserini.

Inference-Free SPLADE: Full Quality, 13× Faster Queries

Vector databases store UTF-8 text; models read tokens. Persist payloads as BPE token IDs with a static entropy coder and you get ~3× lossless compression, zero-cost persistence of LLM output, and a representation agents already speak.

Token-Native Storage: Read and Write in the Language Models Already Speak

Search is everywhere and is one of the hardest problems in CS. Here's why I'm obsessed with it.

Why I'm Obsessed With Search

Reflections on a year of growth, travel, and pursuing mastery

2025: Year in Review

I Reverse-Engineered Exa.ai Infrastructure Cost with Napkin Math

Exploring the JSON embeddings to for matching new products with existing ones

All Posts

FAQs related to GSoC