# lightembed `.bin` Wire Format — v1

> The single artifact a customer buys. Contains **everything language-specific**:
> tokenizer vocabulary, embedding table, transformer weights, output projection.
> The engine (`engine.wasm`) is identical for every customer and every language —
> all variation lives here.
>
> **Normative reference:** `reference/forward.py`. Where this document and the
> reference implementation disagree, the reference wins and this document is a bug.

---

## 1. Design constraints

These drove every decision below. Read them before proposing a change.

| Constraint | Consequence |
|---|---|
| One engine, many models | Architecture is read from the header, never compiled in |
| Model ships separately from engine | Tokenizer must live **here**, not in the WASM |
| Static and transformer models share a path | `n_layers = 0` is a first-class configuration, not a special case |
| Matryoshka is mandatory | Output projection is required in every configuration — it is where the nested ordering lives |
| Per-customer fingerprinting | A rotation is folded into the projection at pack time; zero runtime cost, zero runtime check |
| Silent corruption is unacceptable | Trailing SHA-256 over the whole file; version mismatches are hard errors |
| Format must evolve without breaking customers | Section table, not derived offsets — unknown sections can be skipped |

## 2. File layout

```
┌────────────────────────────────────┐
│ Header                    64 bytes │
├────────────────────────────────────┤
│ Section table    2 + 20×n_sections │
├────────────────────────────────────┤
│ Sections (in table order)          │
│   … tokenizer vocab                │
│   … tokenizer config               │
│   … embedding weights              │
│   … embedding scales               │
│   … layers                         │
│   … final layer norm               │
│   … projection                     │
│   … metadata (optional)            │
├────────────────────────────────────┤
│ SHA-256 of all preceding  32 bytes │
└────────────────────────────────────┘
```

All integers are **little-endian**. All floats are IEEE-754 **fp32 little-endian**.
Sections are 8-byte aligned; padding bytes are zero and not covered by section `length`.

## 3. Header (64 bytes, offset 0)

| Off | Size | Field | Notes |
|---:|---:|---|---|
| 0 | 4 | `magic` | `"LEMB"` = `0x4C 0x45 0x4D 0x42` |
| 4 | 2 | `format_version` | u16. This document describes **1** |
| 6 | 1 | `embedding_quant` | 0=fp32, 1=int8, 2=int4, 3=ternary |
| 7 | 1 | `weight_quant` | 0=fp32, 1=ternary. Ignored when `n_layers = 0` |
| 8 | 4 | `vocab_size` | u32 |
| 12 | 2 | `d_model` | u16 |
| 14 | 1 | `n_layers` | u8. **0 = static model** |
| 15 | 1 | `n_heads` | u8. Must divide `d_model`. Ignored when `n_layers = 0` |
| 16 | 2 | `ffn_dim` | u16. Ignored when `n_layers = 0` |
| 18 | 2 | `output_dim` | u16. Length of the vector `embed()` returns |
| 20 | 2 | `max_seq_len` | u16. Tokenizer truncates here |
| 22 | 1 | `n_matryoshka` | u8, 0–8. 0 means only `output_dim` is valid |
| 23 | 1 | `flags` | see below |
| 24 | 16 | `matryoshka_dims` | 8 × u16, ascending, unused entries 0 |
| 40 | 4 | `license_id` | u32. 0 = unlicensed / development build |
| 44 | 1 | `version_major` | u8 |
| 45 | 1 | `version_minor` | u8 |
| 46 | 2 | `version_patch` | u16 |
| 48 | 16 | `model_id` | UTF-8, NUL-padded. e.g. `"lightembed-tr"` |

### `flags` bits

| Bit | Meaning |
|---:|---|
| 0 | `has_fingerprint` — a customer rotation is folded into the projection |
| 1 | `do_lower_case` — tokenizer lowercases before matching |
| 2 | `strip_accents` — tokenizer strips combining marks |
| 3–7 | reserved, must be 0 |

> ⚠️ **Turkish caveat.** `do_lower_case` must be **0** for `lightembed-tr`. Turkish
> case mapping is locale-dependent (`İ`→`i`, `I`→`ı`); a locale-naive lowercase in
> Rust or JS produces `i̇` (i + combining dot) and silently diverges from the
> training-time tokenization. BERTurk is a cased model, which sidesteps this
> entirely. Any future language that needs lowercasing must specify its case
> mapping explicitly rather than relying on the host's default.

### Validation the engine MUST perform on load

1. `magic == "LEMB"` — else reject
2. `format_version` major compatible — else reject with the expected range
3. `d_model % n_heads == 0` when `n_layers > 0`
4. `matryoshka_dims` strictly ascending, each `<= output_dim`
5. Trailing SHA-256 matches — else reject
6. Every section in the table lies within the file and does not overlap the header or hash

A rejected model is a hard error with a human-readable message. **The engine never
falls back to a partial load** — a model that loads is a model that is correct.

## 4. Section table (offset 64)

```
u16  n_sections
repeat n_sections:
    u16  section_type
    u16  reserved            (0)
    u64  offset              (from file start)
    u64  length              (bytes, excluding alignment padding)
```

| Type | Name | Required |
|---:|---|---|
| 1 | `TOKENIZER_VOCAB` | yes |
| 2 | `TOKENIZER_CONFIG` | yes |
| 3 | `EMBEDDING_WEIGHTS` | yes |
| 4 | `EMBEDDING_SCALES` | when `embedding_quant != fp32` |
| 5 | `LAYERS` | when `n_layers > 0` |
| 6 | `FINAL_LN` | yes |
| 7 | `PROJECTION` | yes |
| 8 | `METADATA` | no |

Sections may appear in any order. **Unknown section types MUST be skipped, not
rejected** — this is what lets v1 engines load v1.x models that carry new optional
data.

## 5. Sections

### 5.1 `TOKENIZER_VOCAB` (type 1)

WordPiece vocabulary in token-id order. Each entry is a length-prefixed UTF-8 string:

```
repeat vocab_size:
    u8   len
    u8[] bytes        (UTF-8, len bytes, no NUL terminator)
```

Token id is the entry's ordinal position, starting at 0. The engine builds a
string→id hash map at load time.

No compression here: HTTP gzip handles it (~2.5× on this data), and an in-format
compression scheme would have to be reimplemented in Rust, Node, and Python.
Front-coding the shared `##` prefixes would save ~40% and is a candidate for
`format_version` 2 if it ever matters.

**Size reference** — BERTurk 32k: ~290 KB raw, ~120 KB gzipped.

### 5.2 `TOKENIZER_CONFIG` (type 2)

```
u32  unk_token_id
u32  cls_token_id
u32  sep_token_id
u32  pad_token_id
u32  mask_token_id
u16  max_input_chars_per_word      (BERT default 100)
u8   prefix_len                    (length of continuing-subword prefix)
u8[] prefix_bytes                  (UTF-8, typically "##")
```

Tokenization is standard BERT WordPiece: whitespace + punctuation pre-tokenization,
then greedy longest-match-first subword segmentation per word. A word longer than
`max_input_chars_per_word`, or one with no valid segmentation, emits `unk_token_id`.
`[CLS]` and `[SEP]` wrap the sequence; the result is truncated to `max_seq_len`.

### 5.3 `EMBEDDING_WEIGHTS` (type 3)

Row-major, `vocab_size` rows of `d_model` values. Encoding depends on `embedding_quant`:

| `embedding_quant` | Bytes per row | Encoding |
|---|---:|---|
| 0 `fp32` | `d_model × 4` | fp32 |
| 1 `int8` | `d_model` | signed int8 |
| 2 `int4` | `d_model / 2` | two per byte: **low nibble = element 2k**, high nibble = 2k+1; signed, range `[-7, +7]` |
| 3 `ternary` | `d_model / 4` | four per byte, 2 bits each, **low bits = low index**; `00`=0, `01`=+1, `10`=−1, `11` reserved |

`d_model` MUST be a multiple of 4 so that every encoding divides evenly.

Dequantization is `value = code × scale[row]` for all quantized formats. The int4
`-8` code is never emitted by the packer (symmetric range) but a reader MUST
sign-extend correctly anyway.

**Why per-row scales, never a global scale:** token embedding magnitudes vary by
orders of magnitude between frequent and rare tokens. A single scale quantizes rare
tokens to zero and collapses them onto one another — exactly the failure mode this
project exists to fix.

### 5.4 `EMBEDDING_SCALES` (type 4)

`vocab_size` × fp32, one per row. Absent when `embedding_quant == fp32`.

### 5.5 `LAYERS` (type 5)

`n_layers` blocks, concatenated. Each block, in order:

```
ln1.weight                    d_model × fp32
ln1.bias                      d_model × fp32
attn.q.weight                 quantized, d_model × d_model      (no bias)
attn.q.scale                  fp32
attn.k.weight                 quantized, d_model × d_model      (no bias)
attn.k.scale                  fp32
attn.v.weight                 quantized, d_model × d_model      (no bias)
attn.v.scale                  fp32
attn.out.weight               quantized, d_model × d_model
attn.out.scale                fp32
attn.out.bias                 d_model × fp32
ln2.weight                    d_model × fp32
ln2.bias                      d_model × fp32
ffn.up.weight                 quantized, ffn_dim × d_model
ffn.up.scale                  fp32
ffn.up.bias                   ffn_dim × fp32
ffn.down.weight               quantized, d_model × ffn_dim
ffn.down.scale                fp32
ffn.down.bias                 d_model × fp32
```

Weight matrices are row-major `[out_features × in_features]`. When
`weight_quant == ternary`, each is 2-bit packed at four values per byte
(same encoding as ternary embeddings); `in_features` MUST be a multiple of 4.

LayerNorm parameters and biases stay fp32 — they are a rounding error in size terms
and quantizing them buys nothing.

### 5.6 `FINAL_LN` (type 6)

```
weight    d_model × fp32
bias      d_model × fp32
```

Present even when `n_layers == 0`. A static model may write identity
(`weight = 1`, `bias = 0`) or learned values; the engine does not care.

### 5.7 `PROJECTION` (type 7)

```
weight    output_dim × d_model × fp32     (row-major [out × in])
bias      output_dim × fp32
```

**Always fp32, never quantized.** Two independent reasons:

1. It bridges the student's coordinate frame to the teacher's. Quantization noise
   here corrupts the distillation signal directly, and it is the smallest matrix in
   the model — the size saved is negligible against the quality lost.
2. It carries the Matryoshka ordering and the customer fingerprint. Both are
   rotations of a basis; quantizing a rotation destroys its orthogonality.

### 5.8 `METADATA` (type 8, optional)

UTF-8 JSON. Not parsed by the engine — provenance for humans and for the CLI:

```json
{
  "trained_at": "2026-09-01T12:00:00Z",
  "teacher": "emrecan/bert-base-turkish-cased-mean-nli-stsb-tr",
  "teacher_license": "apache-2.0",
  "corpus": ["wikipedia-tr", "oscar-tr", "quora-tr"],
  "tokenizer": "dbmdz/bert-base-turkish-cased",
  "train_commit": "…",
  "eval": { "tr_mteb_mean": 0.0, "stsb_tr_spearman": 0.0 }
}
```

## 6. Forward pass

Normative implementation: `reference/forward.py`. Stated here so the Rust engine
and the Python reference can be checked against a single written description.

### 6.1 Transformer path (`n_layers > 0`)

```
ids       = tokenize(text)                       truncate to max_seq_len
n_active  = count of ids != pad_token_id
x         = embedding_lookup(ids[:n_active])     [n_active, d_model]

for each layer:
    h = layer_norm(x, ln1.weight, ln1.bias)      parametric, eps 1e-5, biased var
    Q = bitlinear(h, attn.q, bias=None)
    K = bitlinear(h, attn.k, bias=None)
    V = bitlinear(h, attn.v, bias=None)
    A = softmax(Q Kᵀ / sqrt(d_head)) V           per head, d_head = d_model / n_heads
    x = x + bitlinear(merge_heads(A), attn.out, bias=attn.out.bias)
    h = layer_norm(x, ln2.weight, ln2.bias)
    f = gelu(bitlinear(h, ffn.up, bias=ffn.up.bias))
    x = x + bitlinear(f, ffn.down, bias=ffn.down.bias)

x         = layer_norm(x, final_ln.weight, final_ln.bias)
pooled    = mean(x, axis=0)                      over n_active rows only
projected = pooled @ projection.weightᵀ + projection.bias
out       = projected / ‖projected‖₂
```

**No positional encoding.** Sequences are short and pooling is order-invariant;
ternlight's POC found sinusoidal positions did not help. Revisit only with a
measurement, not an intuition.

**Padding is never processed.** Only `n_active` rows are computed, so there is no
attention mask — there are no padding keys to mask. Latency scales with input
length, not with `max_seq_len`.

**GELU is the exact erf form**, `0.5x(1 + erf(x/√2))`, matching
`torch.nn.functional.gelu(approximate='none')`. The tanh approximation diverges
from training-time math and MUST NOT be substituted.

### 6.2 Static path (`n_layers == 0`)

```
x         = embedding_lookup(ids[:n_active])
x         = layer_norm(x, final_ln.weight, final_ln.bias)
pooled    = mean(x, axis=0)
projected = pooled @ projection.weightᵀ + projection.bias
out       = projected / ‖projected‖₂
```

Identical tail. The only difference is that no layer blocks run — which is why the
engine needs no separate static code path.

### 6.3 `bitlinear(x, W, scale, bias)` — exact semantics

This function is where quantized inference goes silently wrong. Every step matters.

```python
x_norm  = layer_norm(x, normalized_shape=[in_features], eps=1e-5)   # NO affine
x_scale = 128.0 / max(abs(x_norm), axis=-1, keepdims=True).clamp(min=1e-5)
x_quant = round(x_norm * x_scale).clamp(-128, 127)                  # int8
y       = x_quant @ W_ternaryᵀ + bias                               # bias PRE-rescale
out     = y / (scale * x_scale)
```

Five details that are easy to get wrong, listed because each has bitten a previous
implementation:

1. **There is an internal, parameter-less LayerNorm** inside `bitlinear`, separate
   from `ln1`/`ln2`. Omitting it changes every downstream value.
2. **Activations are int8-quantized before the matmul**, not passed as fp32.
3. `x_scale` uses **128.0**, not 127.0 — this follows the `bitlinear` library's
   `activation_range = (-128, 127)` convention of taking `max(|range|)`.
4. **The bias is added before the rescale division**, matching `F.linear` semantics.
   Adding it after — the intuitive reading — is off by a factor of `scale · x_scale`.
5. `scale` is the packer's **AbsMedian**-derived value:
   `scale = 1 / clamp(median(|W_fp32|), min=1e-5)`, and the stored ternary weights
   are `round(W_fp32 × scale).clamp(-1, 1)`. AbsMedian, not AbsMean or AbsMax — a
   different threshold for snapping to zero, and therefore a different model.

When `weight_quant == fp32`, `bitlinear` degenerates to a plain
`F.linear(x, W, bias)` with **no** internal LayerNorm and no activation
quantization. This variant exists so engine correctness can be verified
independently of quantization correctness.

### 6.4 Matryoshka truncation

`embed(text, dims)` where `dims` is one of `matryoshka_dims` (or `output_dim`):

```
out = normalize(projected[:dims])
```

Truncate **then** re-normalize. The nested prefixes are trained to be independently
valid; the engine performs no other adjustment.

## 7. Fingerprinting

When `flags.has_fingerprint` is set, the projection weight stored in the file is
`R · W` for a customer-specific orthogonal `R` (`RᵀR = I`), derived deterministically
from a seed held in the license registry alongside `license_id`.

Because `R` is orthogonal, for any inputs `a`, `b`:

```
cos(Ra, Rb) = (Ra)ᵀ(Rb) = aᵀRᵀRb = aᵀb = cos(a, b)
```

Similarity is preserved **exactly**. Ranking, retrieval quality, and every downstream
metric are unchanged; only the raw coordinates differ. Query and corpus are embedded
by the same model, so an index built with a fingerprinted model is self-consistent.

Detection: embed a fixed probe set with the suspect model, solve orthogonal
Procrustes against the reference embeddings, match the recovered `R̂` against the
registry.

**The engine performs no license check of any kind.** There is no domain lock, no
phone-home, no obfuscation. `license_id` is provenance, not enforcement.

## 8. Versioning

Three independent versions with three separate compatibility contracts.

| Version | Lives in | Bumped by | Engine behaviour |
|---|---|---|---|
| `format_version` | this header | wire format change | Major mismatch → **reject** with expected range |
| `model_version` | this header | a new training run | Loads regardless; the *index* may not match |
| `index_version` | index manifest | index layout change | Client checks before searching |

An index manifest records the `model_id`, `model_version`, and `dims` it was built
with. The client compares them against the loaded model on startup. On mismatch it
reports:

> Index was built with lightembed-tr v1.2 but the loaded model is v2.0.
> Re-run `lightembed index`.

Minor and patch model bumps keep existing indexes valid. **A major model bump
invalidates them** — the embedding space has moved, and comparing vectors across it
produces plausible nonsense. That failure is silent by nature, which is exactly why
the check is mandatory and fails loudly.

## 9. Size reference

`lightembed-tr` as currently specified — `vocab_size` 32000, `d_model` 256,
`n_layers` 2, `n_heads` 4, `ffn_dim` 1024, `output_dim` 384:

| Section | ternary emb | int4 emb | int8 emb | fp32 emb |
|---|---:|---:|---:|---:|
| Tokenizer vocab | 290 KB | 290 KB | 290 KB | 290 KB |
| Embedding weights | 2.05 MB | 4.10 MB | 8.19 MB | 32.8 MB |
| Embedding scales | 128 KB | 128 KB | 128 KB | — |
| Layers (ternary) | 402 KB | 402 KB | 402 KB | 402 KB |
| Final LN | 2 KB | 2 KB | 2 KB | 2 KB |
| Projection (fp32) | 394 KB | 394 KB | 394 KB | 394 KB |
| **Total** | **≈ 3.2 MB** | **≈ 5.3 MB** | **≈ 9.4 MB** | **≈ 33.9 MB** |

The embedding table dominates every configuration — 64 % to 97 % of the file. This
is the number to attack when the model needs to get smaller, and it is attacked with
**vocabulary size**, not with architecture. Halving the vocab halves the model;
adding a transformer layer costs 201 KB.

> Note the projection is now a visible line item (394 KB) precisely because it stays
> fp32. At `output_dim` 384 and `d_model` 256 that is 98 304 weights. Shrinking
> `output_dim` shrinks it proportionally — another reason Matryoshka pays for itself.

## 10. Reserved for `format_version` 2

Deliberately **not** in v1, recorded so the ideas are not relitigated:

- Front-coded vocabulary (~40 % smaller, needs three implementations)
- 1.585-bit ternary packing, five values per byte (~18 % smaller weights)
- fp16 projection (halves 394 KB, needs a quality measurement first)
- Sparse embedding rows for rare tokens
- Multiple models in one file (a language pack)