pplx-pii-masking-GGUF (llama.cpp / llama-server)

GGUF split of perplexity-ai/pplx-pii-masking for llama.cpp serving:

File	What
pplx-pii-masking-backbone-f16.gguf	bidirectional Qwen3 backbone, f16 (1.20 GB)
heads.safetensors	token-classification head (1024x37) + sensitivity head (1024x1), fp32, 155 KB
pii_head_config.json	BIOES label list (37), Viterbi biases, head wiring notes
example_client.py	end-to-end client: embeddings -> heads -> constrained Viterbi -> spans
tokenizer.json	HF fast tokenizer (client-side offset mapping)

A Q8_0 variant (639 MB) also passed parity (8/8 identical span sets) and can be regenerated with convert_hf_to_gguf.py --outtype q8_0; only f16 is published for now.

Why the heads are outside the GGUF

llama.cpp has no per-token classification head support (its cls.output tensor is applied only under rerank pooling, per sequence), and refuses to load GGUFs containing tensors the architecture does not request. The backbone GGUF carries qwen3.attention.causal = false, which llama.cpp honors generically, so the stock qwen3 graph runs fully bidirectional, matching the HF model. Server per-token embeddings are the post-output_norm hidden states, exactly the input the heads consume.

Serve

llama-server -m pplx-pii-masking-backbone-f16.gguf \
  --embeddings --pooling none -c 4096 -b 4096 -ub 4096

Non-causal attention requires the whole input in one micro-batch: keep -ub = -b = max sequence length (4096 = the model's training window).

Query

Use the native /embeddings endpoint (/v1/embeddings rejects --pooling none):

curl -s http://localhost:8080/embeddings -d '{"content": "My email is jane@acme.com"}'
# -> [{"index": 0, "embedding": [[1024 floats] x n_tokens]}]   (unnormalized)

Then per token: logits37 = h @ W_cls.T + b_cls; sensitivity = mean_pool(h) @ W_sen.T + b_sen; decode spans with the constrained BIOES Viterbi from the pplx-pii-masking repo (model/viterbi.py). example_client.py does all of this.

Parity (validated through llama-server end to end)

8 internal benchmark docs, 4384 tokens, vs the torch fp32 reference:

GGUF	top-1 label agreement	logit cosine	identical span sets	max sens-logit delta
f16	4384/4384	0.999992	8/8	0.042
Q8_0	4384/4384	0.999951	8/8	0.089