ajaxdavis/donto-qwen3.8-27b-predicate-extractor

Donto-Qwen3.8-27B Predicate Extractor

An experimental, anchor-free, abundance-oriented predicate-extraction LoRA for short English document chunks.

This repository contains the selected V15 adapter for Qwen/Qwen3.8-27B, pinned to base revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. It turns a general Qwen3.8 language backbone into a specialist that emits validated Donto fact objects instead of conversational prose.

The headline result is 89.21% exact recall and 89.44% exact precision over a frozen, source- and template-disjoint, 100-document graph-first synthetic gold suite containing 5,338 target facts. All 2,718 forced structured calls succeeded.

This is a research release, not a claim that open-world extraction is solved. Read What the score does and does not mean and Limitations before deploying it.

The ambition

Knowledge systems were designed for a world in which producing structured claims was scarce. Generative models invert that constraint: candidate properties and relations can be produced in abundance. The new problem is not merely generating one summary or one canonical graph. It is holding a large, multi-directional, possibly contradictory field of claims without erasing disagreement.

Donto is built for that second problem. It is a contradiction-preserving claim substrate: emit free predicates now; preserve separate claims; defer ontology alignment, entity resolution, joining, standing, and reconciliation to later query-time processes. Donto-Qwen is the extraction front door. Its job is to read a bounded source unit carefully and produce many compact candidate claims that the larger Donto system can hold, compare, cite, align, and re-rank.

The purpose of this adapter is deliberately narrower than general chat:

1. recover atomic subject–predicate–object relations;
2. preserve attribution, negation, modality, qualification, and disagreement;
3. represent direct claims separately from derived or hypothetical views;
4. use free, concise predicates instead of forcing every source into a fixed ontology;
5. stop cleanly when a bounded extraction task is exhausted;
6. return only a validated tool call—never prose around hand-parsed JSON.

What this model is

Property	Value
Release	V15 / pilot-live-v15-exhaustive-unit-r16-v1
Base	Qwen3.8-27B, exact revision 1d4bf0f...
Artifact	PEFT LoRA adapter; base weights are not duplicated here
Primary modality	Text; the Qwen vision tower was neither trained nor evaluated
Intended context	Short document chunks; trained and served at 4,096 tokens
Output	Forced submit_facts tool call
Fact fields	s, p, o, c, h
Anchors	Disabled in this release
Thinking	Disabled for training and evaluated serving
Training hardware	One NVIDIA RTX 3090, 24 GB
Training method	4-bit NF4 QLoRA, BF16 compute, rank 16
Status	Experimental predicate extractor; not a general assistant

This adapter changes the base model's behaviour, not its underlying factual knowledge. It does not intrinsically make Qwen3.8 decode faster. Its efficiency advantages come from short inputs, compact structured outputs, shared prompt prefixes, fewer retries, and greater document concurrency.

Output contract

Every call must produce the submit_facts tool with this shape:

{
  "facts": [
    {
      "s": "ex:review-panel",
      "p": "postponed",
      "o": "ex:deployment",
      "c": 0.99,
      "h": false
    },
    {
      "s": "ex:postponement-wasted-money",
      "p": "contrastsWith",
      "o": "ex:postponement-protected-staff",
      "c": 0.92,
      "h": true
    }
  ],
  "more_supported_facts": false
}

• s: subject IRI, normally a stable ex:kebab-case identifier.
• p: concise camelCase free predicate, with standard RDF predicates allowed.
• o: entity IRI or string, numeric, or Boolean literal.
• c: model confidence in [0, 1]; this is not calibrated probability.
• h: false for direct/source-licensed claims; true for a derived, interpretive, hypothetical, or otherwise non-direct view.
• more_supported_facts: continuation signal for the current bounded task.

The exact OpenAI-compatible tool declaration is in schemas/tools.json.

Extraction architecture used for the reported scores

The benchmark did not ask one vague prompt to "extract everything." Each document was traversed through explicit bounded tasks:

Stage	Purpose
Metadata	Document, creator, date, and topic relations
D0	Atomic propositions directly expressed in a source unit
Local residual	Recover a deliberately omitted supported fact, then prove exhaustion
D1 unit	Licensed semantic views and local relational expansion
D2	Discourse-level, contrastive, causal, or interpretive relations (h:true)
Structural batches	Relations over the dynamically discovered entity inventory
Global residual	Final bounded coverage audit

This decomposition is important. A single generic extraction prompt is not expected to reproduce the published multi-stage score.

Results

Headline evaluations

Evaluation	Documents	Calls	Gold facts	Exact recall	Exact precision	Mean document recall	95% bootstrap recall floor	Structured success
Source-disjoint V15 validation	40	1,085	2,140	93.32%	93.89%	93.36%	91.74%	100%
Frozen graph-first Gold 100	100	2,718	5,338	89.21%	89.44%	89.50%	87.82%	100%

Gold 100 also produced:

• 4,762 exact matches from 5,338 targets;
• 5,324 valid predicted facts;
• zero failed calls;
• zero malformed fact objects;
• zero duplicate exact items;
• two reflexive entity errors.

Gold 100 by task

Task	Calls	Targets	Exact recall	Exact precision
D0 direct extraction	428	1,751	75.10%	75.14%
D1 unit views	766	645	95.35%	97.77%
D2 discourse views	212	446	93.72%	94.78%
Metadata	100	361	95.01%	95.28%
Global residual	200	300	96.33%	96.01%
Local residual	590	162	97.53%	98.14%
Structural batches	322	1,673	97.07%	96.55%

The overall release clears the project's predeclared 80% aggregate-gold target, but the direct D0 slice does not. That distinction is intentional and public. The weakest blind families were long wetlands argument/procurement documents; the strongest were bounded incidents, metadata, residual stopping, and structural relations.

Full row-level outputs—not summaries alone—are published under evaluation/.

What the score does and does not mean

The metric uses one-to-one exact keys over (s, p, o, h). A supported free predicate synonym or stable entity alias receives no credit, which can make the score conservative. Conversely, the gold documents are programmatically generated from explicit graphs and templates. They measure controlled compositional transfer, not the full ambiguity, OCR damage, genre diversity, or factual uncertainty of unseen real-world documents.

The 100-document suite was generated after V15 training, then frozen and hashed before the first V15 inference. It has:

• 100 unique document hashes;
• 73 parent templates;
• zero source-hash overlap with the 240 V15 training/validation documents;
• zero parent-template overlap with the 102 training/validation templates;
• 45–59 unique gold facts per document (mean 52.88).

The run used one seed. It is not a multi-seed reliability estimate and is not a human-blind real-document benchmark. The prior real llm_nerd fixture was heavily used for development diagnosis and is therefore not presented as blind evidence.

Training data

All distributable training, validation, validator, gold, manifest, and audit files are published at ajaxdavis/donto-qwen3.8-27b-predicate-extraction-data.

Canonical V15 training mixture:

• 4,255 training rows;
• 9,013 target facts;
• 200 source families;
• 4,123 synthetic task rows from 152 generated documents;
• 132 reviewed-real rows from 48 source families;
• 1,085 validation rows with 2,140 facts from 40 source- and template-disjoint synthetic documents;
• no training/validation source-family overlap;
• no training/validation parent-template overlap;
• every rendered row below the 4,096-token contract.

The dataset card separates synthetic and reviewed-real provenance and explains its mixed licensing status.

Training recipe

Parameter	Value
Base revision	1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Seed	38
Epochs / optimizer steps	1 / 532
LoRA rank / alpha / dropout	16 / 32 / 0.05
Trainable parameters	116,727,808
Learning rate	2e-5, cosine decay
Warmup	5% / 27 steps
Microbatch / accumulation	1 / 8
Maximum length	4,096
Quantization	bitsandbytes NF4, double quantization
Compute	BF16; TF32 enabled
Optimizer	paged AdamW 8-bit
Kernel	Liger enabled
Gradient checkpointing	Enabled, non-reentrant
Train runtime	20,860 seconds including epoch evaluation
Final explicit eval loss	0.0010853700805455446

Hybrid-layer LoRA coverage

Qwen3.8-27B contains 48 Gated DeltaNet layers and 16 full-attention layers. A generic q_proj/v_proj LoRA recipe would miss most recurrent projections. This adapter explicitly targeted 496 language modules / 992 LoRA tensors and no vision module:

• 48 each: linear_attn.in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj;
• 16 each: self_attn.q_proj, k_proj, v_proj, o_proj;
• 64 each: mlp.gate_proj, up_proj, down_proj.

The complete name-level audit is artifacts/lora-target-audit.json.

Serving with vLLM on one RTX 3090

The physically tested path used vLLM 0.27.1, Transformers 5.15.0, bitsandbytes 0.50.1, a 4,096-token limit, four concurrent sequences, and prefix caching. vLLM required the tensor-preserving conditional-key remap under runtime/vllm-conditional/; the root adapter remains the canonical native PEFT artifact.

hf download ajaxdavis/donto-qwen3.8-27b-predicate-extractor \
  --local-dir ./donto-qwen

vllm serve Qwen/Qwen3.8-27B \
  --served-model-name qwen3.8-27b \
  --language-model-only \
  --quantization bitsandbytes \
  --load-format bitsandbytes \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching \
  --generation-config vllm \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-lora \
  --lora-modules donto-qwen=./donto-qwen/runtime/vllm-conditional \
  --max-loras 1 \
  --max-lora-rank 16

The exact tested launcher includes additional GPU-specific cache and attention flags and is preserved at training/source/start_runpod_quality_canary.sh.

Then call the forced tool and validate its arguments with a real schema. A complete Pydantic/OpenAI-compatible example is in examples/structured_client.py.

Native PEFT use

The root adapter_model.safetensors is the canonical PEFT adapter. Load it over the exact pinned base revision with current Transformers and PEFT. The training script in training/source/train_qwen38_donto_3090.py contains the memory-safe text-only 3090 loader and exact QLoRA configuration.

Do not attach the root native adapter directly to vLLM if its Qwen3.8 key resolver expects the conditional language-model root; use the verified remap supplied in runtime/vllm-conditional.

Efficiency profile

• Adapter weights are about 467 MB; the base model remains the dominant memory and compute cost.
• The adapter does not improve raw decode tokens/second by itself.
• Short chunks reduce prefill and KV-cache costs.
• Prefix caching reuses the large common tool/prompt prefix across documents.
• The anchor-free five-field format reduces output tokens per useful fact.
• Four concurrent sequences fit in the physically tested 24 GB 3090 serving path.
• Gold 100 completed 2,718 structured calls in 8,511 seconds at concurrency four, about 19.2 bounded decisions per minute. This is not a generic tokens/second benchmark.

Limitations

1. Direct extraction is the weakest blind slice. Gold-100 D0 exact recall and precision are both about 75%.
2. Synthetic-domain bias. The main quantitative evidence comes from generated graph/template documents, not a large human-authored blind corpus.
3. One training seed. The recipe has not been repeated across multiple seeds.
4. Two reflexive errors occurred in Gold 100. Role direction and reflexivity still require downstream validation.
5. No evidence anchors. This release deliberately omits source spans. A separate citer or human review is required when provenance is important.
6. Free predicates are not ontology alignment. Predicate and entity aliases may vary; alignment belongs downstream.
7. Confidence is uncalibrated. Do not interpret c as a probability of truth.
8. h:true is not established fact. It marks derived or interpretive output.
9. Not a contradiction resolver. It may emit incompatible claims by design; Donto preserves and later ranks them.
10. Not for unattended high-stakes use. Legal, medical, financial, historical, or personal claims require source review.
11. Not a general assistant. Fine-tuning trades conversational versatility for structured extraction behaviour.
12. Multimodal behaviour is unevaluated. No vision module was adapted.

Development history and negative results

This release is the fifteenth curriculum iteration, trained fresh from the pinned stock base. Rejected adapters were never reused as weight bases. The repository preserves rejection reports for V11–V14 because the failures are informative:

• V11 learned structured calls but missed its precision gate and had semantic contamination in one generator family.
• V12 completed finitely but was too terse and failed real-document coverage.
• V13 reached strong source-disjoint exact metrics but retained role/scope errors.
• V14 nearly memorized its supervised contract (about 99% exact) yet transferred poorly to the real development document because its curriculum rewarded short, clean pages.
• V15 started fresh and taught dense D0 pages, nonempty recovery passes, explicit exhaustion, larger local D1 views, and broader graph families.

See docs/ for the plan, audits, launch receipt, runbook, and complete negative-result reports.

Reproducibility and integrity

• Native final adapter SHA-256: 40d11bcad20061f8291741c4dd8a3701405902f336c27b60592d62d6b2f7a974
• vLLM remapped adapter SHA-256: 28df663ffa70bf541cafff2c9bb60df11e4fe000a2ea9124b346a9cc3eb20d8b
• Training corpus SHA-256: 1b3630d00d11476c7f1e941a60a442a5eb5eb57dd882b7d188ef9b85d461e738
• V15 validation SHA-256: c360e8296e5ec7973f2fab8416d0b30e2aa850fe6b2cc618bb3fa52894d818b2
• Gold-100 dataset SHA-256: 6f16c87d06c2a0b850321b8750e54a63364937cfa2823f34f698f6c00f160f1c
• Gold-100 evaluation SHA-256: a21e32e6db21f217af4ad7fc66cbcb3836f0607567bdaab9fcc5c11859e2fd55

The repository includes intermediate adapter checkpoints 100–500, trainer states, the exact trainer/evaluator/generator sources, training and serving logs, run contracts, target-module audit, and full row-level evaluation outputs. Unsafe and unnecessary optimizer/RNG pickle states are not part of the public release.

The Hub upload, clean-download hash checks, and unauthenticated browser proof are recorded in PUBLICATION-RECEIPT-20260825.md.

Licence and acknowledgements

The adapter and code are released under Apache-2.0, consistent with the pinned Qwen3.8-27B base. Dataset licensing is documented separately because the corpus contains both generated material and a small reviewed-real research tier.

Built on Qwen3.8-27B, PEFT, Transformers, bitsandbytes, Liger Kernel, and vLLM. The practical RTX 3090 serving work was informed by syv-ai/qwen38-27b-rtx3090.

Citation

@misc{donto_qwen38_v15_2026,
  author       = {Davis, Thomas},
  title        = {Donto-Qwen3.8-27B Predicate Extractor V15},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/ajaxdavis/donto-qwen3.8-27b-predicate-extractor}},
  note         = {Experimental LoRA adapter for structured Donto predicate extraction}
}