---
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
datasets:
- the-clanker-lover/steelman-sft-ada
- nuprl/MultiPL-E
language: en
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
- ada
- spark
- code
model-index:
- name: ada-coder-qwen2.5-7b
  results:
  - task:
      type: text-generation
    dataset:
      name: MultiPL-E HumanEval-Ada
      type: nuprl/MultiPL-E
      config: humaneval-adb
    metrics:
    - type: pass@1
      value: 0.414
      name: pass@1
    - type: compile_rate
      value: 0.771
      name: Compile rate
---

# ada-coder-qwen2.5-7b

<p align="center"><img src="assets/logo.svg" alt="Ada coder logo: a Jacquard punched card, after Ada Lovelace's loom analogy, whose holes form a teal check mark" width="200"></p>

[Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) fine-tuned with LoRA to write Ada 2022 and SPARK code from a task description, spec or signature.

> [!WARNING]
> **Review every line.** The output is not qualified for DO-178C, EN 50128, ISO 26262 or any other certification process. Compile it, test it and, for SPARK, prove it before use.

## 🚀 Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jgalego/ada-coder-qwen2.5-7b"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto")

messages = [
    {"role": "system", "content": "You write Ada 2022 and SPARK code that compiles cleanly with GNAT."},
    {"role": "user", "content": "Write a function that returns the greatest common divisor of two Positive numbers."},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True))
```

It also completes raw Ada source with no chat template, which is how it is scored below.

## 🏋️ Training

| | |
|---|---|
| Base | [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) |
| Data | `r7_dataset.jsonl` from [the-clanker-lover/steelman-sft-ada](https://huggingface.co/datasets/the-clanker-lover/steelman-sft-ada), 6,110 instruction/code pairs that compile with GNAT |
| Method | LoRA, r=16, alpha=32, all linear layers, merged |
| Steps | 378 (1.0 epochs), batch size 16, max length 2048 |
| Learning rate | 0.0002, cosine |
| Mean train loss | 0.2128 |
| Hardware | NVIDIA A10G, 69 min |

## 📊 Results

[nuprl/MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E) HumanEval-Ada (`humaneval-adb`, 157 problems). Greedy decoding from the raw prompt, then built and run with the GNAT flags from MultiPL-E's `eval_adb.py`. A program passes when every `pragma Assert` holds.

| Model | n | pass@1 | Compile rate | Runtime errors | Timeouts |
|---|---|---|---|---|---|
| jgalego/ada-coder-qwen2.5-7b | 157 | 0.414 | 0.771 | 56 | 0 |
| jgalego/ada-coder-qwen2.5-7b@grpo-1 | 157 | 0.414 | 0.796 | 60 | 0 |
| jgalego/ada-coder-qwen2.5-7b@sft-2 | 157 | 0.401 | 0.771 | 58 | 0 |
| Qwen/Qwen2.5-Coder-7B-Instruct | 157 | 0.312 | 0.471 | 25 | 0 |

Reference scores reported by the Steelman authors ([model card](https://huggingface.co/the-clanker-lover/steelman-14b-ada), [report](https://huggingface.co/datasets/the-clanker-lover/steelman-sft-ada/blob/main/humaneval_ada/multipl_e_r6v3_report.md)) on the same benchmark with their own harness, temperature 0:

| Model | pass@1 | Compile rate |
|---|---|---|
| Steelman 14B v0.3 (Qwen2.5-Coder-14B + R7 data) | 0.459 | 0.854 |
| Steelman 14B v0.2 (Qwen2.5-Coder-14B-Instruct + R6 data) | 0.471 | 0.752 |
| Qwen2.5-Coder-14B, untuned | 0.344 | 0.510 |

## ⚠️ Limitations

- Scored on small HumanEval-style functions, not on full programs or packages.
- Contracts are at most checked at run time (`-gnata`); nothing here runs GNATprove.
- Trained on a single dataset; style follows that data.