Update README.md
Browse files
README.md
CHANGED
|
@@ -1,18 +1,18 @@
|
|
| 1 |
---
|
| 2 |
-
language:
|
| 3 |
- en
|
| 4 |
-
- zh
|
| 5 |
-
- ja
|
| 6 |
-
- ko
|
| 7 |
-
- fr
|
| 8 |
-
- es
|
| 9 |
-
- pt
|
| 10 |
-
- de
|
| 11 |
-
- it
|
| 12 |
-
- ru
|
| 13 |
-
- ar
|
| 14 |
-
- vi
|
| 15 |
-
- th
|
| 16 |
tags:
|
| 17 |
- code
|
| 18 |
- coding
|
|
@@ -25,50 +25,56 @@ license: gpl-3.0
|
|
| 25 |
|
| 26 |
# JiRack Coder 7B INT8
|
| 27 |
|
| 28 |
-
A fast and efficient coding assistant with a clean built-in web UI, powered by Qwen2.5-Coder-7B-Instruct
|
| 29 |
|
| 30 |
-
- JiRack is cloud model
|
| 31 |
|
| 32 |
## Quick Start
|
| 33 |
-
Watch the JiRack Coder 7B in action:
|
| 34 |
-
**DEMO**: [JiRack Coder 7B Web UI](https://youtu.be/I8AAITUiI64)
|
| 35 |
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
### Run with Docker
|
| 38 |
|
| 39 |
-
|
| 40 |
-
--Default CPU--
|
| 41 |
|
| 42 |
-
|
|
|
|
| 43 |
--name jirack_coder_7b \
|
| 44 |
-p 7869:7869 \
|
| 45 |
--restart unless-stopped \
|
| 46 |
cmsmanhattan/jirack_coder_7b_int8_qwenbase:latest
|
|
|
|
| 47 |
|
| 48 |
-
|
| 49 |
|
| 50 |
-
|
|
|
|
| 51 |
--name jirack_coder_7b \
|
| 52 |
-p 7869:7869 \
|
| 53 |
--restart unless-stopped \
|
| 54 |
--memory=20g \
|
| 55 |
--cpus=12 \
|
| 56 |
cmsmanhattan/jirack_coder_7b_int8_qwenbase:latest
|
|
|
|
| 57 |
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
- docker run -d \
|
| 61 |
-
--name jirack_coder_7b \
|
| 62 |
-
-p 7869:7869 \
|
| 63 |
-
--gpus all \
|
| 64 |
-
--restart unless-stopped \
|
| 65 |
-
cmsmanhattan/jirack_coder_7b_int8:1.0.2
|
| 66 |
|
| 67 |
-
|
| 68 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
|
|
|
| 70 |
|
| 71 |
-
|
|
|
|
|
|
|
| 72 |
image: cmsmanhattan/jirack_coder_7b_int8:1.0.2
|
| 73 |
container_name: jirack_onnx_service
|
| 74 |
ports:
|
|
@@ -86,15 +92,16 @@ services:
|
|
| 86 |
deploy:
|
| 87 |
resources:
|
| 88 |
limits:
|
| 89 |
-
memory: 16g
|
|
|
|
| 90 |
|
| 91 |
## Access the UI
|
| 92 |
|
| 93 |
Once the container is running, open your browser and navigate to:
|
| 94 |
|
| 95 |
-
|
| 96 |
|
| 97 |
-
This opens the
|
| 98 |
|
| 99 |
## Changing the Port
|
| 100 |
|
|
@@ -102,56 +109,56 @@ The listening port can be easily modified directly from the **Settings** panel w
|
|
| 102 |
|
| 103 |
## Licensing
|
| 104 |
|
| 105 |
-
- The
|
| 106 |
-
- All
|
| 107 |
- However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.
|
| 108 |
|
| 109 |
-
|
| 110 |
|
| 111 |
-
|
| 112 |
|
| 113 |
-
- JiRack Coder
|
|
|
|
|
|
|
| 114 |
|
| 115 |
-
|
| 116 |
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
- Live email chat with model via support@cmsmanhattan.com
|
| 120 |
|
|
|
|
| 121 |
|
| 122 |
## Hardware Recommendations for AMD Systems
|
| 123 |
|
| 124 |
-
### Recommended Hardware for JiRack Coder 7B INT8
|
| 125 |
|
| 126 |
-
| Use Case
|
| 127 |
-
|------------------
|
| 128 |
-
|
|
| 129 |
-
|
|
| 130 |
-
|
|
| 131 |
-
|
|
| 132 |
|
| 133 |
-
##
|
| 134 |
|
| 135 |
-
Even though the 7B INT8 model itself takes approximately
|
| 136 |
|
| 137 |
-
- KV-cache consumption during generation
|
| 138 |
- ONNX Runtime overhead and temporary buffers
|
| 139 |
-
- System stability and
|
| 140 |
- Room for larger context windows
|
| 141 |
|
| 142 |
**Minimum recommended:** 24GB VRAM (RX 7900 series)
|
| 143 |
**Ideal:** 24–32GB VRAM
|
| 144 |
|
| 145 |
-
For pure CPU inference (no GPU), we recommend at least
|
| 146 |
|
| 147 |
-
---
|
| 148 |
I added the default model in full FP32 precision, which is approximately 62 GB in size. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.
|
| 149 |
|
| 150 |
-
|
| 151 |
## 📧 Contact & Licensing
|
| 152 |
-
For joint venture opportunities, hardware integration, or licensing inquiries:
|
| 153 |
-
- **Email:** [grabko@cmsmanhattan.com](mailto:grabko@cmsmanhattan.com)
|
| 154 |
-
- **Phone:** +1 (516) 777-0945
|
| 155 |
-
- **Location:** New York, USA
|
| 156 |
|
|
|
|
| 157 |
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
language:
|
| 3 |
- en
|
| 4 |
+
- zh
|
| 5 |
+
- ja
|
| 6 |
+
- ko
|
| 7 |
+
- fr
|
| 8 |
+
- es
|
| 9 |
+
- pt
|
| 10 |
+
- de
|
| 11 |
+
- it
|
| 12 |
+
- ru
|
| 13 |
+
- ar
|
| 14 |
+
- vi
|
| 15 |
+
- th
|
| 16 |
tags:
|
| 17 |
- code
|
| 18 |
- coding
|
|
|
|
| 25 |
|
| 26 |
# JiRack Coder 7B INT8
|
| 27 |
|
| 28 |
+
A fast and efficient coding assistant with a clean built-in web UI, powered by **Qwen2.5-Coder-7B-Instruct** and optimized using Microsoft ONNX Runtime.
|
| 29 |
|
| 30 |
+
- JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative.
|
| 31 |
|
| 32 |
## Quick Start
|
|
|
|
|
|
|
| 33 |
|
| 34 |
+
Watch JiRack Coder 7B in action:
|
| 35 |
+
|
| 36 |
+
**DEMO**: [JiRack Coder 7B Web UI](https://youtu.be/I8AAITUiI64)
|
| 37 |
|
| 38 |
### Run with Docker
|
| 39 |
|
| 40 |
+
**Default CPU**
|
|
|
|
| 41 |
|
| 42 |
+
```bash
|
| 43 |
+
docker run -d \
|
| 44 |
--name jirack_coder_7b \
|
| 45 |
-p 7869:7869 \
|
| 46 |
--restart unless-stopped \
|
| 47 |
cmsmanhattan/jirack_coder_7b_int8_qwenbase:latest
|
| 48 |
+
```
|
| 49 |
|
| 50 |
+
**Multi CPU**
|
| 51 |
|
| 52 |
+
```bash
|
| 53 |
+
docker run -d \
|
| 54 |
--name jirack_coder_7b \
|
| 55 |
-p 7869:7869 \
|
| 56 |
--restart unless-stopped \
|
| 57 |
--memory=20g \
|
| 58 |
--cpus=12 \
|
| 59 |
cmsmanhattan/jirack_coder_7b_int8_qwenbase:latest
|
| 60 |
+
```
|
| 61 |
|
| 62 |
+
**GPU**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
+
```bash
|
| 65 |
+
docker run -d \
|
| 66 |
+
--name jirack_coder_7b \
|
| 67 |
+
-p 7869:7869 \
|
| 68 |
+
--gpus all \
|
| 69 |
+
--restart unless-stopped \
|
| 70 |
+
cmsmanhattan/jirack_coder_7b_int8:1.0.2
|
| 71 |
+
```
|
| 72 |
|
| 73 |
+
### Docker Compose Example
|
| 74 |
|
| 75 |
+
```yaml
|
| 76 |
+
services:
|
| 77 |
+
jirack:
|
| 78 |
image: cmsmanhattan/jirack_coder_7b_int8:1.0.2
|
| 79 |
container_name: jirack_onnx_service
|
| 80 |
ports:
|
|
|
|
| 92 |
deploy:
|
| 93 |
resources:
|
| 94 |
limits:
|
| 95 |
+
memory: 16g
|
| 96 |
+
```
|
| 97 |
|
| 98 |
## Access the UI
|
| 99 |
|
| 100 |
Once the container is running, open your browser and navigate to:
|
| 101 |
|
| 102 |
+
`http://localhost:7869`
|
| 103 |
|
| 104 |
+
This opens the JiRack Coder UI — a clean web interface designed for coding.
|
| 105 |
|
| 106 |
## Changing the Port
|
| 107 |
|
|
|
|
| 109 |
|
| 110 |
## Licensing
|
| 111 |
|
| 112 |
+
- The JiRack Coder 7B model is released under the GNU General Public License v3.0 (GPL-3.0).
|
| 113 |
+
- All JiRack UI clients are provided under a commercial license.
|
| 114 |
- However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.
|
| 115 |
|
| 116 |
+
JiRack Coder 32B is available exclusively under a commercial enterprise license.
|
| 117 |
|
| 118 |
+
## Other JiRack Coder Models
|
| 119 |
|
| 120 |
+
- JiRack Coder 32B
|
| 121 |
+
- JiRack Coder 14B
|
| 122 |
+
- JiRack Coder 8B
|
| 123 |
|
| 124 |
+
For commercial licensing, cluster deployment, or enterprise use, please contact us.
|
| 125 |
|
| 126 |
+
- **JiRack MS Windows 11 Desktop Client (with Ollama API):**
|
| 127 |
+
https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
|
|
|
|
| 128 |
|
| 129 |
+
- **Live email chat with the model:** support@cmsmanhattan.com
|
| 130 |
|
| 131 |
## Hardware Recommendations for AMD Systems
|
| 132 |
|
| 133 |
+
### Recommended Hardware for JiRack Coder 7B INT8 (single Docker container)
|
| 134 |
|
| 135 |
+
| Use Case | CPU | GPU (ROCm) | VRAM / RAM | Expected Speed | Recommendation |
|
| 136 |
+
|------------------|--------------------------|---------------------------|------------|------------------|----------------|
|
| 137 |
+
| Recommended | Ryzen 7 7700 / 9700X | RX 7900 XTX / 7900 XT | 24GB VRAM | 50-75 tokens/s | Best choice |
|
| 138 |
+
| High Performance | Ryzen 9 7950X / 9950X | RX 7900 XTX | 24GB+ VRAM | 65-90 tokens/s | Excellent |
|
| 139 |
+
| Enterprise | EPYC 7003/9004 series | MI300X or 2x RX 7900 XTX | 48GB+ VRAM | 90-140 tokens/s | For 32B model |
|
| 140 |
+
| Budget Option | Ryzen 5 7600 / 9600X | RX 7800 XT (16GB) | 16GB VRAM | 35-50 tokens/s | Acceptable |
|
| 141 |
|
| 142 |
+
## Important Memory Notes
|
| 143 |
|
| 144 |
+
Even though the 7B INT8 model itself takes approximately 8–9 GB, we recommend at least 24GB VRAM for the following reasons:
|
| 145 |
|
| 146 |
+
- KV-cache consumption during generation, especially with long context
|
| 147 |
- ONNX Runtime overhead and temporary buffers
|
| 148 |
+
- System stability and avoiding out-of-memory errors
|
| 149 |
- Room for larger context windows
|
| 150 |
|
| 151 |
**Minimum recommended:** 24GB VRAM (RX 7900 series)
|
| 152 |
**Ideal:** 24–32GB VRAM
|
| 153 |
|
| 154 |
+
For pure CPU inference (no GPU), we recommend at least 64GB system RAM (Ryzen 9 7950X/9950X).
|
| 155 |
|
|
|
|
| 156 |
I added the default model in full FP32 precision, which is approximately 62 GB in size. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.
|
| 157 |
|
|
|
|
| 158 |
## 📧 Contact & Licensing
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
|
| 160 |
+
For joint venture opportunities, hardware integration, or licensing inquiries:
|
| 161 |
|
| 162 |
+
- **Email:** grabko@cmsmanhattan.com
|
| 163 |
+
- **Phone:** +1 (516) 777-0945
|
| 164 |
+
- **Location:** New York, USA
|