kgrabko commited on
Commit
e5e25be
·
verified ·
1 Parent(s): 45556d1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +70 -63
README.md CHANGED
@@ -1,18 +1,18 @@
1
  ---
2
- language:
3
  - en
4
- - zh
5
- - ja
6
- - ko
7
- - fr
8
- - es
9
- - pt
10
- - de
11
- - it
12
- - ru
13
- - ar
14
- - vi
15
- - th
16
  tags:
17
  - code
18
  - coding
@@ -25,50 +25,56 @@ license: gpl-3.0
25
 
26
  # JiRack Coder 7B INT8
27
 
28
- A fast and efficient coding assistant with a clean built-in web UI, powered by Qwen2.5-Coder-7B-Instruct base and optimized using Microsoft ONNX Runtime.
29
 
30
- - JiRack is cloud model and save money on cloud and can be used as expert model in RAG on cloud with ONNX JiRack java server as alternative.
31
 
32
  ## Quick Start
33
- Watch the JiRack Coder 7B in action:
34
- **DEMO**: [JiRack Coder 7B Web UI](https://youtu.be/I8AAITUiI64)
35
 
 
 
 
36
 
37
  ### Run with Docker
38
 
39
- ---
40
- --Default CPU--
41
 
42
- - docker run -d \
 
43
  --name jirack_coder_7b \
44
  -p 7869:7869 \
45
  --restart unless-stopped \
46
  cmsmanhattan/jirack_coder_7b_int8_qwenbase:latest
 
47
 
48
- --Multi CPU--
49
 
50
- - docker run -d \
 
51
  --name jirack_coder_7b \
52
  -p 7869:7869 \
53
  --restart unless-stopped \
54
  --memory=20g \
55
  --cpus=12 \
56
  cmsmanhattan/jirack_coder_7b_int8_qwenbase:latest
 
57
 
58
- ---GPU--
59
-
60
- - docker run -d \
61
- --name jirack_coder_7b \
62
- -p 7869:7869 \
63
- --gpus all \
64
- --restart unless-stopped \
65
- cmsmanhattan/jirack_coder_7b_int8:1.0.2
66
 
67
- ---
68
- services:
 
 
 
 
 
 
69
 
 
70
 
71
-
 
 
72
  image: cmsmanhattan/jirack_coder_7b_int8:1.0.2
73
  container_name: jirack_onnx_service
74
  ports:
@@ -86,15 +92,16 @@ services:
86
  deploy:
87
  resources:
88
  limits:
89
- memory: 16g
 
90
 
91
  ## Access the UI
92
 
93
  Once the container is running, open your browser and navigate to:
94
 
95
- **`http://localhost:7869`**
96
 
97
- This opens the **JiRack Coder UI** — a clean web interface designed for coding.
98
 
99
  ## Changing the Port
100
 
@@ -102,56 +109,56 @@ The listening port can be easily modified directly from the **Settings** panel w
102
 
103
  ## Licensing
104
 
105
- - The **JiRack Coder 7B model** is released under the **GNU General Public License v3.0 (GPL-3.0)**.
106
- - All **JiRack UI clients** are provided under a commercial license.
107
  - However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.
108
 
109
- **JiRack Coder 32B** is available exclusively under a commercial enterprise license.
110
 
111
- - JiRack Coder 32B https://huggingface.co/CMSManhattan/JiRackCoderReasoing_32b
112
 
113
- - JiRack Coder 14B https://huggingface.co/CMSManhattan/JiRackCoderReasoing_14b
 
 
114
 
115
- - JiRack Coder 8B https://huggingface.co/CMSManhattan/JiRackCoderReasoing_8b
116
 
117
- For commercial licensing, cluster deployment, or enterprise use of the JiRack Coder 32B and JiRack Coder 14B , please contact us.
118
- - JiRack MS Windows 11 Desktop chat client with ollama API setup : https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
119
- - Live email chat with model via support@cmsmanhattan.com
120
 
 
121
 
122
  ## Hardware Recommendations for AMD Systems
123
 
124
- ### Recommended Hardware for JiRack Coder 7B INT8 . It is one dcoker container
125
 
126
- | Use Case | CPU | GPU (ROCm) | VRAM / RAM | Expected Speed | Recommendation |
127
- |-----------------------|----------------------------------|-----------------------------------|----------------|---------------------|--------------------|
128
- | **Recommended** | Ryzen 7 7700 / 9700X | RX 7900 XTX / 7900 XT | 24GB VRAM | 50-75 tokens/s | Best choice |
129
- | **High Performance** | Ryzen 9 7950X / 9950X | RX 7900 XTX | 24GB+ VRAM | 65-90 tokens/s | Excellent |
130
- | **Enterprise** | EPYC 7003/9004 series | MI300X or 2x RX 7900 XTX | 48GB+ VRAM | 90-140 tokens/s | For 32B model |
131
- | **Budget Option** | Ryzen 5 7600 / 9600X | RX 7800 XT (16GB) | 16GB VRAM | 35-50 tokens/s | Acceptable |
132
 
133
- ### Important Memory Notes
134
 
135
- Even though the 7B INT8 model itself takes approximately **8–9 GB**, we recommend **at least 24GB VRAM** for the following reasons:
136
 
137
- - KV-cache consumption during generation (especially with long context)
138
  - ONNX Runtime overhead and temporary buffers
139
- - System stability and to avoid Out of Memory errors
140
  - Room for larger context windows
141
 
142
  **Minimum recommended:** 24GB VRAM (RX 7900 series)
143
  **Ideal:** 24–32GB VRAM
144
 
145
- For pure CPU inference (no GPU), we recommend at least **64GB system RAM** (Ryzen 9 7950X/9950X).
146
 
147
- ---
148
  I added the default model in full FP32 precision, which is approximately 62 GB in size. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.
149
 
150
-
151
  ## 📧 Contact & Licensing
152
- For joint venture opportunities, hardware integration, or licensing inquiries:
153
- - **Email:** [grabko@cmsmanhattan.com](mailto:grabko@cmsmanhattan.com)
154
- - **Phone:** +1 (516) 777-0945
155
- - **Location:** New York, USA
156
 
 
157
 
 
 
 
 
1
  ---
2
+ language:
3
  - en
4
+ - zh
5
+ - ja
6
+ - ko
7
+ - fr
8
+ - es
9
+ - pt
10
+ - de
11
+ - it
12
+ - ru
13
+ - ar
14
+ - vi
15
+ - th
16
  tags:
17
  - code
18
  - coding
 
25
 
26
  # JiRack Coder 7B INT8
27
 
28
+ A fast and efficient coding assistant with a clean built-in web UI, powered by **Qwen2.5-Coder-7B-Instruct** and optimized using Microsoft ONNX Runtime.
29
 
30
+ - JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative.
31
 
32
  ## Quick Start
 
 
33
 
34
+ Watch JiRack Coder 7B in action:
35
+
36
+ **DEMO**: [JiRack Coder 7B Web UI](https://youtu.be/I8AAITUiI64)
37
 
38
  ### Run with Docker
39
 
40
+ **Default CPU**
 
41
 
42
+ ```bash
43
+ docker run -d \
44
  --name jirack_coder_7b \
45
  -p 7869:7869 \
46
  --restart unless-stopped \
47
  cmsmanhattan/jirack_coder_7b_int8_qwenbase:latest
48
+ ```
49
 
50
+ **Multi CPU**
51
 
52
+ ```bash
53
+ docker run -d \
54
  --name jirack_coder_7b \
55
  -p 7869:7869 \
56
  --restart unless-stopped \
57
  --memory=20g \
58
  --cpus=12 \
59
  cmsmanhattan/jirack_coder_7b_int8_qwenbase:latest
60
+ ```
61
 
62
+ **GPU**
 
 
 
 
 
 
 
63
 
64
+ ```bash
65
+ docker run -d \
66
+ --name jirack_coder_7b \
67
+ -p 7869:7869 \
68
+ --gpus all \
69
+ --restart unless-stopped \
70
+ cmsmanhattan/jirack_coder_7b_int8:1.0.2
71
+ ```
72
 
73
+ ### Docker Compose Example
74
 
75
+ ```yaml
76
+ services:
77
+ jirack:
78
  image: cmsmanhattan/jirack_coder_7b_int8:1.0.2
79
  container_name: jirack_onnx_service
80
  ports:
 
92
  deploy:
93
  resources:
94
  limits:
95
+ memory: 16g
96
+ ```
97
 
98
  ## Access the UI
99
 
100
  Once the container is running, open your browser and navigate to:
101
 
102
+ `http://localhost:7869`
103
 
104
+ This opens the JiRack Coder UI — a clean web interface designed for coding.
105
 
106
  ## Changing the Port
107
 
 
109
 
110
  ## Licensing
111
 
112
+ - The JiRack Coder 7B model is released under the GNU General Public License v3.0 (GPL-3.0).
113
+ - All JiRack UI clients are provided under a commercial license.
114
  - However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.
115
 
116
+ JiRack Coder 32B is available exclusively under a commercial enterprise license.
117
 
118
+ ## Other JiRack Coder Models
119
 
120
+ - JiRack Coder 32B
121
+ - JiRack Coder 14B
122
+ - JiRack Coder 8B
123
 
124
+ For commercial licensing, cluster deployment, or enterprise use, please contact us.
125
 
126
+ - **JiRack MS Windows 11 Desktop Client (with Ollama API):**
127
+ https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
 
128
 
129
+ - **Live email chat with the model:** support@cmsmanhattan.com
130
 
131
  ## Hardware Recommendations for AMD Systems
132
 
133
+ ### Recommended Hardware for JiRack Coder 7B INT8 (single Docker container)
134
 
135
+ | Use Case | CPU | GPU (ROCm) | VRAM / RAM | Expected Speed | Recommendation |
136
+ |------------------|--------------------------|---------------------------|------------|------------------|----------------|
137
+ | Recommended | Ryzen 7 7700 / 9700X | RX 7900 XTX / 7900 XT | 24GB VRAM | 50-75 tokens/s | Best choice |
138
+ | High Performance | Ryzen 9 7950X / 9950X | RX 7900 XTX | 24GB+ VRAM | 65-90 tokens/s | Excellent |
139
+ | Enterprise | EPYC 7003/9004 series | MI300X or 2x RX 7900 XTX | 48GB+ VRAM | 90-140 tokens/s | For 32B model |
140
+ | Budget Option | Ryzen 5 7600 / 9600X | RX 7800 XT (16GB) | 16GB VRAM | 35-50 tokens/s | Acceptable |
141
 
142
+ ## Important Memory Notes
143
 
144
+ Even though the 7B INT8 model itself takes approximately 8–9 GB, we recommend at least 24GB VRAM for the following reasons:
145
 
146
+ - KV-cache consumption during generation, especially with long context
147
  - ONNX Runtime overhead and temporary buffers
148
+ - System stability and avoiding out-of-memory errors
149
  - Room for larger context windows
150
 
151
  **Minimum recommended:** 24GB VRAM (RX 7900 series)
152
  **Ideal:** 24–32GB VRAM
153
 
154
+ For pure CPU inference (no GPU), we recommend at least 64GB system RAM (Ryzen 9 7950X/9950X).
155
 
 
156
  I added the default model in full FP32 precision, which is approximately 62 GB in size. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.
157
 
 
158
  ## 📧 Contact & Licensing
 
 
 
 
159
 
160
+ For joint venture opportunities, hardware integration, or licensing inquiries:
161
 
162
+ - **Email:** grabko@cmsmanhattan.com
163
+ - **Phone:** +1 (516) 777-0945
164
+ - **Location:** New York, USA