Instructions to use Trelis/Llama-2-7b-chat-hf-function-calling-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Trelis/Llama-2-7b-chat-hf-function-calling-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Trelis/Llama-2-7b-chat-hf-function-calling-v2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Trelis/Llama-2-7b-chat-hf-function-calling-v2")
model = AutoModelForCausalLM.from_pretrained("Trelis/Llama-2-7b-chat-hf-function-calling-v2", device_map="auto")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Trelis/Llama-2-7b-chat-hf-function-calling-v2 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M
# Run inference directly in the terminal:
./llama-cli -hf Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M

Use Docker

docker model run hf.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M

LM Studio
Jan

vLLM

How to use Trelis/Llama-2-7b-chat-hf-function-calling-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Trelis/Llama-2-7b-chat-hf-function-calling-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Trelis/Llama-2-7b-chat-hf-function-calling-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M

SGLang

How to use Trelis/Llama-2-7b-chat-hf-function-calling-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Trelis/Llama-2-7b-chat-hf-function-calling-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Trelis/Llama-2-7b-chat-hf-function-calling-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Trelis/Llama-2-7b-chat-hf-function-calling-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Trelis/Llama-2-7b-chat-hf-function-calling-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use Trelis/Llama-2-7b-chat-hf-function-calling-v2 with Ollama:
```
ollama run hf.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Trelis/Llama-2-7b-chat-hf-function-calling-v2 with Docker Model Runner:
```
docker model run hf.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M
```

Lemonade

How to use Trelis/Llama-2-7b-chat-hf-function-calling-v2 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Trelis/Llama-2-7b-chat-hf-function-calling-v2:Q3_K_M

Run and chat with the model

lemonade run user.Llama-2-7b-chat-hf-function-calling-v2-Q3_K_M

List all available models

lemonade list

Atomic Chat

RonanMcGovern commited on Nov 24, 2023

Commit

055776a

1 Parent(s): cf55c8a

add runpod inference notes

Browse files

Files changed (1) hide show

README.md +19 -3

README.md CHANGED Viewed

@@ -73,10 +73,10 @@ The dataset used for training this model can be found at [Trelis Function Callin
 !!! Make sure to check the prompt format below and adjust inference accordingly !!!
-**Quick Start in Google Colab**
 Try out this notebook [fLlama_Inference notebook](https://colab.research.google.com/drive/1Ow5cQ0JNv-vXsT-apCceH6Na3b4L7JyW?usp=sharing)
-**Commercial Applications**
 You can this model with [text-generation-interface](https://github.com/huggingface/text-generation-inference) and [chat-ui](https://github.com/huggingface/chat-ui)
 Here is the [github for setup](https://github.com/TrelisResearch/tgi-chat-ui-function-calling)
@@ -85,7 +85,23 @@ And here is a video showing it working with [llama-2-7b-chat-hf-function-calling
 Note that you'll still need to code the server-side handling of making the function calls (which obviously depends on what functions you want to use).
-**Run on your laptop**
 Run on your laptop [video and juypter notebook](https://youtu.be/nDJMHFsBU7M)
 After running llama.cpp server, you can call the server with this command, with thanks to @jdo300:

 !!! Make sure to check the prompt format below and adjust inference accordingly !!!
+### Quick Start in Google Colab
 Try out this notebook [fLlama_Inference notebook](https://colab.research.google.com/drive/1Ow5cQ0JNv-vXsT-apCceH6Na3b4L7JyW?usp=sharing)
+### Text Generation Inference
 You can this model with [text-generation-interface](https://github.com/huggingface/text-generation-inference) and [chat-ui](https://github.com/huggingface/chat-ui)
 Here is the [github for setup](https://github.com/TrelisResearch/tgi-chat-ui-function-calling)
 Note that you'll still need to code the server-side handling of making the function calls (which obviously depends on what functions you want to use).
+#### Runpod Quickstart
+For a quickstart with runpod, you can use this template: [here](https://runpod.io/gsc?template=edxvuji38p&ref=jmfkcdio)
+Once up and running, you can make queries to:
+```
+https://{YOUR_POD_ID}-8080.proxy.runpod.net
+```
+Then, you can make queries to the api as follows:
+```
+curl https://{YOUR_POD_ID}-8080.proxy.runpod.net/generate \
+    -X POST \
+    -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}' \
+    -H 'Content-Type: application/json'
+```
+Or use /generate_stream for streaming. You can also write python scripts and use python to make requests. More info from the text-generation-inference [github repo](https://github.com/huggingface/text-generation-inference/)
+### Run on your laptop
 Run on your laptop [video and juypter notebook](https://youtu.be/nDJMHFsBU7M)
 After running llama.cpp server, you can call the server with this command, with thanks to @jdo300: