Error hosting endpoint when deploying model

ahsanr
Wed, 27 Mar 2024 20:15:09 +0000

how can I deploy a Quantized model on aws,
this model:

huggingface.co

vicuna-13b-v1.5.Q4_K_M.gguf · TheBloke/vicuna-13B-v1.5-GGUF at main

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

for this scenario:
Deploy this TheBloke/vicuna-13B-v1.5-GGUF model on AWS

I want to use this model as an endpoint in my web application in this format:

Chatbot Requirements

1. Scope: Chatbot (Encoder/Decoder for Text Inference or Conversational)

2. Input via API (JSON): Chatgpt Style – The template can be see belowThe JSON will contain 25 user messages, and the response should be the system response.
Please use this guidelines to understand API consumption: InvokeEndpoint - Amazon SageMaker

3. Prompt Template for the system:
a. template = ‘’’
You are going to be my education assistant.
System:{System}
Question:{question}‘’’

4. LLM Model Parameters: max_new_tokens=512, temperature=0.7, top_p=0.9

5. If possible use a AutomodelforCausalLM otherwise train a LLM model.

6. It will be deployed on AWS Sagemaker using S3 buckets.

7. The GGUF should be saved on a S3 Bucket.

8. Chat Buffer should store 25 conversations and create a session ID (No need to send this to the End point).

9. Use HuggingFace/Langchain when possible.

10. Deliverables: Jupyter notebook/Code – 2 Hours should be used to set up the model in AWS with the customer.

Provide me with complete source code that I can use in my jupyter notebook on aws to make an endpoint.