# Deploying inference model size and performance

**URL:** <https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575>\
**Category:** 🤗Transformers\
**Created:** [February 10, 2021, 3:06am UTC](https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575 "2021-02-10T03:06:08Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![SMB](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@SMB](https://discuss.huggingface.co/u/SMB)\
**Post date:** [February 10, 2021, 3:06am UTC](https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575/1 "2021-02-10T03:06:08Z")

</div>

I’ve got a trained/tuned model based on `Michau/t5-base-en-generate-headline`. I’m looking into options for deploying this model around a simple inference API (Python/Flask). I’m very new to developing and deploying ML models etc. so bear with me!

Though I have it working, the performance it less than optimal. In a local development environment (VM + Docker) each request takes ~30" (compared to ~10" in Collab - no GPU). In a production environment, this is going to be run many 1,000’s of times, daily…ideally.

So far my trained model (`pytorch_model.bin`) is ~900 MB. I do inference pretty simply:

```auto
MODEL_PATH = "/src/model_files"

def infer(title: str) -> str:
    model_path = pathlib.Path(MODEL_PATH).absolute()
    title_model_tokenizer = AutoTokenizer.from_pretrained(model_path)
    title_model = AutoModelWithLMHead.from_pretrained(model_path)

    tokenized_text = title_model_tokenizer.encode(title, return_tensors="pt")
    title_ids = title_model.generate(
        tokenized_text,
        num_beams=1,
        repetition_penalty=1.0,
        length_penalty=1.0,
        early_stopping=False,
        no_repeat_ngram_size=1
    )

    return title_model_tokenizer.decode(title_ids[0], skip_special_tokens=True)

```

I can see from some profiling that the majority of the time is spent on:

```auto
title_model = AutoModelWithLMHead.from_pretrained(model_path)

```

So change #1 is to send in batches rather than one at a time, where possible. So as to not have to reload the model constantly.

Is there anything obvious I’m missing or yet to discover for this type of thing? I’m hoping to not need a GPU, so any ideas or improvement you can throw at me would be appreciated, thanks.

A secondary question is where would be suitable to deploy this kind of thing? Is it something that would be better outsourced to Sagemaker or similar? Or is it reasonable to host it on our own servers (specs notwithstanding)?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [February 10, 2021, 9:17am UTC](https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575/2 "2021-02-10T09:17:46Z")

</div>

Hi @SMB, I think a quick win here would be to load the tokenizer and model just once when you spin up the Flask app and then call them in your `infer` function. For example something like this:

```auto
from flask import Flask
from transformers import AutoTokenizer, AutoModelWithLMHead

app = Flask( __name__ )

MODEL_PATH = "/src/model_files"
model_path = pathlib.Path(MODEL_PATH).absolute()
title_model_tokenizer = AutoTokenizer.from_pretrained(model_path)
title_model = AutoModelWithLMHead.from_pretrained(model_path).to("cpu")

def infer(title: str) -> str:
    tokenized_text = title_model_tokenizer.encode(title, return_tensors="pt")
    title_ids = title_model.generate(
        tokenized_text,
        num_beams=1,
        repetition_penalty=1.0,
        length_penalty=1.0,
        early_stopping=False,
        no_repeat_ngram_size=1
    )

    return title_model_tokenizer.decode(title_ids[0], skip_special_tokens=True)

```

As for deployment, the answer probably depends on your production environment but what I’ve usually done is to Dockerize the Flask app and then deploy the Docker container on Kubernetes or whatever platform is available (e.g. Heroku). There’s quite a few tutorials online on how to do this and one of my favourite resources for this kind of stuff is the Full Stack DL course: [https://fullstackdeeplearning.com/](https://fullstackdeeplearning.com/)

Note that Kubernetes is a complex beast and not recommended for a single-purpose app 🙂

PS. If you’re not bound to using Flask, I suggest having a look at [FastAPI](https://fastapi.tiangolo.com/). It makes web app development much simpler, can handle concurrency, and comes with neat built in features like data validation which is really useful for ML!

---

<div class="post-metadata">

**Author:** ![SMB](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@SMB](https://discuss.huggingface.co/u/SMB)\
**Post date:** [February 10, 2021, 4:20pm UTC](https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575/3 "2021-02-10T16:20:13Z")

</div>

Thanks @lewtun that’s an obvious change I can make that will be a quick-win as you say. Thanks also for the deployment suggestions, I’m familiar with Heroku, but will look at the other options also, and FastAPI, cheers.

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [February 10, 2021, 8:36pm UTC](https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575/4 "2021-02-10T20:36:28Z")

</div>

In case this still doesn’t meet your latency requirements, my next suggestion would be to quantize the model’s weights: [Quantization — PyTorch 1.7.1 documentation](https://pytorch.org/docs/stable/quantization.html)

It basically amounts to running this line of code

```auto
model_int8 = torch.quantization.quantize_dynamic(
    model, # the original model
    {torch.nn.Linear}, # a set of layers to dynamically quantize
    dtype=torch.qint8) # the target dtype for quantized weights

```

but you should do some sanity checks to make sure the resulting accuracy hasn’t degraded too much (in which case you can try 16-bit precision).

If you want to take it further, you can try serialising the model with ONNX (although this can be a bit finnicky): [Exporting transformers models — transformers 4.3.0 documentation](https://huggingface.co/transformers/serialization.html)

---

<div class="post-metadata">

**Author:** ![SMB](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@SMB](https://discuss.huggingface.co/u/SMB)\
**Post date:** [March 6, 2021, 2:28am UTC](https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575/5 "2021-03-06T02:28:12Z")

</div>

Revisiting this @lewtun You’re comments proved very helpful in putting me on the right path. I’ve moved to FastAPI and have some questions about concurrency.

I’m testing locally and running within a docker container using the command:

`gunicorn app:app --worker-class uvicorn.workers.UvicornWorker --workers 2 --bind 0.0.0.0:8080`

When I run 10 requests (each containing 50 pieces of text to run inference on) from two separate PHP workers (run by supervisor) that call `http//0.0.0.0:8080/predict/batch` response times are:

```auto
app_1 | 0:00:37.766941
app_1 | 0:00:38.065883
app_1 | 0:00:35.876927
app_1 | 0:00:38.612610
app_1 | 0:00:38.268158
app_1 | 0:00:37.676483
app_1 | 0:00:36.451956
app_1 | 0:00:39.346421
app_1 | 0:00:32.831906
app_1 | 0:00:33.972420

```

Total time for queue of 10 requests to clear ~3’

When I run 10 requests from 10 PHP workers I get response times of:

```auto
app_1 | 0:02:57.640630
app_1 | 0:03:09.019433
app_1 | 0:03:13.338791
app_1 | 0:03:16.411248
app_1 | 0:03:16.476156
app_1 | 0:03:20.084982
app_1 | 0:03:21.052187
app_1 | 0:03:23.064294
app_1 | 0:03:23.681471
app_1 | 0:03:23.975776

```

So you can see first request returned at ~2’57", whilst #10 returned after ~3’23"

So, really no different. Or, rather, when using 10 workers it appears they are not really running concurrently?

My code is essentially now:

```auto
import pathlib
import uvicorn
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelWithLMHead
from typing import List
from datetime import datetime

title_model = None
title_model_tokenizer = None

app = FastAPI()

class TitleIn(BaseModel):
    id: int
    text: str

class TitleOut(BaseModel):
    id: int
    text: str

def infer(title: str) -> str:
    tokenized_text = title_model_tokenizer.encode(title, return_tensors="pt")
    title_ids = title_model.generate(
        tokenized_text,
        num_beams=1,
        repetition_penalty=1.0,
        length_penalty=1.0,
        early_stopping=False,
        no_repeat_ngram_size=1
    )

    return title_model_tokenizer.decode(title_ids[0], skip_special_tokens=True)

@app.on_event("startup")
async def startup_event():
    global title_model, title_model_tokenizer

    model_path = "/src/model_files"
    title_model = AutoModelWithLMHead.from_pretrained(pathlib.Path(model_path).absolute()).to("cpu")
    title_model_tokenizer = AutoTokenizer.from_pretrained(pathlib.Path(model_path).absolute(), use_fast=False)

@app.post('/predict/batch', response_model=List[TitleOut])
def predict_batch(title_list: List[TitleIn]):
    batch_predictions = []

    startTime = datetime.now()

    for title in title_list:
        batch_predictions.append(TitleOut(id=title.id, text=infer(title.text)))

    print(datetime.now() - startTime)
    return batch_predictions

```

I feel like I’m missing something? Any further tips? Thank you.

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [March 6, 2021, 8:00am UTC](https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575/6 "2021-03-06T08:00:46Z")

</div>

Hi @SMB, one thing that might be causing trouble is that PyTorch will attempt to use all cores by default to process each request. You could try setting the number of threads to 1 in your `startup_event` function with

```auto
torch.set_num_threads(1)

```

See [here](https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html#tuning-the-number-of-threads) for more details. I’m not sure this _will_ solve your concurrency problem, but it’s a good place to start 🙂

---

<div class="post-metadata">

**Author:** ![Dev101LFG](https://avatars.discourse-cdn.com/v4/letter/d/f475e1/32.png) [@Dev101LFG](https://discuss.huggingface.co/u/Dev101LFG)\
**Post date:** [July 9, 2024, 3:11am UTC](https://discuss.huggingface.co/t/deploying-inference-model-size-and-performance/3575/7 "2024-07-09T03:11:46Z")

</div>

I’m currently using Flask but am considering switching to FastAPI based on this discussion.

@SMB @lewtun, are you aware of any limitations with using FastAPI on the free tier when running it via a Dockerfile?

I came across information suggesting that the inference API has a limit of 10,000 requests per month and a 2-second delay between requests. Can anyone confirm if this applies in our context?

Thanks