# Execute finetuned QA model in parallel **URL:** \ **Category:** Intermediate\ **Created:** [September 29, 2022, 2:25am UTC](https://discuss.huggingface.co/t/execute-finetuned-qa-model-in-parallel/23758 "2022-09-29T02:25:13Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Madhanlal](https://avatars.discourse-cdn.com/v4/letter/m/e47774/32.png) [@Madhanlal](https://discuss.huggingface.co/u/Madhanlal)\ **Post date:** [September 29, 2022, 2:25am UTC](https://discuss.huggingface.co/t/execute-finetuned-qa-model-in-parallel/23758/1 "2022-09-29T02:25:13Z")
Hi I am trying to build a QA model which gets a question from front end and pass it backs the answers. The latency time is 95 seconds right now which is very high for the online application. Internally, my backend code fetches a maximum of 100 contexts related to the user’s question from a database and fetches the answers. I would take top 20 best answers based on business logic. My current logic is looping thru each context: answers = for context in all\_context: answer = call qamodel (question,context) answers.append(answer) loaded\_mdl = torch.load(“/location/finetuned\_model.bin”) def qamodel(question,context): answers, \_ = loaded\_mdl.predict(to\_predict) return answers[0] The latency of this QA model alone is 90 seconds out of total 95 seconds. I tried to call this qamodel in threads so parallel processing can occur there by reducing the latency time. But the latency time increased from 90 seconds to 240 seconds. The code I tried: def parallel\_qa(context): answer = qamodel(context) return [answer] ``` result = [] with ThreadPoolExecutor(max_workers=20) as exe: print ("calling ThreadPoolExecutor") result = exe.map(parallel_qa,all_contexts) ``` How can I achieve multi processing to reduce the latency time? Is there any other recommended approach to reduce latency time? Any help appreciated. Thanks.