Q:

what causes a large upstream_header_time in nginx?

Setup:
Angular / Django / Nginx in ECS.

Route 53 -> ELB -> Target group ECS

With Nginx proxying /api | /admin calls to Django.

We have a large lag or delay on some of our API requests with the following log being shown: (Line broken for easier reading)
timestamp="16/Jul/2019:16:24:43 +0000" 
client=10.96.192.168 
request="GET /api/redacted/?
redacted=0&
redacted=52.895649&
redacted=53.226591&
redacted=-2.900391&
redacted=-1.873169 HTTP/1.1" 
request_length=919 
bytes_sent=200514 
body_bytes_sent=200245 
referer=https://redacted.redacted.redacted.redacted.redacted/redacted 
user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" 
upstream_addr=127.0.0.1:8000 
upstream_status=200 
request_time=19.077 
upstream_response_time=19.064 
upstream_connect_time=0.000 
upstream_header_time=19.064 
app_db_read_time=- 
app_db_write_time=- 
app_analysis_time=- 
app_other_time=-

what has caught our eye as the main delay is the upstream_header_time and upstream_response_time being around 20 seconds!.
However being new to Nginx I have no idea where to start to debug this, working locally everything works as expected with no delay like this.
Question: What is causing this large 20 second upstream header / response time?
Config File: https://gist.github.com/jackdh/4fdffe6b11799f544d7cb211cc66bfe2
Thanks

A:

Per the NGINX docs:
$upstream_header_timekeeps time spent on receiving the response header from the upstream server (1.7.10); the time is kept in seconds with millisecond resolution. Times of several responses are separated by commas and colons like addresses in the $upstream_addr variable.

So that's your Python backend taking forever to answer.
http://nginx.org/en/docs/http/ngx_http_upstream_module.html

