Error response body not sent if upload is incomplete

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Error response body not sent if upload is incomplete

sonpg
This happens using the ngx_http_uwsgi_module, but it seems this might be
more generic (i.e. also affects at least upstream servers).

Here's what happens:
 * I send a HTTP/1.1 POST request with a Content-Type: multipart/form-data;
header and a ~600kb file
 * Nginx receives the first part of the request and passes it to a uwsgi
app
 * The uwsgi app determines that a 403 response along with a JSON body
should be returned
 * Nginx sends the 403 response to the client, but only containing the
headers (not the JSON body)

However, if I do everything the same way, but the uploaded file is tiny
(e.g. 1 byte), I do get the error response body as expected. Non-error
responses also work fine.

It seems that nginx for some reason decides to ignore the response body (but
still sends the headers) if the payload hasn't finished uploading.

This looks like an inconsistent behaviour (or even a bug), but correct me
know if there is something I misunderstood.

Please find curl outputs and links to other users complaining about a
similar thing below.


Here's curl verbose output when uploading a bigger file:
$ curl -v -F 'content=@large_file' http://0.0.0.0:5000/
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Connected to 0.0.0.0 (127.0.0.1) port 5000 (#0)
> POST / HTTP/1.1
> Host: 0.0.0.0:5000
> User-Agent: curl/7.55.1
> Accept: */*
> Content-Length: 654430
> Expect: 100-continue
> Content-Type: multipart/form-data;
boundary=------------------------6404e93291dc3c9f
>
< HTTP/1.1 100 Continue
< HTTP/1.1 403 FORBIDDEN
< Server: nginx/1.9.11
< Date: Fri, 29 Dec 2017 19:41:57 GMT
< Content-Type: application/json
< Content-Length: 54
< Connection: keep-alive
* HTTP error before end of send, stop sending
<
* transfer closed with 54 bytes remaining to read
* Closing connection 0
curl: (18) transfer closed with 54 bytes remaining to read

And this is curl output with the smaller fine (this is what I would expect
independently of the payload size):
$ curl -v -F 'content=@tiny_file' http://0.0.0.0:5000/
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Connected to 0.0.0.0 (127.0.0.1) port 5000 (#0)
> POST / HTTP/1.1
> Host: 0.0.0.0:5000
> User-Agent: curl/7.55.1
> Accept: */*
> Content-Length: 205
> Expect: 100-continue
> Content-Type: multipart/form-data;
boundary=------------------------8cc5b005486613a4
>
< HTTP/1.1 100 Continue
< HTTP/1.1 403 FORBIDDEN
< Server: nginx/1.9.11
< Date: Fri, 29 Dec 2017 20:12:41 GMT
< Content-Type: application/json
< Content-Length: 54
< Connection: keep-alive
* HTTP error before end of send, stop sending
<
* Closing connection 0
{"error": {"message": "Invalid key", "code": 403}}


Other users reporting similar behaviour:
https://stackoverflow.com/questions/32208360/return-a-body-through-nginx-when-theres-an-error-mid-post
https://stackoverflow.com/questions/34771225/nginx-http-error-before-end-of-send

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277935,277935#msg-277935

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Error response body not sent if upload is incomplete

Valentin V. Bartenev-3
On Friday, 29 December 2017 23:23:07 MSK naktinis wrote:

> This happens using the ngx_http_uwsgi_module, but it seems this might be
> more generic (i.e. also affects at least upstream servers).
>
> Here's what happens:
>  * I send a HTTP/1.1 POST request with a Content-Type: multipart/form-data;
> header and a ~600kb file
>  * Nginx receives the first part of the request and passes it to a uwsgi
> app
>  * The uwsgi app determines that a 403 response along with a JSON body
> should be returned
>  * Nginx sends the 403 response to the client, but only containing the
> headers (not the JSON body)
>
> However, if I do everything the same way, but the uploaded file is tiny
> (e.g. 1 byte), I do get the error response body as expected. Non-error
> responses also work fine.
>
> It seems that nginx for some reason decides to ignore the response body (but
> still sends the headers) if the payload hasn't finished uploading.
>
> This looks like an inconsistent behaviour (or even a bug), but correct me
> know if there is something I misunderstood.
>
> Please find curl outputs and links to other users complaining about a
> similar thing below.
[..]

What's in the error log?

The error log from one of your links suggests, that the problme in
uwsgi server, not in nginx:

2015/08/25 15:28:49 [error] 10#0: *103 readv() failed (104: Connection reset by peer) while reading upstream

  wbr, Valentin V. Bartenev

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Error response body not sent if upload is incomplete

sonpg
Find the server logs below. It does seem to match what you've quoted.

Do you think the upstream server (uwsgi) is the one not returning the body?

I was able to fix this by consuming the request body in my application
before returning the response. However, I'm still wondering how nginx is
supposed to behave in such situations.

Log for the request with the larger file:
api-server_1   | [pid: 21|app: 0|req: 1/1] 172.19.0.1 () {38 vars in 601
bytes} [Sat Dec 30 11:14:46 2017] POST / => generated 54 bytes in 2 msecs
(HTTP/1.1 403) 2 headers in 78 bytes (1 switches on core 0)
api-server_1   | 2017/12/30 11:14:46 [error] 12#12: *1 readv() failed (104:
Connection reset by peer) while reading upstream, client: 172.19.0.1,
server: , request: "POST / HTTP/1.1", upstream:
"uwsgi://unix:///tmp/uwsgi.sock:", host: "0.0.0.0:5000"
api-server_1   | 172.19.0.1 - - [30/Dec/2017:11:14:46 +0000] "POST /
HTTP/1.1" 403 25 "-" "curl/7.55.1" "-"

With the smaller file:
api-server_1   | 172.19.0.1 - - [30/Dec/2017:11:15:41 +0000] "POST /
HTTP/1.1" 403 79 "-" "curl/7.55.1" "-"
api-server_1   | [pid: 20|app: 0|req: 3/4] 172.19.0.1 () {38 vars in 595
bytes} [Sat Dec 30 11:15:41 2017] POST / => generated 54 bytes in 1 msecs
(HTTP/1.1 403) 2 headers in 78 bytes (1 switches on core 0)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,277935,277950#msg-277950

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Error response body not sent if upload is incomplete

Maxim Dounin
Hello!

On Sat, Dec 30, 2017 at 06:04:26PM -0500, naktinis wrote:

> Find the server logs below. It does seem to match what you've quoted.
>
> Do you think the upstream server (uwsgi) is the one not returning the body?
>
> I was able to fix this by consuming the request body in my application
> before returning the response. However, I'm still wondering how nginx is
> supposed to behave in such situations.
>
> Log for the request with the larger file:
> api-server_1   | [pid: 21|app: 0|req: 1/1] 172.19.0.1 () {38 vars in 601
> bytes} [Sat Dec 30 11:14:46 2017] POST / => generated 54 bytes in 2 msecs
> (HTTP/1.1 403) 2 headers in 78 bytes (1 switches on core 0)
> api-server_1   | 2017/12/30 11:14:46 [error] 12#12: *1 readv() failed (104:
> Connection reset by peer) while reading upstream, client: 172.19.0.1,
> server: , request: "POST / HTTP/1.1", upstream:
> "uwsgi://unix:///tmp/uwsgi.sock:", host: "0.0.0.0:5000"
> api-server_1   | 172.19.0.1 - - [30/Dec/2017:11:14:46 +0000] "POST /
> HTTP/1.1" 403 25 "-" "curl/7.55.1" "-"
>
> With the smaller file:
> api-server_1   | 172.19.0.1 - - [30/Dec/2017:11:15:41 +0000] "POST /
> HTTP/1.1" 403 79 "-" "curl/7.55.1" "-"
> api-server_1   | [pid: 20|app: 0|req: 3/4] 172.19.0.1 () {38 vars in 595
> bytes} [Sat Dec 30 11:15:41 2017] POST / => generated 54 bytes in 1 msecs
> (HTTP/1.1 403) 2 headers in 78 bytes (1 switches on core 0)

The "readv() failed (104: Connection reset by peer)" error
indicate that there is a backend problem which makes it impossible
to reliably receive the body of the response.  To make it possible
for nginx to receive the body, backend must either read the whole
body, or implement proper connection teardown (in nginx, this is
called lingering_close, see http://nginx.org/r/lingering_close).

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx