flood detected with file uploads over http2

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

flood detected with file uploads over http2

Jasper Wallace
We are having intermittent problems uploading files via nginx to a
flask backend over http2:

2019/12/16 16:07:08 [debug] 27658#27658: *1 event timer: 3, old:
1576512608187, new: 1576512608301
2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 idle handler
2019/12/16 16:07:08 [info] 27658#27658: *1 http2 flood detected while
processing HTTP/2 connection, client: x.x.x.x, server: 0.0.0.0:443
2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 send GOAWAY frame, status:0
2019/12/16 16:07:08 [debug] 27658#27658: *1 posix_memalign:
0000563642B8EE20:512 @16
2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 frame out:
0000563642B8EE40 sid:0 bl:0 len:8
2019/12/16 16:07:08 [debug] 27658#27658: *1 malloc: 0000563642D0A870:16384
2019/12/16 16:07:08 [debug] 27658#27658: *1 SSL buf copy: 17
2019/12/16 16:07:08 [debug] 27658#27658: *1 SSL to write: 17

Is there anyway of getting information on what might be triggering this?

We've changed some defaults:

client_max_body_size 10m;
http2_body_preread_size 256k;
http2_recv_buffer_size 1m;
proxy_headers_hash_max_size 512;
proxy_headers_hash_bucket_size 128;

Client is Chrome:

Version 78.0.3904.97 (Developer Build) built on Debian 10.1, running
on Debian 10.2 (64-bit)

openssl:

OpenSSL 1.1.0l  10 Sep 2019

nginx:

nginx version: nginx/1.10.3
built with OpenSSL 1.1.0k  28 May 2019 (running with OpenSSL 1.1.0l
10 Sep 2019)
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2
-fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now'
--prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf
--http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log
--lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug
--with-pcre-jit --with-ipv6 --with-http_ssl_module
--with-http_stub_status_module --with-http_realip_module
--with-http_auth_request_module --with-http_v2_module
--with-http_dav_module --with-http_slice_module --with-threads
--with-http_addition_module --with-http_flv_module
--with-http_geoip_module=dynamic --with-http_gunzip_module
--with-http_gzip_static_module --with-http_image_filter_module=dynamic
--with-http_mp4_module --with-http_perl_module=dynamic
--with-http_random_index_module --with-http_secure_link_module
--with-http_sub_module --with-http_xslt_module=dynamic
--with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic
--with-stream_ssl_module
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/headers-more-nginx-module
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-auth-pam
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-cache-purge
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-dav-ext-module
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-development-kit
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-echo
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/ngx-fancyindex
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nchan
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-lua
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-upload-progress
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-upstream-fair
--add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/ngx_http_substitutions_filter_module

--
Your hydrogen & fuel cell partner
Arcola Energy Ltd, 24 Ashwin Street,
London E8 3DL. www.arcolaenergy.com <https://www.arcolaenergy.com/> / +44
20 7503 1386
Registered in England and Wales, Company Number 7257863, VAT
Number 110085273. Copyright 2019. Confidential and Proprietary. Not to be
disseminated or copied in full or in part.
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: flood detected with file uploads over http2

Ruslan Ermilov
On Mon, Dec 16, 2019 at 05:45:55PM +0000, Jasper Wallace wrote:

> We are having intermittent problems uploading files via nginx to a
> flask backend over http2:
>
> 2019/12/16 16:07:08 [debug] 27658#27658: *1 event timer: 3, old:
> 1576512608187, new: 1576512608301
> 2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 idle handler
> 2019/12/16 16:07:08 [info] 27658#27658: *1 http2 flood detected while
> processing HTTP/2 connection, client: x.x.x.x, server: 0.0.0.0:443
> 2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 send GOAWAY frame, status:0
> 2019/12/16 16:07:08 [debug] 27658#27658: *1 posix_memalign:
> 0000563642B8EE20:512 @16
> 2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 frame out:
> 0000563642B8EE40 sid:0 bl:0 len:8
> 2019/12/16 16:07:08 [debug] 27658#27658: *1 malloc: 0000563642D0A870:16384
> 2019/12/16 16:07:08 [debug] 27658#27658: *1 SSL buf copy: 17
> 2019/12/16 16:07:08 [debug] 27658#27658: *1 SSL to write: 17
>
> Is there anyway of getting information on what might be triggering this?
>
> We've changed some defaults:
>
> client_max_body_size 10m;
> http2_body_preread_size 256k;
> http2_recv_buffer_size 1m;
> proxy_headers_hash_max_size 512;
> proxy_headers_hash_bucket_size 128;
>
> Client is Chrome:
>
> Version 78.0.3904.97 (Developer Build) built on Debian 10.1, running
> on Debian 10.2 (64-bit)
>
> openssl:
>
> OpenSSL 1.1.0l  10 Sep 2019
>
> nginx:
>
> nginx version: nginx/1.10.3
> built with OpenSSL 1.1.0k  28 May 2019 (running with OpenSSL 1.1.0l
> 10 Sep 2019)
> TLS SNI support enabled
> configure arguments: --with-cc-opt='-g -O2
> -fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=.
> -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
> -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now'
> --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf
> --http-log-path=/var/log/nginx/access.log
> --error-log-path=/var/log/nginx/error.log
> --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
> --modules-path=/usr/lib/nginx/modules
> --http-client-body-temp-path=/var/lib/nginx/body
> --http-fastcgi-temp-path=/var/lib/nginx/fastcgi
> --http-proxy-temp-path=/var/lib/nginx/proxy
> --http-scgi-temp-path=/var/lib/nginx/scgi
> --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug
> --with-pcre-jit --with-ipv6 --with-http_ssl_module
> --with-http_stub_status_module --with-http_realip_module
> --with-http_auth_request_module --with-http_v2_module
> --with-http_dav_module --with-http_slice_module --with-threads
> --with-http_addition_module --with-http_flv_module
> --with-http_geoip_module=dynamic --with-http_gunzip_module
> --with-http_gzip_static_module --with-http_image_filter_module=dynamic
> --with-http_mp4_module --with-http_perl_module=dynamic
> --with-http_random_index_module --with-http_secure_link_module
> --with-http_sub_module --with-http_xslt_module=dynamic
> --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic
> --with-stream_ssl_module
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/headers-more-nginx-module
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-auth-pam
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-cache-purge
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-dav-ext-module
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-development-kit
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-echo
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/ngx-fancyindex
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nchan
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-lua
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-upload-progress
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-upstream-fair
> --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/ngx_http_substitutions_filter_module

nginx/1.10.3 doesn't have HTTP/2 flood detection.  It appeared
in later versions.
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: flood detected with file uploads over http2

Jasper Wallace
Hmmm, maybe it got packported by Debian...

I think we'll just disable http2 for the time being.

On Tue, 17 Dec 2019 at 09:13, Ruslan Ermilov <[hidden email]> wrote:

>
> On Mon, Dec 16, 2019 at 05:45:55PM +0000, Jasper Wallace wrote:
> > We are having intermittent problems uploading files via nginx to a
> > flask backend over http2:
> >
> > 2019/12/16 16:07:08 [debug] 27658#27658: *1 event timer: 3, old:
> > 1576512608187, new: 1576512608301
> > 2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 idle handler
> > 2019/12/16 16:07:08 [info] 27658#27658: *1 http2 flood detected while
> > processing HTTP/2 connection, client: x.x.x.x, server: 0.0.0.0:443
> > 2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 send GOAWAY frame, status:0
> > 2019/12/16 16:07:08 [debug] 27658#27658: *1 posix_memalign:
> > 0000563642B8EE20:512 @16
> > 2019/12/16 16:07:08 [debug] 27658#27658: *1 http2 frame out:
> > 0000563642B8EE40 sid:0 bl:0 len:8
> > 2019/12/16 16:07:08 [debug] 27658#27658: *1 malloc: 0000563642D0A870:16384
> > 2019/12/16 16:07:08 [debug] 27658#27658: *1 SSL buf copy: 17
> > 2019/12/16 16:07:08 [debug] 27658#27658: *1 SSL to write: 17
> >
> > Is there anyway of getting information on what might be triggering this?
> >
> > We've changed some defaults:
> >
> > client_max_body_size 10m;
> > http2_body_preread_size 256k;
> > http2_recv_buffer_size 1m;
> > proxy_headers_hash_max_size 512;
> > proxy_headers_hash_bucket_size 128;
> >
> > Client is Chrome:
> >
> > Version 78.0.3904.97 (Developer Build) built on Debian 10.1, running
> > on Debian 10.2 (64-bit)
> >
> > openssl:
> >
> > OpenSSL 1.1.0l  10 Sep 2019
> >
> > nginx:
> >
> > nginx version: nginx/1.10.3
> > built with OpenSSL 1.1.0k  28 May 2019 (running with OpenSSL 1.1.0l
> > 10 Sep 2019)
> > TLS SNI support enabled
> > configure arguments: --with-cc-opt='-g -O2
> > -fdebug-prefix-map=/build/nginx-DhOtPd/nginx-1.10.3=.
> > -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
> > -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now'
> > --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf
> > --http-log-path=/var/log/nginx/access.log
> > --error-log-path=/var/log/nginx/error.log
> > --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
> > --modules-path=/usr/lib/nginx/modules
> > --http-client-body-temp-path=/var/lib/nginx/body
> > --http-fastcgi-temp-path=/var/lib/nginx/fastcgi
> > --http-proxy-temp-path=/var/lib/nginx/proxy
> > --http-scgi-temp-path=/var/lib/nginx/scgi
> > --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug
> > --with-pcre-jit --with-ipv6 --with-http_ssl_module
> > --with-http_stub_status_module --with-http_realip_module
> > --with-http_auth_request_module --with-http_v2_module
> > --with-http_dav_module --with-http_slice_module --with-threads
> > --with-http_addition_module --with-http_flv_module
> > --with-http_geoip_module=dynamic --with-http_gunzip_module
> > --with-http_gzip_static_module --with-http_image_filter_module=dynamic
> > --with-http_mp4_module --with-http_perl_module=dynamic
> > --with-http_random_index_module --with-http_secure_link_module
> > --with-http_sub_module --with-http_xslt_module=dynamic
> > --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic
> > --with-stream_ssl_module
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/headers-more-nginx-module
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-auth-pam
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-cache-purge
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-dav-ext-module
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-development-kit
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-echo
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/ngx-fancyindex
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nchan
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-lua
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-upload-progress
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/nginx-upstream-fair
> > --add-dynamic-module=/build/nginx-DhOtPd/nginx-1.10.3/debian/modules/ngx_http_substitutions_filter_module
>
> nginx/1.10.3 doesn't have HTTP/2 flood detection.  It appeared
> in later versions.
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

--
Your hydrogen & fuel cell partner
Arcola Energy Ltd, 24 Ashwin Street,
London E8 3DL. www.arcolaenergy.com <https://www.arcolaenergy.com/> / +44
20 7503 1386
Registered in England and Wales, Company Number 7257863, VAT
Number 110085273. Copyright 2019. Confidential and Proprietary. Not to be
disseminated or copied in full or in part.
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx