Extremely slow file (~5MB) upload via POST

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Extremely slow file (~5MB) upload via POST

ramirezc
Hi everyone

I'm new here, and i've searched if the problem appeared before but couldn't
find anything useful.

[DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1,
openssl-1.1.1) which allow people upload files from their browser. The files
are simply stored on disk. Nothing else is performed on them.

[SYSTEM CONFIG]
. Linux 4.15.0 Ubuntu 18.04 LTS SMP x86_64
. RAM: 32GB
. CPU: 8-Cores Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
. DISK: 1TB SSD
. NETWORK CARD: 10Gbps
. System is never under load. We usually upload 10 files per hour at max.

[DATA CONFIG]
. File size is between 5MB to 20MB

[NGINX CONFIG]

We are running Nginx 1.16.1 with TLSv1.3 support (built on openssl 1.1.1).

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

$ cat /etc/nginx/nginx.conf
worker_processes auto;
worker_rlimit_nofile 100000;
pid /run/nginx.pid;

error_log  off; #/var/log/nginx/error.log  info;

events {
        worker_connections 655350;
        multi_accept           on;
        use                 epoll;
}

http {
      include       mime.types;
      default_type  application/octet-stream;

      server_tokens off;

      keepalive_timeout 3600;

      access_log    off; #/var/log/nginx/access.log;
      sendfile      on;
      tcp_nopush    on;
      tcp_nodelay   on;

      types_hash_max_size 2048;

      open_file_cache          max=10000 inactive=10m;
      open_file_cache_valid    1h;
      open_file_cache_min_uses 1;
      open_file_cache_errors   on;
      include /etc/nginx/conf.d/*.conf;

}

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

$ cat /etc/nginx/conf.d/uploader.conf
server {
        listen  443 ssl;

        server_name  BACKEND_HOST_NAME:

        ssl_certificate        /etc/nginx/certs/bundle.pem;
        ssl_certificate_key /etc/nginx/certs/key.pem;
        ssl_dhparam         /etc/nginx/certs/dh.pem; ## 2048-bit

        ssl_protocols       TLSv1.2 TLSv1.3;

        ssl_prefer_server_ciphers on;
        ssl_ciphers
ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;

        client_max_body_size 30m;

        location / {
           proxy_pass <a href="http://127.0.0.1:7777;">http://127.0.0.1:7777;
        }
}

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

[PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.
When uploading it directly to the upstream backend, it takes ~400 millisec
at max.

Running strace, we've got this:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 47.96    0.041738          11      3887      1489 read
 21.73    0.018909          13      1509           epoll_wait
 17.95    0.015622          22       708           writev
 10.62    0.009241          13       712           write
  0.47    0.000407          19        21        21 connect
  0.39    0.000338          17        20           close
  0.21    0.000180           8        22           epoll_ctl
  0.20    0.000173           8        21           socket
  0.13    0.000110         110         1           accept4
  0.11    0.000095           5        21           getsockopt
  0.10    0.000091           4        21           recvfrom
  0.07    0.000060           3        21           ioctl
  0.04    0.000037          12         3           brk
  0.03    0.000023           8         3           setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00    0.087024                  6970      1510 total

A lot of errors in "read" calls: 1489 errors. They all correspond to (thanks
again to strace):

22807 read(3,
"\26\3\1\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205\360\322\35\237_\240"...,
16709) = 517
22807 read(3, 0x559de2a23f03, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\24\3\3\0\1\1\26\3\3\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205"...,
16709) = 523
22807 read(3, 0x559de2a23f03, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3, "\27\3\3\0E\271m'\306\262\26X\36J\25lC/\202_7\241\32\342XN
\357\303%\264\0"..., 16709) = 74
22807 read(3,
"\27\3\3\0\245\240\204\304KJ\260\207\301\232\3147\217\357I$\243\266p+*\343L\335\6v\276\323"...,
16709) = 478
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0\32`\324\324\237\v\266n\300x\24\277\357z\374)\365\260F\235\24\346#A%\300\376",
16709) = 31
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0\177\310*W\352\265\230\357\325\177\302\275\357=\246`\246^\372\214T\206\264b\352;\273z"...,
16709) = 814
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0Y\330\276PNY\220\245\254E\0066\2016\355\334\237Yo\2510\253\320+\26z\342\275"...,
16709) = 644
22807 read(3, 0x559de2a229e3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3, "\27\3\3\0Z
\237j\230\f\331\222\246\325\1\272Y]\252\255%\31\257L\25\10\226\267
\253\353\367"..., 16709) = 285
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0\212\216j6\256\370\367\310\366Hjs\275r\276>\217\216\374\377a\375\363\4\2yr\23"...,
16709) = 176
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0\227K2\345P\200Ls\234\10\230f\362\221\273\270V\2371X\261|\245\315\240B\177\224"...,
16709) = 1717
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3>\232\344\316\245i\375hM\362\376\frr\340\21umx&\3311\373}\35\4\3069`"...,
16709) = 4380
22807 read(3, 0x559de2a1fabf, 11651)    = -1 EAGAIN (Resource temporarily
unavailable)


We tried to tune our Nginx's config, but the result is always the same:
22807 read(3, 0x559de2a1fabf, 11651)    = -1 EAGAIN (Resource temporarily
unavailable)


Help appreciated

/F.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286161,286161#msg-286161

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Extremely slow file (~5MB) upload via POST

Andrei
Bit off-topic, but, if you really want to improve the performance and ditch an upstream service which just takes in file uploads, you can do it directly in nginx with some Lua. For example https://www.yanxurui.cc/posts/server/2017-03-21-NGINX-as-a-file-server/. I used this method for large sites (20k online) and it worked way better than having to pass the file to a backend for saving.

There are many other articles on "nginx file server" or "nginx image server" with details on how to also process images. Hope this helps.

On Sun, Nov 10, 2019, 20:03 frank.muller <[hidden email]> wrote:
Hi everyone

I'm new here, and i've searched if the problem appeared before but couldn't
find anything useful.

[DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1,
openssl-1.1.1) which allow people upload files from their browser. The files
are simply stored on disk. Nothing else is performed on them.

[SYSTEM CONFIG]
. Linux 4.15.0 Ubuntu 18.04 LTS SMP x86_64
. RAM: 32GB
. CPU: 8-Cores Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
. DISK: 1TB SSD
. NETWORK CARD: 10Gbps
. System is never under load. We usually upload 10 files per hour at max.

[DATA CONFIG]
. File size is between 5MB to 20MB

[NGINX CONFIG]

We are running Nginx 1.16.1 with TLSv1.3 support (built on openssl 1.1.1).

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

$ cat /etc/nginx/nginx.conf
worker_processes auto;
worker_rlimit_nofile 100000;
pid /run/nginx.pid;

error_log  off; #/var/log/nginx/error.log  info;

events {
        worker_connections 655350;
        multi_accept           on;
        use                 epoll;
}

http {
      include       mime.types;
      default_type  application/octet-stream;

      server_tokens off;

      keepalive_timeout 3600;

      access_log    off; #/var/log/nginx/access.log;
      sendfile      on;
      tcp_nopush    on;
      tcp_nodelay   on;

      types_hash_max_size 2048;

      open_file_cache          max=10000 inactive=10m;
      open_file_cache_valid    1h;
      open_file_cache_min_uses 1;
      open_file_cache_errors   on;
      include /etc/nginx/conf.d/*.conf;

}

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

$ cat /etc/nginx/conf.d/uploader.conf
server {
        listen  443 ssl;

        server_name  BACKEND_HOST_NAME:

        ssl_certificate        /etc/nginx/certs/bundle.pem;
        ssl_certificate_key /etc/nginx/certs/key.pem;
        ssl_dhparam         /etc/nginx/certs/dh.pem; ## 2048-bit

        ssl_protocols       TLSv1.2 TLSv1.3;

        ssl_prefer_server_ciphers on;
        ssl_ciphers
ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;

        client_max_body_size 30m;

        location / {
           proxy_pass http://127.0.0.1:7777;
        }
}

-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8-8

[PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.
When uploading it directly to the upstream backend, it takes ~400 millisec
at max.

Running strace, we've got this:
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 47.96    0.041738          11      3887      1489 read
 21.73    0.018909          13      1509           epoll_wait
 17.95    0.015622          22       708           writev
 10.62    0.009241          13       712           write
  0.47    0.000407          19        21        21 connect
  0.39    0.000338          17        20           close
  0.21    0.000180           8        22           epoll_ctl
  0.20    0.000173           8        21           socket
  0.13    0.000110         110         1           accept4
  0.11    0.000095           5        21           getsockopt
  0.10    0.000091           4        21           recvfrom
  0.07    0.000060           3        21           ioctl
  0.04    0.000037          12         3           brk
  0.03    0.000023           8         3           setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00    0.087024                  6970      1510 total

A lot of errors in "read" calls: 1489 errors. They all correspond to (thanks
again to strace):

22807 read(3,
"\26\3\1\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205\360\322\35\237_\240"...,
16709) = 517
22807 read(3, 0x559de2a23f03, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\24\3\3\0\1\1\26\3\3\2\0\1\0\1\374\3\3\304\353\3\333\314\0\36\223\244z\246\322n\375\205"...,
16709) = 523
22807 read(3, 0x559de2a23f03, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3, "\27\3\3\0E\271m'\306\262\26X\36J\25lC/\202_7\241\32\342XN
\357\303%\264\0"..., 16709) = 74
22807 read(3,
"\27\3\3\0\245\240\204\304KJ\260\207\301\232\3147\217\357I$\243\266p+*\343L\335\6v\276\323"...,
16709) = 478
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0\32`\324\324\237\v\266n\300x\24\277\357z\374)\365\260F\235\24\346#A%\300\376",
16709) = 31
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0\177\310*W\352\265\230\357\325\177\302\275\357=\246`\246^\372\214T\206\264b\352;\273z"...,
16709) = 814
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0Y\330\276PNY\220\245\254E\0066\2016\355\334\237Yo\2510\253\320+\26z\342\275"...,
16709) = 644
22807 read(3, 0x559de2a229e3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3, "\27\3\3\0Z
\237j\230\f\331\222\246\325\1\272Y]\252\255%\31\257L\25\10\226\267
\253\353\367"..., 16709) = 285
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0\212\216j6\256\370\367\310\366Hjs\275r\276>\217\216\374\377a\375\363\4\2yr\23"...,
16709) = 176
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3\0\227K2\345P\200Ls\234\10\230f\362\221\273\270V\2371X\261|\245\315\240B\177\224"...,
16709) = 1717
22807 read(3, 0x559de2a1e9a3, 16709)    = -1 EAGAIN (Resource temporarily
unavailable)
22807 read(3,
"\27\3\3>\232\344\316\245i\375hM\362\376\frr\340\21umx&\3311\373}\35\4\3069`"...,
16709) = 4380
22807 read(3, 0x559de2a1fabf, 11651)    = -1 EAGAIN (Resource temporarily
unavailable)


We tried to tune our Nginx's config, but the result is always the same:
22807 read(3, 0x559de2a1fabf, 11651)    = -1 EAGAIN (Resource temporarily
unavailable)


Help appreciated

/F.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286161,286161#msg-286161

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Extremely slow file (~5MB) upload via POST

Francis Daly
In reply to this post by ramirezc
On Sun, Nov 10, 2019 at 01:03:20PM -0500, frank.muller wrote:

Hi there,

I don't have an answer for you, but there are some things you could
perhaps try, if you are happy to keep investigating.

> [DESCRIPTION] I've an upstream backend service behind Niginx (1.16.1,
> openssl-1.1.1) which allow people upload files from their browser. The files
> are simply stored on disk. Nothing else is performed on them.

The sequence is: client writes to nginx; nginx writes to upstream.

Can you see: is the extra slowness in the first part, or the second?

The usual first place to look is in the log files.

> error_log  off; #/var/log/nginx/error.log  info;

You can probably look in the file /usr/local/nginx/off to see what nginx
says is happening; but you might want to increase the log level to see
more details.

>       access_log    off; #/var/log/nginx/access.log;

You don't have an access log to look in. That probably does not matter
much here.

>       sendfile      on;
>       tcp_nopush    on;
>       tcp_nodelay   on;
>
>       types_hash_max_size 2048;
>
>       open_file_cache          max=10000 inactive=10m;
>       open_file_cache_valid    1h;
>       open_file_cache_min_uses 1;
>       open_file_cache_errors   on;

I think that those directives should not affect this test, either way.

>         listen  443 ssl;
>         location / {
>            proxy_pass <a href="http://127.0.0.1:7777;">http://127.0.0.1:7777;
>         }

> [PROBLEM] A 5MB file takes almost 30 seconds to upload via Nginx.
> When uploading it directly to the upstream backend, it takes ~400 millisec
> at max.

That does sound unnecessarily slow. The (presumed) ssl/no-ssl difference
should not account for that much overhead.

> Running strace, we've got this:
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  47.96    0.041738          11      3887      1489 read
>  21.73    0.018909          13      1509           epoll_wait
>  17.95    0.015622          22       708           writev
>  10.62    0.009241          13       712           write
>   0.47    0.000407          19        21        21 connect

I don't know the details, but what is nginx trying to connect() to that
is erroring every time? Is that likely relevant to the problem?

> A lot of errors in "read" calls: 1489 errors. They all correspond to (thanks
> again to strace):

It is possible that the nginx debug log might have more nginx-related
details than the bare strace.

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx