different Memory consumption for H1 and H2

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

different Memory consumption for H1 and H2

Muhui Jiang
Hi

Recently, I did an experiment to test the memory consumption of nginx. I request a large static zip file. I explored the debug information of nginx.

For H2, below is a part of the log, I noticed that every time server will allocate 65536 bytes, I increase the connection number, I noticed that the server's memory consumption will reach to a threshhold and then increase very slowly:
2017/05/11 04:54:20 [debug] 29451#0: *10499 http2:1 HEADERS frame 00000000026155F0 was sent
2017/05/11 04:54:20 [debug] 29451#0: *10499 http2 frame sent: 00000000026155F0 sid:1 bl:1 len:119
2017/05/11 04:54:20 [debug] 29451#0: *10499 http output filter "/image/test.zip?"
2017/05/11 04:54:20 [debug] 29451#0: *10499 http copy filter: "/image/test.zip?"
2017/05/11 04:54:20 [debug] 29451#0: *10499 malloc: 0000000002699A80:65536
2017/05/11 04:54:20 [debug] 29451#0: *10499 read: 14, 0000000002699A80, 65536, 0
2017/05/11 04:54:20 [debug] 29451#0: *10499 http postpone filter "/image/test.zip?" 0000000002616098
2017/05/11 04:54:20 [debug] 29451#0: *10499 write new buf t:1 f:1 0000000002699A80, pos 0000000002699A80, size: 65536 file: 0, size: 65536
2017/05/11 04:54:20 [debug] 29451#0: *10499 http write filter: l:0 f:1 s:65536
2017/05/11 04:54:20 [debug] 29451#0: *10499 http write filter limit 0
2017/05/11 04:54:20 [debug] 29451#0: *10499 http2:1 create DATA frame 00000000026155F0: len:1 flags:0
2017/05/11 04:54:20 [debug] 29451#0: *10499 http2 frame out: 00000000026155F0 sid:1 bl:0 len:1
2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL buf copy: 9
2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL buf copy: 1
2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL to write: 138
2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL_write: 138
2017/05/11 04:54:20 [debug] 29451#0: *10499 http2:1 DATA frame 00000000026155F0 was sent
2017/05/11 04:54:20 [debug] 29451#0: *10499 http2 frame sent: 00000000026155F0 sid:1 bl:0 len:1
2017/05/11 04:54:20 [debug] 29451#0: *10499 http write filter 00000000026160A8
2017/05/11 04:54:20 [debug] 29451#0: *10499 malloc: 00000000026A9A90:65536

For H/1.1, below is a part of the debug log, no malloc is noticed during the send file process. And even when I increase the connection number to a very large value, the result shows nginx's memory consumption is still very low. :
2017/05/11 22:29:06 [debug] 29451#0: *11015 http run request: "/image/test.zip?"
2017/05/11 22:29:06 [debug] 29451#0: *11015 http writer handler: "/image/test.zip?"
2017/05/11 22:29:06 [debug] 29451#0: *11015 http output filter "/image/test.zip?"
2017/05/11 22:29:06 [debug] 29451#0: *11015 http copy filter: "/image/test.zip?"
2017/05/11 22:29:06 [debug] 29451#0: *11015 http postpone filter "/image/test.zip?" 0000000000000000
2017/05/11 22:29:06 [debug] 29451#0: *11015 write old buf t:0 f:1 0000000000000000, pos 0000000000000000, size: 0 file: 72470952, size: 584002
2017/05/11 22:29:06 [debug] 29451#0: *11015 http write filter: l:1 f:0 s:584002
2017/05/11 22:29:06 [debug] 29451#0: *11015 http write filter limit 0
2017/05/11 22:29:06 [debug] 29451#0: *11015 sendfile: @72470952 584002
2017/05/11 22:29:06 [debug] 29451#0: *11015 sendfile: 260640 of 584002 @72470952
2017/05/11 22:29:06 [debug] 29451#0: *11015 http write filter 0000000002670F70
2017/05/11 22:29:06 [debug] 29451#0: *11015 http copy filter: -2 "/image/test.zip?"
2017/05/11 22:29:06 [debug] 29451#0: *11015 http writer output filter: -2, "/image/test.zip?"
2017/05/11 22:29:06 [debug] 29451#0: *11015 event timer: 3, old: 1494513006630, new: 1494513006763

Hope to get your comments and what are the difference of nginx's memory allocation mechanisms between HTTP/2.0 and HTTP/1.1. Many Thanks

Regards
Muhui

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: different Memory consumption for H1 and H2

Maxim Dounin
Hello!

On Thu, May 11, 2017 at 10:32:41PM +0800, Muhui Jiang wrote:

> Recently, I did an experiment to test the memory consumption of nginx. I
> request a large static zip file. I explored the debug information of nginx.
>
> For H2, below is a part of the log, I noticed that every time server will
> allocate 65536 bytes, I increase the connection number, I noticed that the
> server's memory consumption will reach to a threshhold and then increase
> very slowly:

[...]

> 2017/05/11 04:54:20 [debug] 29451#0: *10499 http output filter
> "/image/test.zip?"
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 http copy filter:
> "/image/test.zip?"
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 malloc: 0000000002699A80:65536
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 read: 14, 0000000002699A80,
> 65536, 0

[...]

> 2017/05/11 04:54:20 [debug] 29451#0: *10499 http2 frame out:
> 00000000026155F0 sid:1 bl:0 len:1
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL buf copy: 9
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL buf copy: 1
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL to write: 138
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL_write: 138
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 http2:1 DATA frame
> 00000000026155F0 was sent

[...]

> For H/1.1, below is a part of the debug log, no malloc is noticed during
> the send file process. And even when I increase the connection number to a
> very large value, the result shows nginx's memory consumption is still very
> low. :

[...]

> 2017/05/11 22:29:06 [debug] 29451#0: *11015 http write filter limit 0
> 2017/05/11 22:29:06 [debug] 29451#0: *11015 sendfile: @72470952 584002
> 2017/05/11 22:29:06 [debug] 29451#0: *11015 sendfile: 260640 of 584002

[...]

> Hope to get your comments and what are the difference of nginx's memory
> allocation mechanisms between HTTP/2.0 and HTTP/1.1. Many Thanks

The difference is due to sendfile(), which is used in case of
plain HTTP, and can't be used with SSL-encrypted connections.  
HTTP/2 is normally used with SSL encryption, so it is usually not
possible to use sendfile() with HTTP/2.

When sendfile() is not available or switched off, nginx uses
output_buffers (http://nginx.org/r/output_buffers) to read a file
from disk, and then writes these buffers to the connection.

When it is possible to use the sendfile(), nginx does not try to
read contents of static files it returns, but simply calls
sendfile().  This is usually most effecient approach , as it
avoids additional buffers and copying between kernel space and
userland.  Unfortunately, it is not available when using HTTPS
(including HTTP/2).

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: different Memory consumption for H1 and H2

Muhui Jiang
HI

Thanks for your great answer. you mentioned that sendfile() is to copy between kernel space and userland. I am curious, why this whole process don't need to malloc any memory? Could you please explain more on the detail implementation of the sendfile(). Many Thanks

Regards
Muhui

2017-05-11 23:11 GMT+08:00 Maxim Dounin <[hidden email]>:
Hello!

On Thu, May 11, 2017 at 10:32:41PM +0800, Muhui Jiang wrote:

> Recently, I did an experiment to test the memory consumption of nginx. I
> request a large static zip file. I explored the debug information of nginx.
>
> For H2, below is a part of the log, I noticed that every time server will
> allocate 65536 bytes, I increase the connection number, I noticed that the
> server's memory consumption will reach to a threshhold and then increase
> very slowly:

[...]

> 2017/05/11 04:54:20 [debug] 29451#0: *10499 http output filter
> "/image/test.zip?"
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 http copy filter:
> "/image/test.zip?"
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 malloc: 0000000002699A80:65536
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 read: 14, 0000000002699A80,
> 65536, 0

[...]

> 2017/05/11 04:54:20 [debug] 29451#0: *10499 http2 frame out:
> 00000000026155F0 sid:1 bl:0 len:1
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL buf copy: 9
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL buf copy: 1
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL to write: 138
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 SSL_write: 138
> 2017/05/11 04:54:20 [debug] 29451#0: *10499 http2:1 DATA frame
> 00000000026155F0 was sent

[...]

> For H/1.1, below is a part of the debug log, no malloc is noticed during
> the send file process. And even when I increase the connection number to a
> very large value, the result shows nginx's memory consumption is still very
> low. :

[...]

> 2017/05/11 22:29:06 [debug] 29451#0: *11015 http write filter limit 0
> 2017/05/11 22:29:06 [debug] 29451#0: *11015 sendfile: @72470952 584002
> 2017/05/11 22:29:06 [debug] 29451#0: *11015 sendfile: 260640 of 584002

[...]

> Hope to get your comments and what are the difference of nginx's memory
> allocation mechanisms between HTTP/2.0 and HTTP/1.1. Many Thanks

The difference is due to sendfile(), which is used in case of
plain HTTP, and can't be used with SSL-encrypted connections.
HTTP/2 is normally used with SSL encryption, so it is usually not
possible to use sendfile() with HTTP/2.

When sendfile() is not available or switched off, nginx uses
output_buffers (http://nginx.org/r/output_buffers) to read a file
from disk, and then writes these buffers to the connection.

When it is possible to use the sendfile(), nginx does not try to
read contents of static files it returns, but simply calls
sendfile().  This is usually most effecient approach , as it
avoids additional buffers and copying between kernel space and
userland.  Unfortunately, it is not available when using HTTPS
(including HTTP/2).

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: different Memory consumption for H1 and H2

Maxim Dounin
Hello!

On Sat, May 13, 2017 at 12:02:13PM +0800, Muhui Jiang wrote:

> Thanks for your great answer. you mentioned that sendfile() is to copy
> between kernel space and userland. I am curious, why this whole process
> don't need to malloc any memory? Could you please explain more on the
> detail implementation of the sendfile(). Many Thanks

No, I said that sendfile() avoids copying between kernel space and
userland.  The sendfile() system call is to send a file to a
socket.  As such, it allows nginx to simply open the file
requested by a client and then call sendfile() to instruct OS to
transfer it to the socket.

--
Maxim Dounin
http://nginx.org/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx