upstream_response_length and upstream_addr can't work

classic Classic list List threaded Threaded
5 messages Options
lx
Reply | Threaded
Open this post in threaded view
|

upstream_response_length and upstream_addr can't work

lx
hi all:
     When I use module of slice,  upstream_response_length and  upstream_addr can't work. 
nginx.conf :
#########################################################################
  include       mime.types;
    default_type  application/octet-stream;

    log_format  main '$status^$scheme^$request^$body_bytes_sent^$request_time^$upstream_cache_status^$remote_addr^$http_referer^$http_user_agent^$content_type^$http_range^$cookie_name^$upstream_addr^$upstream_response_time^$upstream_bytes_received^$upstream_response_length^[$time_local]';


    access_log  logs/access.log  main;
    rewrite_log on;

    sendfile        on;
    aio             threads;

    keepalive_timeout  65;

 if ($uri ~ ^/([a-zA-Z0-9\.]+)/([a-zA-Z0-9\.]+)/(.*)) {
      set $cdn $1;
      set $new_host $2;
     set $new_uri  $3;
}

        location / {
            slice 1m;
            proxy_cache_lock on;
            proxy_cache my_cache;
            proxy_cache_key $uri$is_args$args$slice_range;
            proxy_set_header Range $slice_range;
           proxy_cache_valid 200 206 24h;
           proxy_pass http://$cdn/$new_uri;
        }
#########################################################################
I Initiate a rang htttp request, for example 
#########################################################################
#########################################################################
upstream_response_length and upstream_bytes_received is just 1 MB, not 4.9MB.  I find nginx build 5 http request to A.com by tcpdump, and nginx implement slice by subrequest.

This is why? How to fix it? 

Thank you









_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: upstream_response_length and upstream_addr can't work

Roman Arutyunyan
Hi,

On Tue, Nov 26, 2019 at 07:24:00PM +0800, lx wrote:

> hi all:
>      When I use module of slice,  upstream_response_length and
> upstream_addr can't work.
> nginx.conf :
> #########################################################################
>   include       mime.types;
>     default_type  application/octet-stream;
>
>     log_format  main
> '$status^$scheme^$request^$body_bytes_sent^$request_time^$upstream_cache_status^$remote_addr^$http_referer^$http_user_agent^$content_type^$http_range^$cookie_name^$upstream_addr^$upstream_response_time^$upstream_bytes_received^$upstream_response_length^[$time_local]';
>
>
>     access_log  logs/access.log  main;
>     rewrite_log on;
>
>     sendfile        on;
>     aio             threads;
>
>     keepalive_timeout  65;
>
>  if ($uri ~ ^/([a-zA-Z0-9\.]+)/([a-zA-Z0-9\.]+)/(.*)) {
>       set $cdn $1;
>       set $new_host $2;
>      set $new_uri  $3;
> }
>
>         location / {
>             slice 1m;
>             proxy_cache_lock on;
>             proxy_cache my_cache;
>             proxy_cache_key $uri$is_args$args$slice_range;
>             proxy_set_header Range $slice_range;
>            proxy_cache_valid 200 206 24h;
>            proxy_pass http://$cdn/$new_uri;
>         }
> #########################################################################
> I Initiate a rang htttp request, for example
> #########################################################################
>  curl -o result -H 'Range: bytes=2001-4932000' "
> http://127.0.0.1:64002/A.com/B.com/appstore/developer/soft/20191008/201910081449521157660.patch
> "
> #########################################################################
> upstream_response_length and upstream_bytes_received is just 1 MB, not
> 4.9MB.  I find nginx build 5 http request to A.com by tcpdump, and nginx
> implement slice by subrequest.
>
> This is why? How to fix it?

Yes.  When using the slice module,  response is served by multiple subrequests.
Each subrequest serves its own part.  It has a separate cache key, fetches a
separate cache entry and contacts the upstream server using a serparate
connection.  When you use an $upstream_XXX variable, it returns data from
a subrequest.

If you want combined numbers, use client-side variables like $bytes_sent
instead.

>
> Thank you

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
lx
Reply | Threaded
Open this post in threaded view
|

Re: upstream_response_length and upstream_addr can't work

lx
Hi:
    When we use module of slice,  the nuber of bytes from upstream server is more than the bytes which sent to client,   So I want to get the number of  bytes from upstream server, How to get it?

Thank you

Roman Arutyunyan <[hidden email]> 于2019年11月26日周二 下午9:10写道:
Hi,

On Tue, Nov 26, 2019 at 07:24:00PM +0800, lx wrote:
> hi all:
>      When I use module of slice,  upstream_response_length and
> upstream_addr can't work.
> nginx.conf :
> #########################################################################
>   include       mime.types;
>     default_type  application/octet-stream;
>
>     log_format  main
> '$status^$scheme^$request^$body_bytes_sent^$request_time^$upstream_cache_status^$remote_addr^$http_referer^$http_user_agent^$content_type^$http_range^$cookie_name^$upstream_addr^$upstream_response_time^$upstream_bytes_received^$upstream_response_length^[$time_local]';
>
>
>     access_log  logs/access.log  main;
>     rewrite_log on;
>
>     sendfile        on;
>     aio             threads;
>
>     keepalive_timeout  65;
>
>  if ($uri ~ ^/([a-zA-Z0-9\.]+)/([a-zA-Z0-9\.]+)/(.*)) {
>       set $cdn $1;
>       set $new_host $2;
>      set $new_uri  $3;
> }
>
>         location / {
>             slice 1m;
>             proxy_cache_lock on;
>             proxy_cache my_cache;
>             proxy_cache_key $uri$is_args$args$slice_range;
>             proxy_set_header Range $slice_range;
>            proxy_cache_valid 200 206 24h;
>            proxy_pass http://$cdn/$new_uri;
>         }
> #########################################################################
> I Initiate a rang htttp request, for example
> #########################################################################
>  curl -o result -H 'Range: bytes=2001-4932000' "
> http://127.0.0.1:64002/A.com/B.com/appstore/developer/soft/20191008/201910081449521157660.patch
> "
> #########################################################################
> upstream_response_length and upstream_bytes_received is just 1 MB, not
> 4.9MB.  I find nginx build 5 http request to A.com by tcpdump, and nginx
> implement slice by subrequest.
>
> This is why? How to fix it?

Yes.  When using the slice module,  response is served by multiple subrequests.
Each subrequest serves its own part.  It has a separate cache key, fetches a
separate cache entry and contacts the upstream server using a serparate
connection.  When you use an $upstream_XXX variable, it returns data from
a subrequest.

If you want combined numbers, use client-side variables like $bytes_sent
instead.

>
> Thank you

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: upstream_response_length and upstream_addr can't work

Roman Arutyunyan
Hi,

On Wed, Nov 27, 2019 at 09:14:09AM +0800, lx wrote:
> Hi:
>     When we use module of slice,  the nuber of bytes from upstream server
> is more than the bytes which sent to client,   So I want to get the number
> of  bytes from upstream server, How to get it?

We don't have a variable that returns the sum of all upstream response sizes
from all subrequests.

Also, there can be multiple upstream servers.  And each slice can be fetched
from a different one.

> Thank you
>
> Roman Arutyunyan <[hidden email]> 于2019年11月26日周二 下午9:10写道:
>
> > Hi,
> >
> > On Tue, Nov 26, 2019 at 07:24:00PM +0800, lx wrote:
> > > hi all:
> > >      When I use module of slice,  upstream_response_length and
> > > upstream_addr can't work.
> > > nginx.conf :
> > > #########################################################################
> > >   include       mime.types;
> > >     default_type  application/octet-stream;
> > >
> > >     log_format  main
> > >
> > '$status^$scheme^$request^$body_bytes_sent^$request_time^$upstream_cache_status^$remote_addr^$http_referer^$http_user_agent^$content_type^$http_range^$cookie_name^$upstream_addr^$upstream_response_time^$upstream_bytes_received^$upstream_response_length^[$time_local]';
> > >
> > >
> > >     access_log  logs/access.log  main;
> > >     rewrite_log on;
> > >
> > >     sendfile        on;
> > >     aio             threads;
> > >
> > >     keepalive_timeout  65;
> > >
> > >  if ($uri ~ ^/([a-zA-Z0-9\.]+)/([a-zA-Z0-9\.]+)/(.*)) {
> > >       set $cdn $1;
> > >       set $new_host $2;
> > >      set $new_uri  $3;
> > > }
> > >
> > >         location / {
> > >             slice 1m;
> > >             proxy_cache_lock on;
> > >             proxy_cache my_cache;
> > >             proxy_cache_key $uri$is_args$args$slice_range;
> > >             proxy_set_header Range $slice_range;
> > >            proxy_cache_valid 200 206 24h;
> > >            proxy_pass http://$cdn/$new_uri;
> > >         }
> > > #########################################################################
> > > I Initiate a rang htttp request, for example
> > > #########################################################################
> > >  curl -o result -H 'Range: bytes=2001-4932000' "
> > >
> > http://127.0.0.1:64002/A.com/B.com/appstore/developer/soft/20191008/201910081449521157660.patch
> > > "
> > > #########################################################################
> > > upstream_response_length and upstream_bytes_received is just 1 MB, not
> > > 4.9MB.  I find nginx build 5 http request to A.com by tcpdump, and nginx
> > > implement slice by subrequest.
> > >
> > > This is why? How to fix it?
> >
> > Yes.  When using the slice module,  response is served by multiple
> > subrequests.
> > Each subrequest serves its own part.  It has a separate cache key, fetches
> > a
> > separate cache entry and contacts the upstream server using a serparate
> > connection.  When you use an $upstream_XXX variable, it returns data from
> > a subrequest.
> >
> > If you want combined numbers, use client-side variables like $bytes_sent
> > instead.
> >
> > >
> > > Thank you
> >
> > > _______________________________________________
> > > nginx mailing list
> > > [hidden email]
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> >
> > --
> > Roman Arutyunyan
> > _______________________________________________
> > nginx mailing list
> > [hidden email]
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
lx
Reply | Threaded
Open this post in threaded view
|

Re: upstream_response_length and upstream_addr can't work

lx
Hi:
   If I want to implement this variable which returns the sum of all upstream response sizes
from all subrequests.  This is feasible? 

Thank you

Roman Arutyunyan <[hidden email]> 于2019年11月27日周三 下午6:20写道:
Hi,

On Wed, Nov 27, 2019 at 09:14:09AM +0800, lx wrote:
> Hi:
>     When we use module of slice,  the nuber of bytes from upstream server
> is more than the bytes which sent to client,   So I want to get the number
> of  bytes from upstream server, How to get it?

We don't have a variable that returns the sum of all upstream response sizes
from all subrequests.

Also, there can be multiple upstream servers.  And each slice can be fetched
from a different one.

> Thank you
>
> Roman Arutyunyan <[hidden email]> 于2019年11月26日周二 下午9:10写道:
>
> > Hi,
> >
> > On Tue, Nov 26, 2019 at 07:24:00PM +0800, lx wrote:
> > > hi all:
> > >      When I use module of slice,  upstream_response_length and
> > > upstream_addr can't work.
> > > nginx.conf :
> > > #########################################################################
> > >   include       mime.types;
> > >     default_type  application/octet-stream;
> > >
> > >     log_format  main
> > >
> > '$status^$scheme^$request^$body_bytes_sent^$request_time^$upstream_cache_status^$remote_addr^$http_referer^$http_user_agent^$content_type^$http_range^$cookie_name^$upstream_addr^$upstream_response_time^$upstream_bytes_received^$upstream_response_length^[$time_local]';
> > >
> > >
> > >     access_log  logs/access.log  main;
> > >     rewrite_log on;
> > >
> > >     sendfile        on;
> > >     aio             threads;
> > >
> > >     keepalive_timeout  65;
> > >
> > >  if ($uri ~ ^/([a-zA-Z0-9\.]+)/([a-zA-Z0-9\.]+)/(.*)) {
> > >       set $cdn $1;
> > >       set $new_host $2;
> > >      set $new_uri  $3;
> > > }
> > >
> > >         location / {
> > >             slice 1m;
> > >             proxy_cache_lock on;
> > >             proxy_cache my_cache;
> > >             proxy_cache_key $uri$is_args$args$slice_range;
> > >             proxy_set_header Range $slice_range;
> > >            proxy_cache_valid 200 206 24h;
> > >            proxy_pass http://$cdn/$new_uri;
> > >         }
> > > #########################################################################
> > > I Initiate a rang htttp request, for example
> > > #########################################################################
> > >  curl -o result -H 'Range: bytes=2001-4932000' "
> > >
> > http://127.0.0.1:64002/A.com/B.com/appstore/developer/soft/20191008/201910081449521157660.patch
> > > "
> > > #########################################################################
> > > upstream_response_length and upstream_bytes_received is just 1 MB, not
> > > 4.9MB.  I find nginx build 5 http request to A.com by tcpdump, and nginx
> > > implement slice by subrequest.
> > >
> > > This is why? How to fix it?
> >
> > Yes.  When using the slice module,  response is served by multiple
> > subrequests.
> > Each subrequest serves its own part.  It has a separate cache key, fetches
> > a
> > separate cache entry and contacts the upstream server using a serparate
> > connection.  When you use an $upstream_XXX variable, it returns data from
> > a subrequest.
> >
> > If you want combined numbers, use client-side variables like $bytes_sent
> > instead.
> >
> > >
> > > Thank you
> >
> > > _______________________________________________
> > > nginx mailing list
> > > [hidden email]
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> >
> > --
> > Roman Arutyunyan
> > _______________________________________________
> > nginx mailing list
> > [hidden email]
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx