mirror delay

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

mirror delay

Eylon Saadon
Hi,
I'm using the mirror module in my "production" nginx in order to mirror real traffic to a test envrionment.
I don't want this mirroring to affect the latency of the production environment, but it looks like the nginx is waiting for the response from the test environment.
is there a way to avoid this? I just want the request to get to the test environment and let it process it. but it shouldn't wait fo r the response from the test environment in order to respond to the request
  
--
Thanks,
      Eylon Saadon

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: mirror delay

Roman Arutyunyan
Hi,

On Thu, Aug 30, 2018 at 04:34:29PM +0300, Eylon Saadon wrote:
> Hi,
> I'm using the mirror module in my "production" nginx in order to mirror
> real traffic to a test envrionment.
> I don't want this mirroring to affect the latency of the production
> environment, but it looks like the nginx is waiting for the response from
> the test environment.
> is there a way to avoid this? I just want the request to get to the test
> environment and let it process it. but it shouldn't wait fo r the response
> from the test environment in order to respond to the request

Usually a mirror subrequest does not affect the main request.  However there
are two issues with mirroring:

- the next request on the same connection will not be processed until all
mirror subrequests finish.  Try disabling keepalive and see if it helps.

- if you use sendfile and tcp_nopush, it's possible that the response is not
pushed properly because of a mirror subrequest, which may result in a delay.
Turn off sendfile and see if it helps.

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: mirror delay

Eylon Saadon
hi, 
thanks for the quick response! 
I'm not using sendfile or tcp_nopush. 
just to make sure. I should disable the keepalive for the mirror location. 
and do it like so?

server {

resolver 8.8.8.8;

listen 80;

location / {
proxy_set_header Host $host;
proxy_pass http://server:9000;
}

location /mirror {
internal;
keepalive_timeout 0;

proxy_pass https://test_server$request_uri;
}
}


Thanks,
    eylon saadon

On Thu, Aug 30, 2018 at 4:52 PM Roman Arutyunyan <[hidden email]> wrote:
Hi,

On Thu, Aug 30, 2018 at 04:34:29PM +0300, Eylon Saadon wrote:
> Hi,
> I'm using the mirror module in my "production" nginx in order to mirror
> real traffic to a test envrionment.
> I don't want this mirroring to affect the latency of the production
> environment, but it looks like the nginx is waiting for the response from
> the test environment.
> is there a way to avoid this? I just want the request to get to the test
> environment and let it process it. but it shouldn't wait fo r the response
> from the test environment in order to respond to the request

Usually a mirror subrequest does not affect the main request.  However there
are two issues with mirroring:

- the next request on the same connection will not be processed until all
mirror subrequests finish.  Try disabling keepalive and see if it helps.

- if you use sendfile and tcp_nopush, it's possible that the response is not
pushed properly because of a mirror subrequest, which may result in a delay.
Turn off sendfile and see if it helps.

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


--
Thanks,
      Eylon Saadon

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: mirror delay

Roman Arutyunyan
Hi,

On Thu, Aug 30, 2018 at 05:19:53PM +0300, Eylon Saadon wrote:
> hi,
> thanks for the quick response!
> I'm not using sendfile or tcp_nopush.
> just to make sure. I should disable the keepalive for the mirror location.
> and do it like so?

No, for the primary location.  This will help us understand the reason
why you have the delay.

> server {
>
>   resolver 8.8.8.8;
>
>   listen 80;
>
>   location / {
>     proxy_set_header Host $host;
>     proxy_pass <a href="http://server:9000;">http://server:9000;
>   }
>
>   location /mirror {
>     internal;
>     keepalive_timeout 0;
>     proxy_pass https://test_server$request_uri;
>   }
> }
>
>
> Thanks,
>
>     eylon saadon
>
>
> On Thu, Aug 30, 2018 at 4:52 PM Roman Arutyunyan <[hidden email]> wrote:
>
> > Hi,
> >
> > On Thu, Aug 30, 2018 at 04:34:29PM +0300, Eylon Saadon wrote:
> > > Hi,
> > > I'm using the mirror module in my "production" nginx in order to mirror
> > > real traffic to a test envrionment.
> > > I don't want this mirroring to affect the latency of the production
> > > environment, but it looks like the nginx is waiting for the response from
> > > the test environment.
> > > is there a way to avoid this? I just want the request to get to the test
> > > environment and let it process it. but it shouldn't wait fo r the
> > response
> > > from the test environment in order to respond to the request
> >
> > Usually a mirror subrequest does not affect the main request.  However
> > there
> > are two issues with mirroring:
> >
> > - the next request on the same connection will not be processed until all
> > mirror subrequests finish.  Try disabling keepalive and see if it helps.
> >
> > - if you use sendfile and tcp_nopush, it's possible that the response is
> > not
> > pushed properly because of a mirror subrequest, which may result in a
> > delay.
> > Turn off sendfile and see if it helps.
> >
> > --
> > Roman Arutyunyan
> > _______________________________________________
> > nginx mailing list
> > [hidden email]
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
>
>
> --
> Thanks,
>       Eylon Saadon

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: mirror delay

Eylon Saadon
HI,
when adding 
keepalive_timeout 0;
to the main location it works fine.
even if the mirrored location doesn't respond immediately the latency doesn't go up.

is this the solution for the issue or just a way to understand the issue?

Thanks,
     Eylon Saadon

On Thu, Aug 30, 2018 at 5:28 PM Roman Arutyunyan <[hidden email]> wrote:
Hi,

On Thu, Aug 30, 2018 at 05:19:53PM +0300, Eylon Saadon wrote:
> hi,
> thanks for the quick response!
> I'm not using sendfile or tcp_nopush.
> just to make sure. I should disable the keepalive for the mirror location.
> and do it like so?

No, for the primary location.  This will help us understand the reason
why you have the delay.

> server {
>
>   resolver 8.8.8.8;
>
>   listen 80;
>
>   location / {
>     proxy_set_header Host $host;
>     proxy_pass http://server:9000;
>   }
>
>   location /mirror {
>     internal;
>     keepalive_timeout 0;
>     proxy_pass https://test_server$request_uri;
>   }
> }
>
>
> Thanks,
>
>     eylon saadon
>
>
> On Thu, Aug 30, 2018 at 4:52 PM Roman Arutyunyan <[hidden email]> wrote:
>
> > Hi,
> >
> > On Thu, Aug 30, 2018 at 04:34:29PM +0300, Eylon Saadon wrote:
> > > Hi,
> > > I'm using the mirror module in my "production" nginx in order to mirror
> > > real traffic to a test envrionment.
> > > I don't want this mirroring to affect the latency of the production
> > > environment, but it looks like the nginx is waiting for the response from
> > > the test environment.
> > > is there a way to avoid this? I just want the request to get to the test
> > > environment and let it process it. but it shouldn't wait fo r the
> > response
> > > from the test environment in order to respond to the request
> >
> > Usually a mirror subrequest does not affect the main request.  However
> > there
> > are two issues with mirroring:
> >
> > - the next request on the same connection will not be processed until all
> > mirror subrequests finish.  Try disabling keepalive and see if it helps.
> >
> > - if you use sendfile and tcp_nopush, it's possible that the response is
> > not
> > pushed properly because of a mirror subrequest, which may result in a
> > delay.
> > Turn off sendfile and see if it helps.
> >
> > --
> > Roman Arutyunyan
> > _______________________________________________
> > nginx mailing list
> > [hidden email]
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
>
>
> --
> Thanks,
>       Eylon Saadon

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


--
Thanks,
      Eylon Saadon

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: mirror delay

Roman Arutyunyan
Hi,

On Sun, Sep 02, 2018 at 05:01:19PM +0300, Eylon Saadon wrote:
> HI,
> when adding
> keepalive_timeout 0;
> to the main location it works fine.
> even if the mirrored location doesn't respond immediately the latency
> doesn't go up.
>
> is this the solution for the issue or just a way to understand the issue?

If you are ok with no keepalive for your clients, then you can use this
configuration.

Delayed processing of the next request is a known side-effect of how mirroring
is implemented in nginx, and this is unlikely to change.  The point was to
make sure this was actually the case.

> Thanks,
>      Eylon Saadon
>
> On Thu, Aug 30, 2018 at 5:28 PM Roman Arutyunyan <[hidden email]> wrote:
>
> > Hi,
> >
> > On Thu, Aug 30, 2018 at 05:19:53PM +0300, Eylon Saadon wrote:
> > > hi,
> > > thanks for the quick response!
> > > I'm not using sendfile or tcp_nopush.
> > > just to make sure. I should disable the keepalive for the mirror
> > location.
> > > and do it like so?
> >
> > No, for the primary location.  This will help us understand the reason
> > why you have the delay.
> >
> > > server {
> > >
> > >   resolver 8.8.8.8;
> > >
> > >   listen 80;
> > >
> > >   location / {
> > >     proxy_set_header Host $host;
> > >     proxy_pass <a href="http://server:9000;">http://server:9000;
> > >   }
> > >
> > >   location /mirror {
> > >     internal;
> > >     keepalive_timeout 0;
> > >     proxy_pass https://test_server$request_uri;
> > >   }
> > > }
> > >
> > >
> > > Thanks,
> > >
> > >     eylon saadon
> > >
> > >
> > > On Thu, Aug 30, 2018 at 4:52 PM Roman Arutyunyan <[hidden email]> wrote:
> > >
> > > > Hi,
> > > >
> > > > On Thu, Aug 30, 2018 at 04:34:29PM +0300, Eylon Saadon wrote:
> > > > > Hi,
> > > > > I'm using the mirror module in my "production" nginx in order to
> > mirror
> > > > > real traffic to a test envrionment.
> > > > > I don't want this mirroring to affect the latency of the production
> > > > > environment, but it looks like the nginx is waiting for the response
> > from
> > > > > the test environment.
> > > > > is there a way to avoid this? I just want the request to get to the
> > test
> > > > > environment and let it process it. but it shouldn't wait fo r the
> > > > response
> > > > > from the test environment in order to respond to the request
> > > >
> > > > Usually a mirror subrequest does not affect the main request.  However
> > > > there
> > > > are two issues with mirroring:
> > > >
> > > > - the next request on the same connection will not be processed until
> > all
> > > > mirror subrequests finish.  Try disabling keepalive and see if it
> > helps.
> > > >
> > > > - if you use sendfile and tcp_nopush, it's possible that the response
> > is
> > > > not
> > > > pushed properly because of a mirror subrequest, which may result in a
> > > > delay.
> > > > Turn off sendfile and see if it helps.
> > > >
> > > > --
> > > > Roman Arutyunyan
> > > > _______________________________________________
> > > > nginx mailing list
> > > > [hidden email]
> > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > >
> > >
> > > --
> > > Thanks,
> > >       Eylon Saadon
> >
> > > _______________________________________________
> > > nginx mailing list
> > > [hidden email]
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> >
> > --
> > Roman Arutyunyan
> > _______________________________________________
> > nginx mailing list
> > [hidden email]
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
>
>
> --
> Thanks,
>       Eylon Saadon

> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx


--
Roman Arutyunyan
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx