nginx update to 1.18.0 broke my wsl ubuntu 16.04 set up

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

nginx update to 1.18.0 broke my wsl ubuntu 16.04 set up

Bruce Klein
Does anyone have nginx 1.18.0 running successfully on WSL?

Previous versions have worked fine for me the past couple years as long as the standard WSL fix "fastcgi_buffering off" was applied.

I just picked up the 1.18.0 update and now I'm back to the old net::ERR_INCOMPLETE_CHUNKED_ENCODING errors you used to see with large pages when fastcgi_buffering was on.

Is it possible that as of the 1.18.0 update a different directive is required to turn fastcgi_buffering off? Or that nginx is now ignoring this directive?

Anyone have any suggestions for me to try? Thanks for any tips!

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx update to 1.18.0 broke my wsl ubuntu 16.04 set up

Maxim Dounin
Hello!

On Mon, May 11, 2020 at 08:39:22AM -1000, Bruce Klein wrote:

> Does anyone have nginx 1.18.0 running successfully on WSL?
>
> Previous versions have worked fine for me the past couple years as long as
> the standard WSL fix "fastcgi_buffering off" was applied.
>
> I just picked up the 1.18.0 update and now I'm back to the old
> net::ERR_INCOMPLETE_CHUNKED_ENCODING errors you used to see with large
> pages when fastcgi_buffering was on.
>
> Is it possible that as of the 1.18.0 update a different directive is
> required to turn fastcgi_buffering off? Or that nginx is now ignoring this
> directive?
>
> Anyone have any suggestions for me to try? Thanks for any tips!

Things are expected to work regardless of "fastcgi_buffering"
being on or off.  And the error suggests that there is something wrong
with the setup.  Instead of trying to work around the problem with
"fastcgi_buffering", it might be a good idea to actually dig into
what goes wrong and why the error appears.  In particular, looking
into error log might be a good idea.

Once the underlying problem is understood, you'll be able to fix
it properly, without involving magic like "the standard WSL fix".  
And the proper fix is expected to work with any nginx version.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx update to 1.18.0 broke my wsl ubuntu 16.04 set up

Bruce Klein
Hi Maxim,

Thank you the reply, which I appreciate very much. I fully agree in spirit.

In practice, the issue of previous versions not working on WSL is a long-standing bug vs WSL that people far more expert than me on unix internals, WSL, nginx, and fpm have not yet solved for two years plus, other than everyone being told to disable fastcgi_buffering. (If you're interested, there's plenty of history in various WSL bug reports to read through.)

No doubt the root cause here is a flaw in WSL. That's not on the nginx team to fix.

That said, as a practical matter, the once easily available workaround is now gone. I'd like to understand what changed in 1.18 and if there is an easy adaptation to it, as that seems the path of least resistance.

For what it's worth, the issue generates no logging in either the nginx error logs, access logs, or php7.1-fpm logs. It's impact is visible only on the web client side, where the user sees it as a partially received page and the net::ERR_INCOMPLETE_CHUNKED_ENCODING is available from the browser developer tools once the browser has timed out on waiting for the rest of the page.

Thanks again,
Bruce


On Mon, May 11, 2020 at 9:21 AM Maxim Dounin <[hidden email]> wrote:
Hello!

On Mon, May 11, 2020 at 08:39:22AM -1000, Bruce Klein wrote:

> Does anyone have nginx 1.18.0 running successfully on WSL?
>
> Previous versions have worked fine for me the past couple years as long as
> the standard WSL fix "fastcgi_buffering off" was applied.
>
> I just picked up the 1.18.0 update and now I'm back to the old
> net::ERR_INCOMPLETE_CHUNKED_ENCODING errors you used to see with large
> pages when fastcgi_buffering was on.
>
> Is it possible that as of the 1.18.0 update a different directive is
> required to turn fastcgi_buffering off? Or that nginx is now ignoring this
> directive?
>
> Anyone have any suggestions for me to try? Thanks for any tips!

Things are expected to work regardless of "fastcgi_buffering"
being on or off.  And the error suggests that there is something wrong
with the setup.  Instead of trying to work around the problem with
"fastcgi_buffering", it might be a good idea to actually dig into
what goes wrong and why the error appears.  In particular, looking
into error log might be a good idea.

Once the underlying problem is understood, you'll be able to fix
it properly, without involving magic like "the standard WSL fix". 
And the proper fix is expected to work with any nginx version.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx update to 1.18.0 broke my wsl ubuntu 16.04 set up

Maxim Dounin
Hello!

On Mon, May 11, 2020 at 09:38:12AM -1000, Bruce Klein wrote:

> Hi Maxim,
>
> Thank you the reply, which I appreciate very much. I fully agree in spirit.
>
> In practice, the issue of previous versions not working on WSL is a
> long-standing bug vs WSL that people far more expert than me on unix
> internals, WSL, nginx, and fpm have not yet solved for two years plus,
> other than everyone being told to disable fastcgi_buffering. (If you're
> interested, there's plenty of history in various WSL bug reports to read
> through.)
>
> No doubt the root cause here is a flaw in WSL. That's not on the nginx team
> to fix.
>
> That said, as a practical matter, the once easily available workaround is
> now gone. I'd like to understand what changed in 1.18 and if there is an
> easy adaptation to it, as that seems the path of least resistance.

To find out how to adapt a workaround - first you'll have to find
out why the workaround used to work.  That is, what is the bug in
WSL we are trying to work around.

Also note that it might not be a good idea to use things which
depend on unexplained workarounds for flaws not fixed for years.  
As long as there is no explanation why the workaround work, this
usually means that it can stop working unexpectedly and/or won't
work in some edge cases.

> For what it's worth, the issue generates no logging in either the nginx
> error logs, access logs, or php7.1-fpm logs. It's impact is visible only on
> the web client side, where the user sees it as a partially received page
> and the net::ERR_INCOMPLETE_CHUNKED_ENCODING is available from the browser
> developer tools once the browser has timed out on waiting for the rest of
> the page.

So, the problem is that transfer stalls at some point, correct?  
This looks like an issue with sockets handling, and some things to
try include:

1. Check the debug log to find out where things stall from nginx
point of view.

2. Try different event methods, such as "select" and "poll"
(http://nginx.org/r/use).  Note that this might require you to
compile nginx yourself.

3. Play with socket-related options, such as tcp_nodelay
(http://nginx.org/r/tcp_nodelay) and tcp_nopush
(http://nginx.org/r/tcp_nopush).  Unlikely to help though.

4. Play with TCP buffers ("listen ... sndbuf=...", assuming it
stalls somewhere while sending to the client) to see if it helps.  
Likely a buffer larger than the response size should help.

5. Play with "fastcgi_max_temp_file_size 0;" and/or "sendfile
on/off".

As long as playing with buffering used to help somehow, this
suggests that there is a problem with event reporting in the epoll
emulation layer.  I don't think that this is something that can be
fixed on nginx side, and any workarounds, including
"fastcgi_buffering off", are likely to fail in some edge cases.  
The working solution might be to use other event methods though,
such as "select" or "poll", see above.  Or to make sure that
socket buffers are large enough to avoid blocking.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx update to 1.18.0 broke my wsl ubuntu 16.04 set up

Bruce Klein
Thanks Maxim! I don't know if I'll be able to fix it with that but I'll sure learn a lot trying. I appreciate all the pointers on where to look.

Best,
Bruce

On Mon, May 11, 2020 at 7:54 PM Maxim Dounin <[hidden email]> wrote:
Hello!

On Mon, May 11, 2020 at 09:38:12AM -1000, Bruce Klein wrote:

> Hi Maxim,
>
> Thank you the reply, which I appreciate very much. I fully agree in spirit.
>
> In practice, the issue of previous versions not working on WSL is a
> long-standing bug vs WSL that people far more expert than me on unix
> internals, WSL, nginx, and fpm have not yet solved for two years plus,
> other than everyone being told to disable fastcgi_buffering. (If you're
> interested, there's plenty of history in various WSL bug reports to read
> through.)
>
> No doubt the root cause here is a flaw in WSL. That's not on the nginx team
> to fix.
>
> That said, as a practical matter, the once easily available workaround is
> now gone. I'd like to understand what changed in 1.18 and if there is an
> easy adaptation to it, as that seems the path of least resistance.

To find out how to adapt a workaround - first you'll have to find
out why the workaround used to work.  That is, what is the bug in
WSL we are trying to work around.

Also note that it might not be a good idea to use things which
depend on unexplained workarounds for flaws not fixed for years. 
As long as there is no explanation why the workaround work, this
usually means that it can stop working unexpectedly and/or won't
work in some edge cases.

> For what it's worth, the issue generates no logging in either the nginx
> error logs, access logs, or php7.1-fpm logs. It's impact is visible only on
> the web client side, where the user sees it as a partially received page
> and the net::ERR_INCOMPLETE_CHUNKED_ENCODING is available from the browser
> developer tools once the browser has timed out on waiting for the rest of
> the page.

So, the problem is that transfer stalls at some point, correct? 
This looks like an issue with sockets handling, and some things to
try include:

1. Check the debug log to find out where things stall from nginx
point of view.

2. Try different event methods, such as "select" and "poll"
(http://nginx.org/r/use).  Note that this might require you to
compile nginx yourself.

3. Play with socket-related options, such as tcp_nodelay
(http://nginx.org/r/tcp_nodelay) and tcp_nopush
(http://nginx.org/r/tcp_nopush).  Unlikely to help though.

4. Play with TCP buffers ("listen ... sndbuf=...", assuming it
stalls somewhere while sending to the client) to see if it helps. 
Likely a buffer larger than the response size should help.

5. Play with "fastcgi_max_temp_file_size 0;" and/or "sendfile
on/off".

As long as playing with buffering used to help somehow, this
suggests that there is a problem with event reporting in the epoll
emulation layer.  I don't think that this is something that can be
fixed on nginx side, and any workarounds, including
"fastcgi_buffering off", are likely to fail in some edge cases. 
The working solution might be to use other event methods though,
such as "select" or "poll", see above.  Or to make sure that
socket buffers are large enough to avoid blocking.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: nginx update to 1.18.0 broke my wsl ubuntu 16.04 set up

Bruce Klein
Following up in case anyone with this same issue discovers this thread via search:

My problems went away once I added these two new directives to nginx.conf:
fastcgi_buffer_size 1024k;
fastcgi_buffers 4 2048k;


I am not recommending this config for any production use or even any new development setups. For myself, I'm happy to have this crutch so I can eke out a little more life from my existing WSL 1 setup prior to the imminent release of WSL 2.

On Tue, May 12, 2020 at 8:13 AM Bruce Klein <[hidden email]> wrote:
Thanks Maxim! I don't know if I'll be able to fix it with that but I'll sure learn a lot trying. I appreciate all the pointers on where to look.

Best,
Bruce

On Mon, May 11, 2020 at 7:54 PM Maxim Dounin <[hidden email]> wrote:
Hello!

On Mon, May 11, 2020 at 09:38:12AM -1000, Bruce Klein wrote:

> Hi Maxim,
>
> Thank you the reply, which I appreciate very much. I fully agree in spirit.
>
> In practice, the issue of previous versions not working on WSL is a
> long-standing bug vs WSL that people far more expert than me on unix
> internals, WSL, nginx, and fpm have not yet solved for two years plus,
> other than everyone being told to disable fastcgi_buffering. (If you're
> interested, there's plenty of history in various WSL bug reports to read
> through.)
>
> No doubt the root cause here is a flaw in WSL. That's not on the nginx team
> to fix.
>
> That said, as a practical matter, the once easily available workaround is
> now gone. I'd like to understand what changed in 1.18 and if there is an
> easy adaptation to it, as that seems the path of least resistance.

To find out how to adapt a workaround - first you'll have to find
out why the workaround used to work.  That is, what is the bug in
WSL we are trying to work around.

Also note that it might not be a good idea to use things which
depend on unexplained workarounds for flaws not fixed for years. 
As long as there is no explanation why the workaround work, this
usually means that it can stop working unexpectedly and/or won't
work in some edge cases.

> For what it's worth, the issue generates no logging in either the nginx
> error logs, access logs, or php7.1-fpm logs. It's impact is visible only on
> the web client side, where the user sees it as a partially received page
> and the net::ERR_INCOMPLETE_CHUNKED_ENCODING is available from the browser
> developer tools once the browser has timed out on waiting for the rest of
> the page.

So, the problem is that transfer stalls at some point, correct? 
This looks like an issue with sockets handling, and some things to
try include:

1. Check the debug log to find out where things stall from nginx
point of view.

2. Try different event methods, such as "select" and "poll"
(http://nginx.org/r/use).  Note that this might require you to
compile nginx yourself.

3. Play with socket-related options, such as tcp_nodelay
(http://nginx.org/r/tcp_nodelay) and tcp_nopush
(http://nginx.org/r/tcp_nopush).  Unlikely to help though.

4. Play with TCP buffers ("listen ... sndbuf=...", assuming it
stalls somewhere while sending to the client) to see if it helps. 
Likely a buffer larger than the response size should help.

5. Play with "fastcgi_max_temp_file_size 0;" and/or "sendfile
on/off".

As long as playing with buffering used to help somehow, this
suggests that there is a problem with event reporting in the epoll
emulation layer.  I don't think that this is something that can be
fixed on nginx side, and any workarounds, including
"fastcgi_buffering off", are likely to fail in some edge cases. 
The working solution might be to use other event methods though,
such as "select" or "poll", see above.  Or to make sure that
socket buffers are large enough to avoid blocking.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx