repeated reloads lead to unresponsive server

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

repeated reloads lead to unresponsive server

zakirenish
we observe that after several days in service, where the server is reloaded
several hundred times a day, it eventually stops responding.

a reload completes, but still all connections time out.
a restart corrects the issue.

is there a limit to the number of times a server permits a reload before it
is necessary to restart it.
when the problem starts, entries like the following appear in the nginx
error log

ter process /usr/sbin/nginx -g daemon on; master_process on;:
/build/nginx-5J5hor/nginx-1.18.0/debian/modules/nchan/src/store/memory/memst\
ore.c:701: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2020/08/31 12:07:18 [alert] 1451759#1451759: worker process 1500846 exited
on signal 6 (core dumped)
2020/08/31 12:07:18 [alert] 1451759#1451759: shared memory zone "memstore"
was locked by 1500846

i see no mention of this issue here. but i did note

    https://github.com/slact/nchan/issues/446

versions:
root@nl12:~# uname -a
Linux nl12.dydra.com 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
root@nl12:~# which nginx
/usr/sbin/nginx
root@nl12:~# /usr/sbin/nginx -v
nginx version: nginx/1.18.0 (Ubuntu)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289247,289247#msg-289247

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: repeated reloads lead to unresponsive server

Thomas Ward

Do you actually use NCHAN for anything?  If you are not actively using nchan, you should consider simply removing `libnginx-mod-nchan`

Note also this is a third party module, so it's not necessarily 'endorsed' by NGINX Upstream per se.  Also, Debian is *ancient* with its nginx version and modules, I'd suggest you switch to a more recent Debian version with newer nginx, or use the nginx.org repositories (no third party plugins there though...)


Thomas

On 8/31/20 11:03 AM, james.anderson wrote:
we observe that after several days in service, where the server is reloaded
several hundred times a day, it eventually stops responding.

a reload completes, but still all connections time out.
a restart corrects the issue.

is there a limit to the number of times a server permits a reload before it
is necessary to restart it.
when the problem starts, entries like the following appear in the nginx
error log

ter process /usr/sbin/nginx -g daemon on; master_process on;:
/build/nginx-5J5hor/nginx-1.18.0/debian/modules/nchan/src/store/memory/memst\
ore.c:701: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2020/08/31 12:07:18 [alert] 1451759#1451759: worker process 1500846 exited
on signal 6 (core dumped)
2020/08/31 12:07:18 [alert] 1451759#1451759: shared memory zone "memstore"
was locked by 1500846

i see no mention of this issue here. but i did note

    https://github.com/slact/nchan/issues/446

versions:
root@nl12:~# uname -a
Linux nl12.dydra.com 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
root@nl12:~# which nginx
/usr/sbin/nginx
root@nl12:~# /usr/sbin/nginx -v
nginx version: nginx/1.18.0 (Ubuntu)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289247,289247#msg-289247

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: repeated reloads lead to unresponsive server

zakirenish
we found it to be impractical to move from ubuntu's released nginx version
in order to avoid that the perl module crashed on _every_ reload.
i was unaware, that the presence of the nchan module could cause an issue if
one did not use it, but will disable it an see if that helps.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289247,289252#msg-289252

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx