Found Nginx 1.19.0 stopped but no idea what happened

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Found Nginx 1.19.0 stopped but no idea what happened

evald ibrahimi
hello everybody,

I'm using nginx 1.19.0 with ModSec on Centos 7. The other day it
happened twice that nginx was stopped. No idea what could cause the
issue but i found on the error.log the following:

So it seems the server was running out of memory and it stopped nginx?
Thank you

2020/06/29 17:23:39 [alert] 1890#1890: sendmsg() failed (9: Bad file descriptor)
2020/06/29 17:23:39 [alert] 1890#1890: sendmsg() failed (9: Bad file descriptor)
2020/06/29 17:23:39 [alert] 1890#1890: sendmsg() failed (9: Bad file descriptor)
2020/06/29 17:24:28 [alert] 1890#1890: fork() failed while spawning
"worker process" (12: Cannot allocate memory)
2020/06/29 17:24:28 [alert] 1890#1890: sendmsg() failed (9: Bad file descriptor)
2020/06/29 17:24:28 [alert] 1890#1890: sendmsg() failed (9: Bad file descriptor)
2020/06/29 17:24:28 [alert] 1890#1890: fork() failed while spawning
"cache manager process" (12: Cannot allocate memory)
2020/06/29 17:24:28 [alert] 1890#1890: sendmsg() failed (9: Bad file descriptor)
2020/06/29 17:24:28 [alert] 1890#1890: sendmsg() failed (9: Bad file descriptor)
2020/06/29 18:10:35 [notice] 2461#2461: ModSecurity-nginx v1.0.1
(rules loaded inline/local/remote: 0/19826/0)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:443 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:80 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:443 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:80 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:443 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:80 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:443 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:80 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:443 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:80 failed
(98: Address already in use)
2020/06/29 18:10:35 [emerg] 2461#2461: still could not bind()
2020/06/29 18:11:16 [alert] 1890#1890: unlink() "/run/nginx.pid"
failed (2: No such file or directory)
2020/06/29 18:11:48 [notice] 927#927: ModSecurity-nginx v1.0.1 (rules
loaded inline/local/remote: 0/19826/0)
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

vergil
hi,

it happened again and i see that nginx is eating all the memory of the
server.
Somebody could help me how to troubleshoot? Mos probably there is a memory
leak but don't know if on nginx or ModSec.

There are 10websites on this server and they are not popular at all. Some
clicks a day, so no superbusy website.

thank you

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288511,288522#msg-288522

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

Francis Daly
In reply to this post by evald ibrahimi
On Wed, Jul 01, 2020 at 08:55:13AM +0200, evald ibrahimi wrote:

Hi there,

I don't have answers for you; but possibly you'll be able to provide
more information that will help someone to help you.

> I'm using nginx 1.19.0 with ModSec on Centos 7. The other day it
> happened twice that nginx was stopped. No idea what could cause the
> issue but i found on the error.log the following:

If unexpected things happen repeatedly, it can be useful to see if there
is a pattern that causes them to happen.

From your normal logs, was there anything usual that happened shortly
before things stopped, that did not happen at other times?

> 2020/06/29 17:23:39 [alert] 1890#1890: sendmsg() failed (9: Bad file descriptor)

That (I think) usually means that something has gone wrong already --
a file or socket that nginx thinks it should have access to, is not
accessible. (That might be a "real" file that has changed; or it might
be due to a coding bug.)

> 2020/06/29 17:24:28 [alert] 1890#1890: fork() failed while spawning
> "worker process" (12: Cannot allocate memory)

That means that your operating system was unwilling to give nginx the
extra memory it asked for. It might be a resource problem on your system,
or it might be a consequence of the previous problem.

> 2020/06/29 18:10:35 [notice] 2461#2461: ModSecurity-nginx v1.0.1
> (rules loaded inline/local/remote: 0/19826/0)

That possibly means that nginx has just started.

> 2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:443 failed
> (98: Address already in use)
> 2020/06/29 18:10:35 [emerg] 2461#2461: bind() to 0.0.0.0:80 failed
> (98: Address already in use)

And they mean that nginx can't start fully, because something else is
already running where nginx wants to. (Possibly a previous nginx which
has not fully shut down.)


"nginx -V" is usually very helpful for people to see what versions of what
code you are using. Often third-party modules written for one version of
nginx and used with another version of nginx, can lead to subtle problems.

ModSecurity-nginx is, I think, not a stock-nginx module. What other
modules are you using?


Often the nginx config can help identify a problem too; it is not clear
to me if that will help in this case.

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

J.R.
In reply to this post by evald ibrahimi
How much RAM is on your machine?

Have you tried disabling modsecurity temporarily?

What other (if any) 3rd party modules are you using?
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

vergil
hello,

There is not too much 1GB. No, i did not disabled mode_sec.

The list of loaded modules:
load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_brotli_static_module.so;
load_module modules/ngx_http_modsecurity_module.so;
load_module modules/ngx_http_geoip2_module.so;

this is an issue that suddenly started to come out. There always has been
this config and software installed.
I think some kind of requests are causing this but does not know if it is a
module or nginx itself.

Can i troubleshoot it? I mean, when the issue will pop up again, try to
gather some info? if yes, how?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288511,288531#msg-288531

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

vergil
In reply to this post by evald ibrahimi
The problem appeared again and at the time of writing is still present and i
did not reboot the machine which will fix it.
The following are the commands i executed in order to get some info:
Basically this seems a problem with nginx and not a library issue:

first of all: is it running nginx?

[root@web ~]# systemctl status nginx
● nginx.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor
preset: disabled)
   Active: inactive (dead) since Mon 2020-07-06 00:00:06 CEST; 10h ago
     Docs: http://nginx.org/en/docs/
  Process: 19475 ExecStop=/bin/sh -c /bin/kill -s TERM $(/bin/cat
/var/run/nginx.pid) (code=exited, status=0/SUCCESS)
  Process: 939 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
(code=exited, status=0/SUCCESS)
 Main PID: 1356 (code=exited, status=0/SUCCESS)

Jul 06 00:00:01 web systemd[1]: Stopping nginx - high performance web
server...
Jul 06 00:00:06 web systemd[1]: Stopped nginx - high performance web
server.
Warning: Journal has been rotated since unit was started. Log output is
incomplete or unavailable.


Seems not active ehm..let's try to see what is resident in memory:

[root@web ~]# ps -el
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 S     0     1     0  0  80   0 - 11564 ep_pol ?        00:00:55 systemd
1 S     0     2     0  0  80   0 -     0 kthrea ?        00:00:00 kthreadd
1 S     0     4     2  0  60 -20 -     0 worker ?        00:00:00
kworker/0:0H
1 S     0     6     2  0  80   0 -     0 smpboo ?        00:00:10
ksoftirqd/0
1 S     0     7     2  0 -40   - -     0 smpboo ?        00:00:00
migration/0
1 S     0     8     2  0  80   0 -     0 rcu_gp ?        00:00:00 rcu_bh
1 R     0     9     2  0  80   0 -     0 -      ?        00:00:34 rcu_sched
1 S     0    10     2  0  60 -20 -     0 rescue ?        00:00:00
lru-add-drain
5 S     0    11     2  0 -40   - -     0 smpboo ?        00:00:01
watchdog/0
5 S     0    13     2  0  80   0 -     0 devtmp ?        00:00:00 kdevtmpfs
1 S     0    14     2  0  60 -20 -     0 rescue ?        00:00:00 netns
1 S     0    15     2  0  80   0 -     0 watchd ?        00:00:00
khungtaskd
1 S     0    16     2  0  60 -20 -     0 rescue ?        00:00:00 writeback
1 S     0    17     2  0  60 -20 -     0 rescue ?        00:00:00
kintegrityd
1 S     0    18     2  0  60 -20 -     0 rescue ?        00:00:00 bioset
1 S     0    19     2  0  60 -20 -     0 rescue ?        00:00:00 bioset
1 S     0    20     2  0  60 -20 -     0 rescue ?        00:00:00 bioset
1 S     0    21     2  0  60 -20 -     0 rescue ?        00:00:00 kblockd
1 S     0    22     2  0  60 -20 -     0 rescue ?        00:00:00 md
1 S     0    23     2  0  60 -20 -     0 rescue ?        00:00:00
edac-poller
1 S     0    24     2  0  60 -20 -     0 rescue ?        00:00:00 watchdogd
1 S     0    30     2  0  80   0 -     0 kswapd ?        00:04:39 kswapd0
1 S     0    31     2  0  85   5 -     0 ksm_sc ?        00:00:00 ksmd
1 S     0    32     2  0  99  19 -     0 khugep ?        00:00:02
khugepaged
1 S     0    33     2  0  60 -20 -     0 rescue ?        00:00:00 crypto
1 S     0    41     2  0  60 -20 -     0 rescue ?        00:00:00 kthrotld
1 S     0    43     2  0  60 -20 -     0 rescue ?        00:00:00
kmpath_rdacd
1 S     0    44     2  0  60 -20 -     0 rescue ?        00:00:00 kaluad
1 S     0    45     2  0  60 -20 -     0 rescue ?        00:00:00 kpsmoused
1 S     0    47     2  0  60 -20 -     0 rescue ?        00:00:00
ipv6_addrconf
1 S     0    60     2  0  60 -20 -     0 rescue ?        00:00:00 deferwq
1 S     0    97     2  0  80   0 -     0 kaudit ?        00:00:02 kauditd
1 S     0   274     2  0  60 -20 -     0 rescue ?        00:00:00 nfit
1 S     0   275     2  0  60 -20 -     0 rescue ?        00:00:00
mpt_poll_0
1 S     0   276     2  0  60 -20 -     0 rescue ?        00:00:00 mpt/0
1 S     0   277     2  0  60 -20 -     0 rescue ?        00:00:00 ata_sff
1 S     0   283     2  0  80   0 -     0 scsi_e ?        00:00:00 scsi_eh_0
1 S     0   286     2  0  60 -20 -     0 rescue ?        00:00:00
scsi_tmf_0
1 S     0   292     2  0  80   0 -     0 scsi_e ?        00:00:00 scsi_eh_1
1 S     0   295     2  0  60 -20 -     0 rescue ?        00:00:00
scsi_tmf_1
1 S     0   296     2  0  80   0 -     0 scsi_e ?        00:00:00 scsi_eh_2
1 S     0   299     2  0  60 -20 -     0 rescue ?        00:00:00
scsi_tmf_2
5 S     0   309     2  0   9   - -     0 irq_th ?        00:00:05
irq/16-vmwgfx
1 S     0   310     2  0  60 -20 -     0 rescue ?        00:00:00 ttm_swap
1 S     0   395     2  0  60 -20 -     0 rescue ?        00:00:00 kdmflush
1 S     0   396     2  0  60 -20 -     0 rescue ?        00:00:00 bioset
1 S     0   406     2  0  60 -20 -     0 rescue ?        00:00:00 kdmflush
1 S     0   407     2  0  60 -20 -     0 rescue ?        00:00:00 bioset
1 S     0   421     2  0  80   0 -     0 kjourn ?        00:00:11
jbd2/dm-0-8
1 S     0   422     2  0  60 -20 -     0 rescue ?        00:00:00
ext4-rsv-conver
4 S     0   508     1  0  80   0 -  9390 ep_pol ?        00:01:42
systemd-journal
4 S     0   529     1  0  80   0 - 50277 poll_s ?        00:00:00 lvmetad
4 S     0   544     1  0  80   0 - 12039 ep_pol ?        00:00:00
systemd-udevd
1 S     0   569     2  0  60 -20 -     0 worker ?        00:00:02
kworker/0:1H
1 S     0   621     2  0  80   0 -     0 kjourn ?        00:00:00
jbd2/sda1-8
1 S     0   622     2  0  60 -20 -     0 rescue ?        00:00:00
ext4-rsv-conver
5 S     0   654     1  0  76  -4 - 13883 ep_pol ?        00:00:12 auditd
4 S    81   677     1  0  80   0 - 16578 ep_pol ?        00:00:26
dbus-daemon
4 S   999   682     1  0  80   0 - 153062 poll_s ?       00:00:02 polkitd
4 S     0   684     1  0  80   0 - 24922 poll_s ?        00:00:00
VGAuthService
4 S     0   685     1  0  80   0 -  6596 ep_pol ?        00:00:12
systemd-logind
4 S     0   686     1  0  80   0 - 76321 poll_s ?        00:05:54 vmtoolsd
4 S     0   692     1  0  80   0 - 31597 hrtime ?        00:00:02 crond
4 S     0   923     1  0  80   0 - 142417 ep_pol ?       00:00:22 php-fpm
4 S     0   926     1  0  80   0 - 160559 poll_s ?       00:00:21 httpd
4 S     0   928     1  0  80   0 - 143577 poll_s ?       00:00:43 tuned
4 S     0   936     1  0  80   0 - 28231 poll_s ?        00:00:23 sshd
4 S     0   938     1  0  80   0 - 66729 poll_s ?        00:00:47 rsyslogd
5 S     0   943     1  0  80   0 - 13322 inet_c ?        00:00:00 vsftpd
4 S     0   953     1  0  80   0 - 152429 poll_s ?       00:09:46
f2b/server
4 S    27   998     1  0  80   0 - 28354 do_wai ?        00:00:00
mysqld_safe
4 S     0  1069     1  0  80   0 - 27552 n_tty_ tty1     00:00:00 agetty
5 S   996  1104   923  0  80   0 - 179445 inet_c ?       00:02:31 php-fpm
5 S   996  1105   923  0  80   0 - 178437 inet_c ?       00:02:39 php-fpm
5 S   996  1108   923  0  80   0 - 178901 inet_c ?       00:02:22 php-fpm
5 S   996  1110   923  0  80   0 - 177776 inet_c ?       00:02:19 php-fpm
5 S   996  1113   923  0  80   0 - 179294 inet_c ?       00:02:29 php-fpm
0 S    27  1220   998  0  80   0 - 297171 poll_s ?       00:05:01 mysqld
5 S     0  1331     1  0  80   0 - 22426 ep_pol ?        00:00:02 master
4 S    89  1333  1331  0  80   0 - 22496 ep_pol ?        00:00:01 qmgr
5 S   996  1377   923  0  80   0 - 178856 inet_c ?       00:02:13 php-fpm
5 S   996  1380   923  0  80   0 - 173811 inet_c ?       00:02:29 php-fpm
5 S   996  1387   923  0  80   0 - 173294 inet_c ?       00:02:16 php-fpm
5 S   996  1393   923  0  80   0 - 177947 inet_c ?       00:02:24 php-fpm
5 S   996  1405   923  0  80   0 - 171703 inet_c ?       00:02:20 php-fpm
5 S   996  1406   923  0  80   0 - 177955 inet_c ?       00:02:12 php-fpm
5 S    48 12523   926  0  80   0 - 160559 inet_c ?       00:00:00 httpd
1 S     0 19606     2  0  80   0 -     0 worker ?        00:00:01
kworker/u128:2
5 S     0 19821     1  0  80   0 - 310302 sigsus ?       00:00:15 nginx
1 S     0 23270     2  0  80   0 -     0 worker ?        00:00:18
kworker/0:1
1 S     0 23271     2  0  80   0 -     0 worker ?        00:00:00
kworker/u128:0
4 S    89 27372  1331  0  80   0 - 22452 ep_pol ?        00:00:00 pickup
4 S     0 27666   936  0  80   0 - 38160 poll_s ?        00:00:00 sshd
4 S     0 27670 27666  0  80   0 - 29151 do_wai pts/0    00:00:00 bash
1 S     0 27787     2  0  80   0 -     0 worker ?        00:00:00
kworker/0:2
1 R     0 27901     2  0  80   0 -     0 -      ?        00:00:00
kworker/0:0
4 S     0 27928   936  0  80   0 - 28231 poll_s ?        00:00:00 sshd
5 S    74 27929 27928  0  80   0 - 28231 poll_s ?        00:00:00 sshd
0 R     0 27930 27670  0  80   0 - 38338 -      pts/0    00:00:00 ps
5 S    48 64818   926  0  80   0 - 160559 inet_c ?       00:00:00 httpd
5 S    48 64819   926  0  80   0 - 160559 inet_c ?       00:00:00 httpd
5 S    48 64820   926  0  80   0 - 160559 inet_c ?       00:00:00 httpd
5 S    48 64821   926  0  80   0 - 160559 inet_c ?       00:00:00 httpd
5 S    48 64822   926  0  80   0 - 160559 inet_c ?       00:00:00 httpd
[root@web ~]#

ok so:
5 S     0 19821     1  0  80   0 - 310302 sigsus ?       00:00:15 nginx

let's take the PID and see what is in memory:

[root@web ~]# pmap 19821
19821:   nginx: master process nginx -c /etc/nginx/nginx.conf
000055e515592000   1164K r-x-- nginx
000055e5158b5000      8K r---- nginx
000055e5158b7000    132K rw--- nginx
000055e5158d8000    128K rw---   [ anon ]
000055e51660e000 346604K rw---   [ anon ]
000055e52b889000 607984K rw---   [ anon ]
00007fda69936000  89792K rwx--   [ anon ]
00007fda6f0e6000   1024K rw-s- zero (deleted)
00007fda6f1e6000   1024K rw-s- zero (deleted)
00007fda6f2e6000   1024K rw-s- zero (deleted)
00007fda6f3e6000   1024K rw-s- zero (deleted)
00007fda6f4e6000   1024K rw-s- zero (deleted)
00007fda6f5e6000   1024K rw-s- zero (deleted)
00007fda6f6e6000   1024K rw-s- zero (deleted)
00007fda6f7e6000   1024K rw-s- zero (deleted)
00007fda6f8e6000   1024K rw-s- zero (deleted)
00007fda6f9e6000   1024K rw-s- zero (deleted)
00007fda6fae6000   1024K rw-s- zero (deleted)
00007fda6fbe6000   1024K rw-s- zero (deleted)
00007fda6fce6000   1024K rw-s- zero (deleted)
00007fda6fde6000  10240K rw-s- zero (deleted)
00007fda707e6000   1024K rw-s- zero (deleted)
00007fda708e6000   2112K rwx--   [ anon ]
00007fda70af7000    256K rwx--   [ anon ]
00007fda70b37000   2048K rwx--   [ anon ]
00007fda70d38000    256K rwx--   [ anon ]
00007fda70d78000   2112K rwx--   [ anon ]
00007fda70f89000    256K rwx--   [ anon ]
00007fda70fc9000   2048K rwx--   [ anon ]
00007fda711ca000    256K rwx--   [ anon ]
00007fda7120a000   2112K rwx--   [ anon ]
00007fda7141b000    256K rwx--   [ anon ]
00007fda7145b000   2048K rwx--   [ anon ]
00007fda7165c000    256K rwx--   [ anon ]
00007fda7169c000   2112K rwx--   [ anon ]
00007fda718ad000    256K rwx--   [ anon ]
00007fda718ed000   2048K rwx--   [ anon ]
00007fda71aee000    256K rwx--   [ anon ]
00007fda71b2e000   2112K rwx--   [ anon ]
00007fda71d3f000    256K rwx--   [ anon ]
00007fda71d7f000   2048K rwx--   [ anon ]
00007fda71f80000    256K rwx--   [ anon ]
00007fda71fc0000   2112K rwx--   [ anon ]
00007fda721d1000    256K rwx--   [ anon ]
00007fda72211000   2048K rwx--   [ anon ]
00007fda72412000    256K rwx--   [ anon ]
00007fda72452000   2112K rwx--   [ anon ]
00007fda72663000    256K rwx--   [ anon ]
00007fda726a3000   2048K rwx--   [ anon ]
00007fda728a4000    256K rwx--   [ anon ]
00007fda728e4000   2112K rwx--   [ anon ]
00007fda72af5000    256K rwx--   [ anon ]
00007fda72b35000   2048K rwx--   [ anon ]
00007fda72d36000    256K rwx--   [ anon ]
00007fda72d76000   2112K rwx--   [ anon ]
00007fda72f87000    256K rwx--   [ anon ]
00007fda72fc7000   2048K rwx--   [ anon ]
00007fda731c8000    256K rwx--   [ anon ]
00007fda73208000   2112K rwx--   [ anon ]
00007fda73419000    256K rwx--   [ anon ]
00007fda73459000   2048K rwx--   [ anon ]
00007fda7365a000    256K rwx--   [ anon ]
00007fda7369a000   2048K rwx--   [ anon ]
00007fda7389b000    256K rwx--   [ anon ]
00007fda738db000   2112K rwx--   [ anon ]
00007fda73aec000    256K rwx--   [ anon ]
00007fda73b2c000    448K rwx--   [ anon ]
00007fda73b9c000     20K r-x-- libmaxminddb.so.0.0.7
00007fda73ba1000   2044K ----- libmaxminddb.so.0.0.7
00007fda73da0000      4K r---- libmaxminddb.so.0.0.7
00007fda73da1000      4K rw--- libmaxminddb.so.0.0.7
00007fda73da2000     12K r-x-- ngx_http_geoip2_module.so
00007fda73da5000   2044K ----- ngx_http_geoip2_module.so
00007fda73fa4000      4K r---- ngx_http_geoip2_module.so
00007fda73fa5000      4K rw--- ngx_http_geoip2_module.so
00007fda73fa6000    112K r-x-- libsasl2.so.3.0.0
00007fda73fc2000   2044K ----- libsasl2.so.3.0.0
00007fda741c1000      4K r---- libsasl2.so.3.0.0
00007fda741c2000      4K rw--- libsasl2.so.3.0.0
00007fda741c3000    148K r-x-- liblzma.so.5.2.2
00007fda741e8000   2044K ----- liblzma.so.5.2.2
00007fda743e7000      4K r---- liblzma.so.5.2.2
00007fda743e8000      4K rw--- liblzma.so.5.2.2
00007fda743e9000    328K r-x-- libldap-2.4.so.2.10.7
00007fda7443b000   2048K ----- libldap-2.4.so.2.10.7
00007fda7463b000      8K r---- libldap-2.4.so.2.10.7
00007fda7463d000      4K rw--- libldap-2.4.so.2.10.7
00007fda7463e000     56K r-x-- liblber-2.4.so.2.10.7
00007fda7464c000   2044K ----- liblber-2.4.so.2.10.7
00007fda7484b000      4K r---- liblber-2.4.so.2.10.7
00007fda7484c000      4K rw--- liblber-2.4.so.2.10.7
00007fda7484d000    232K r-x-- libnspr4.so
00007fda74887000   2044K ----- libnspr4.so
00007fda74a86000      4K r---- libnspr4.so
00007fda74a87000      8K rw--- libnspr4.so
00007fda74a89000      8K rw---   [ anon ]
00007fda74a8b000     16K r-x-- libplc4.so
00007fda74a8f000   2044K ----- libplc4.so
00007fda74c8e000      4K r---- libplc4.so
00007fda74c8f000      4K rw--- libplc4.so
00007fda74c90000     12K r-x-- libplds4.so
00007fda74c93000   2044K ----- libplds4.so
00007fda74e92000      4K r---- libplds4.so
00007fda74e93000      4K rw--- libplds4.so
00007fda74e94000    164K r-x-- libnssutil3.so
00007fda74ebd000   2044K ----- libnssutil3.so
00007fda750bc000     28K r---- libnssutil3.so
00007fda750c3000      4K rw--- libnssutil3.so
00007fda750c4000   1176K r-x-- libnss3.so
00007fda751ea000   2048K ----- libnss3.so
00007fda753ea000     20K r---- libnss3.so
00007fda753ef000      8K rw--- libnss3.so
00007fda753f1000      8K rw---   [ anon ]
00007fda753f3000    148K r-x-- libsmime3.so
00007fda75418000   2044K ----- libsmime3.so
00007fda75617000     12K r---- libsmime3.so
00007fda7561a000      4K rw--- libsmime3.so
00007fda7561b000    332K r-x-- libssl3.so
00007fda7566e000   2048K ----- libssl3.so
00007fda7586e000     16K r---- libssl3.so
00007fda75872000      4K rw--- libssl3.so
00007fda75873000      4K rw---   [ anon ]
00007fda75874000    172K r-x-- libssh2.so.1.0.1
00007fda7589f000   2048K ----- libssh2.so.1.0.1
00007fda75a9f000      4K r---- libssh2.so.1.0.1
00007fda75aa0000      4K rw--- libssh2.so.1.0.1
00007fda75aa1000    200K r-x-- libidn.so.11.6.11
00007fda75ad3000   2044K ----- libidn.so.11.6.11
00007fda75cd2000      4K r---- libidn.so.11.6.11
00007fda75cd3000      4K rw--- libidn.so.11.6.11
00007fda75cd4000     84K r-x-- libgcc_s-4.8.5-20150702.so.1
00007fda75ce9000   2044K ----- libgcc_s-4.8.5-20150702.so.1
00007fda75ee8000      4K r---- libgcc_s-4.8.5-20150702.so.1
00007fda75ee9000      4K rw--- libgcc_s-4.8.5-20150702.so.1
00007fda75eea000    932K r-x-- libstdc++.so.6.0.19
00007fda75fd3000   2044K ----- libstdc++.so.6.0.19
00007fda761d2000     32K r---- libstdc++.so.6.0.19
00007fda761da000      8K rw--- libstdc++.so.6.0.19
00007fda761dc000     84K rw---   [ anon ]
00007fda761f1000     32K r-x-- libyajl.so.2.0.4
00007fda761f9000   2048K ----- libyajl.so.2.0.4
00007fda763f9000      4K r---- libyajl.so.2.0.4
00007fda763fa000      4K rw--- libyajl.so.2.0.4
00007fda763fb000     16K r-x-- libfuzzy.so.2.1.0
00007fda763ff000   2044K ----- libfuzzy.so.2.1.0
00007fda765fe000      4K r---- libfuzzy.so.2.1.0
00007fda765ff000      4K rw--- libfuzzy.so.2.1.0
00007fda76600000    176K r-x-- liblua-5.1.so
00007fda7662c000   2044K ----- liblua-5.1.so
00007fda7682b000      8K r---- liblua-5.1.so
00007fda7682d000      4K rw--- liblua-5.1.so
00007fda7682e000     80K r-x-- liblmdb.so.0.0.0
00007fda76842000   2044K ----- liblmdb.so.0.0.0
00007fda76a41000      4K r---- liblmdb.so.0.0.0
00007fda76a42000      4K rw--- liblmdb.so.0.0.0
00007fda76a43000   1404K r-x-- libxml2.so.2.9.1
00007fda76ba2000   2044K ----- libxml2.so.2.9.1
00007fda76da1000     32K r---- libxml2.so.2.9.1
00007fda76da9000      8K rw--- libxml2.so.2.9.1
00007fda76dab000      8K rw---   [ anon ]
00007fda76dad000     28K r-x-- librt-2.17.so
00007fda76db4000   2044K ----- librt-2.17.so
00007fda76fb3000      4K r---- librt-2.17.so
00007fda76fb4000      4K rw--- librt-2.17.so
00007fda76fb5000    184K r-x-- libGeoIP.so.1.5.0
00007fda76fe3000   2044K ----- libGeoIP.so.1.5.0
00007fda771e2000      4K r---- libGeoIP.so.1.5.0
00007fda771e3000      8K rw--- libGeoIP.so.1.5.0
00007fda771e5000    408K r-x-- libcurl.so.4.3.0
00007fda7724b000   2048K ----- libcurl.so.4.3.0
00007fda7744b000      8K r---- libcurl.so.4.3.0
00007fda7744d000      4K rw--- libcurl.so.4.3.0
00007fda7744e000      4K rw---   [ anon ]
00007fda7744f000   2148K r-x-- libmodsecurity.so.3.0.4
00007fda77668000   2048K ----- libmodsecurity.so.3.0.4
00007fda77868000    192K r---- libmodsecurity.so.3.0.4
00007fda77898000     12K rw--- libmodsecurity.so.3.0.4
00007fda7789b000     20K r-x-- ngx_http_modsecurity_module.so
00007fda778a0000   2044K ----- ngx_http_modsecurity_module.so
00007fda77a9f000      4K r---- ngx_http_modsecurity_module.so
00007fda77aa0000      4K rw--- ngx_http_modsecurity_module.so
00007fda77aa1000      8K r-x-- ngx_http_brotli_static_module.so
00007fda77aa3000   2044K ----- ngx_http_brotli_static_module.so
00007fda77ca2000      4K r---- ngx_http_brotli_static_module.so
00007fda77ca3000      4K rw--- ngx_http_brotli_static_module.so
00007fda77ca4000   1028K r-x-- libm-2.17.so
00007fda77da5000   2044K ----- libm-2.17.so
00007fda77fa4000      4K r---- libm-2.17.so
00007fda77fa5000      4K rw--- libm-2.17.so
00007fda77fa6000    672K r-x-- ngx_http_brotli_filter_module.so
00007fda7804e000   2044K ----- ngx_http_brotli_filter_module.so
00007fda7824d000      4K r---- ngx_http_brotli_filter_module.so
00007fda7824e000      4K rw--- ngx_http_brotli_filter_module.so
00007fda7824f000     48K r-x-- libnss_files-2.17.so
00007fda7825b000   2044K ----- libnss_files-2.17.so
00007fda7845a000      4K r---- libnss_files-2.17.so
00007fda7845b000      4K rw--- libnss_files-2.17.so
00007fda7845c000     24K rw---   [ anon ]
00007fda78462000    144K r-x-- libselinux.so.1
00007fda78486000   2044K ----- libselinux.so.1
00007fda78685000      4K r---- libselinux.so.1
00007fda78686000      4K rw--- libselinux.so.1
00007fda78687000      8K rw---   [ anon ]
00007fda78689000     88K r-x-- libresolv-2.17.so
00007fda7869f000   2048K ----- libresolv-2.17.so
00007fda7889f000      4K r---- libresolv-2.17.so
00007fda788a0000      4K rw--- libresolv-2.17.so
00007fda788a1000      8K rw---   [ anon ]
00007fda788a3000     12K r-x-- libkeyutils.so.1.5
00007fda788a6000   2044K ----- libkeyutils.so.1.5
00007fda78aa5000      4K r---- libkeyutils.so.1.5
00007fda78aa6000      4K rw--- libkeyutils.so.1.5
00007fda78aa7000     56K r-x-- libkrb5support.so.0.1
00007fda78ab5000   2048K ----- libkrb5support.so.0.1
00007fda78cb5000      4K r---- libkrb5support.so.0.1
00007fda78cb6000      4K rw--- libkrb5support.so.0.1
00007fda78cb7000    196K r-x-- libk5crypto.so.3.1
00007fda78ce8000   2044K ----- libk5crypto.so.3.1
00007fda78ee7000      8K r---- libk5crypto.so.3.1
00007fda78ee9000      4K rw--- libk5crypto.so.3.1
00007fda78eea000     12K r-x-- libcom_err.so.2.1
00007fda78eed000   2044K ----- libcom_err.so.2.1
00007fda790ec000      4K r---- libcom_err.so.2.1
00007fda790ed000      4K rw--- libcom_err.so.2.1
00007fda790ee000    868K r-x-- libkrb5.so.3.3
00007fda791c7000   2044K ----- libkrb5.so.3.3
00007fda793c6000     56K r---- libkrb5.so.3.3
00007fda793d4000     12K rw--- libkrb5.so.3.3
00007fda793d7000    296K r-x-- libgssapi_krb5.so.2.2
00007fda79421000   2048K ----- libgssapi_krb5.so.2.2
00007fda79621000      4K r---- libgssapi_krb5.so.2.2
00007fda79622000      8K rw--- libgssapi_krb5.so.2.2
00007fda79624000      8K r-x-- libfreebl3.so
00007fda79626000   2044K ----- libfreebl3.so
00007fda79825000      4K r---- libfreebl3.so
00007fda79826000      4K rw--- libfreebl3.so
00007fda79827000   1804K r-x-- libc-2.17.so
00007fda799ea000   2048K ----- libc-2.17.so
00007fda79bea000     16K r---- libc-2.17.so
00007fda79bee000      8K rw--- libc-2.17.so
00007fda79bf0000     20K rw---   [ anon ]
00007fda79bf5000     84K r-x-- libz.so.1.2.7
00007fda79c0a000   2044K ----- libz.so.1.2.7
00007fda79e09000      4K r---- libz.so.1.2.7
00007fda79e0a000      4K rw--- libz.so.1.2.7
00007fda79e0b000   2264K r-x-- libcrypto.so.1.0.2k
00007fda7a041000   2048K ----- libcrypto.so.1.0.2k
00007fda7a241000    112K r---- libcrypto.so.1.0.2k
00007fda7a25d000     52K rw--- libcrypto.so.1.0.2k
00007fda7a26a000     16K rw---   [ anon ]
00007fda7a26e000    412K r-x-- libssl.so.1.0.2k
00007fda7a2d5000   2048K ----- libssl.so.1.0.2k
00007fda7a4d5000     16K r---- libssl.so.1.0.2k
00007fda7a4d9000     28K rw--- libssl.so.1.0.2k
00007fda7a4e0000    384K r-x-- libpcre.so.1.2.0
00007fda7a540000   2048K ----- libpcre.so.1.2.0
00007fda7a740000      4K r---- libpcre.so.1.2.0
00007fda7a741000      4K rw--- libpcre.so.1.2.0
00007fda7a742000     32K r-x-- libcrypt-2.17.so
00007fda7a74a000   2044K ----- libcrypt-2.17.so
00007fda7a949000      4K r---- libcrypt-2.17.so
00007fda7a94a000      4K rw--- libcrypt-2.17.so
00007fda7a94b000    184K rw---   [ anon ]
00007fda7a979000     92K r-x-- libpthread-2.17.so
00007fda7a990000   2044K ----- libpthread-2.17.so
00007fda7ab8f000      4K r---- libpthread-2.17.so
00007fda7ab90000      4K rw--- libpthread-2.17.so
00007fda7ab91000     16K rw---   [ anon ]
00007fda7ab95000      8K r-x-- libdl-2.17.so
00007fda7ab97000   2048K ----- libdl-2.17.so
00007fda7ad97000      4K r---- libdl-2.17.so
00007fda7ad98000      4K rw--- libdl-2.17.so
00007fda7ad99000    136K r-x-- ld-2.17.so
00007fda7adc6000   1600K rwx--   [ anon ]
00007fda7af57000    256K rwx--   [ anon ]
00007fda7af97000     64K rwx--   [ anon ]
00007fda7afa7000     36K rw---   [ anon ]
00007fda7afb7000      4K rw-s- zero (deleted)
00007fda7afb8000      4K rw-s-   [ shmid=0x16 ]
00007fda7afb9000      4K rw---   [ anon ]
00007fda7afba000      4K r---- ld-2.17.so
00007fda7afbb000      4K rw--- ld-2.17.so
00007fda7afbc000      4K rw---   [ anon ]
00007ffe7bef6000    132K rw---   [ stack ]
00007ffe7bfea000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]
 total          1241212K
[root@web ~]#

the interesting part is:

000055e51660e000 346604K rw---   [ anon ]
000055e52b889000 607984K rw---   [ anon ]
00007fda69936000  89792K rwx--   [ anon ]

there is almost 1.1GB of virtual mem.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288511,288567#msg-288567

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

Francis Daly
On Mon, Jul 06, 2020 at 04:17:46AM -0400, Evald80 wrote:

Hi there,

> [root@web ~]# systemctl status nginx
> ● nginx.service - nginx - high performance web server
>    Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor
> preset: disabled)

What is the history of your nginx install?

Where did this /usr/lib/systemd/system/nginx.service come from?

This shows:

>   ExecStop=/bin/sh -c /bin/kill -s TERM $(/bin/cat
> /var/run/nginx.pid) (code=exited, status=0/SUCCESS)

and TERM is probably not the best signal to say "please close down
tidily".

https://www.nginx.com/resources/wiki/start/topics/examples/systemd/
uses QUIT.

Your (lack of) nginx -V output makes me think that you are running a
random third party module that is not exiting cleanly.

I don't know if you can see any pattern in your access.log for the request
that was handled just before your nginx stopped responding? That might
help make a reproduction case, which might in turn help isolate where
the problem is.

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

vergil
Hi,
this is a vm created from the cloud provider. Than i just added the repo of
nginx, the official one.
Than did regular upgrades. Today also i run the yum update and got the
1.19.1 version of nginx.

[root@web ~]# nginx -V
nginx version: nginx/1.19.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx
--with-compat --with-file-aio --with-threads --with-http_addition_module
--with-http_auth_request_module --with-http_dav_module
--with-http_flv_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_mp4_module
--with-http_random_index_module --with-http_realip_module
--with-http_secure_link_module --with-http_slice_module
--with-http_ssl_module --with-http_stub_status_module --with-http_sub_module
--with-http_v2_module --with-mail --with-mail_ssl_module --with-stream
--with-stream_realip_module --with-stream_ssl_module
--with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC'
--with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288511,288631#msg-288631

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

Francis Daly
On Wed, Jul 08, 2020 at 06:10:14AM -0400, Evald80 wrote:

Hi there,

> this is a vm created from the cloud provider. Than i just added the repo of
> nginx, the official one.
> Than did regular upgrades. Today also i run the yum update and got the
> 1.19.1 version of nginx.

Thanks for the information.

Curious -- the "ExecStop" command in that systemd.service file does
use TERM ("Quick shutdown") where the other suggested one uses QUIT
("Graceful shutdown").

Either way -- that presumably is not the cause of the failure you see.

You showed that you load four modules. Can you see where those modules
came from?

http://nginx.org/packages/mainline/centos/7/x86_64/RPMS/ does include a
nginx-module-geoip-1.19.1-1.el7.ngx.x86_64.rpm but I don't know if that
matches the one you are using.

Might it be that the combination of modules you load has some
incompatibility with the running nginx? I'm afraid I don't have a better
suggestion that "the usual" of trying to find a reproduction recipe,
and then turning off bits to see what causes the failure to go away.

Did you fetch pre-built module.so files, or did you build them yourself?

Good luck with the investigations! (Presuming that the update to 1.19.1
didn't make it all just start working.)

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

vergil
Hi,

The issue appeared again. I checked the memory and it is the same as
previously. Something(within the nginx process) is eating the whole memory
on the server or the nginx itself(who knows).

In order to troubleshoot. The strategy will be to remove all the  modules
and leave only ModSec and wait for at least one week. I saw that in less
than a week, usually it appears...ModSec module is built by me and it is the
latest version on git.

#load_module modules/ngx_http_brotli_filter_module.so;
#load_module modules/ngx_http_brotli_static_module.so;
load_module modules/ngx_http_modsecurity_module.so;
#load_module modules/ngx_http_geoip2_module.so;


[root@web ~]# nginx -V
nginx version: nginx/1.19.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx
--with-compat --with-file-aio --with-threads --with-http_addition_module
--with-http_auth_request_module --with-http_dav_module
--with-http_flv_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_mp4_module
--with-http_random_index_module --with-http_realip_module
--with-http_secure_link_module --with-http_slice_module
--with-http_ssl_module --with-http_stub_status_module --with-http_sub_module
--with-http_v2_module --with-mail --with-mail_ssl_module --with-stream
--with-stream_realip_module --with-stream_ssl_module
--with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC'
--with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
[root@web ~]#

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288511,288672#msg-288672

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

vergil
Problem appeared again.
Well i have to disable the last module which is mod_sec and see again.
If the issue appears again, can we say that we have a memory leak on nginx?

I have noticed that

Currrent config is the following.

[root@web ~]# nginx -V
nginx version: nginx/1.19.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx
--with-compat --with-file-aio --with-threads --with-http_addition_module
--with-http_auth_request_module --with-http_dav_module
--with-http_flv_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_mp4_module
--with-http_random_index_module --with-http_realip_module
--with-http_secure_link_module --with-http_slice_module
--with-http_ssl_module --with-http_stub_status_module --with-http_sub_module
--with-http_v2_module --with-mail --with-mail_ssl_module --with-stream
--with-stream_realip_module --with-stream_ssl_module
--with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC'
--with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'
[root@web ~]#

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288511,288713#msg-288713

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Found Nginx 1.19.0 stopped but no idea what happened

vergil
Seems the problem is with the modsec module. After disabling it, the problem
disappeared...
Also nginx -t command is very slow with modsec module enabled. Pretty
strange that nobody has encountered this issue. Mos probably people are not
running modsec and websites on the same installation of nginx.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,288511,288950#msg-288950

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx