Flush access log buffer

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Flush access log buffer

Gary
When I was using FreeBSD, the access log was real time. Since I went to
Centos, that doesn't seem to be the case. Is there some way to flush
the buffer?
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Flush access log buffer

Gary
On Thu, 22 Feb 2018 18:40:12 -0800
"[hidden email]" <[hidden email]> wrote:

> When I was using FreeBSD, the access log was real time. Since I went
> to Centos, that doesn't seem to be the case. Is there some way to
> flush the buffer?
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

I found a flush=x option on the command line. I set it for 1m for
testing. Note that you need to specify a buffer size else nginx will
choke.

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Flush access log buffer

Gary
On Fri, 23 Feb 2018 18:54:48 -0800
"[hidden email]" <[hidden email]> wrote:

> On Thu, 22 Feb 2018 18:40:12 -0800
> "[hidden email]" <[hidden email]> wrote:
>
> > When I was using FreeBSD, the access log was real time. Since I went
> > to Centos, that doesn't seem to be the case. Is there some way to
> > flush the buffer?
> > _______________________________________________
> > nginx mailing list
> > [hidden email]
> > http://mailman.nginx.org/mailman/listinfo/nginx 
>
> I found a flush=x option on the command line. I set it for 1m for
> testing. Note that you need to specify a buffer size else nginx will
> choke.
>
> _______________________________________________

This flush=time option isn't working. I'm at a loss here.

Here is some of a ls -l:
-rw-r----- 1 nginx adm    12936 Feb 27 02:17 access.log
-rw-r--r-- 1 nginx root    4760 Feb 24 03:06 access.log-20180224.gz
-rw-r----- 1 nginx adm  1738667 Feb 26 03:21 access.log-20180226

This is the ls -l on /var/log/nginx:
drwxr-xr-x. 2 root   root       4096 Feb 27 02:11 nginx

I'm not requesting a compressed log, so I assume centos is creating the
gunzip files. Usually the access.log file has content, but sometimes it
is empty and the log data is on the access.log-"date" file, which I
suspect is a roll over from access.log. That is maybe centos rolls it
but doesn't zip it right away.


http {
    log_format  main  '$status $remote_addr - $remote_user [$time_local] "$request" '
                      '$body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main buffer=32k flush=1m;


uname -a
Linux 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

nginx -V
nginx version: nginx/1.12.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib64/nginx/modules
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
--group=nginx --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_dav_module --with-http_flv_module --with-http_mp4_module
--with-http_gunzip_module --with-http_gzip_static_module
--with-http_random_index_module --with-http_secure_link_module
--with-http_stub_status_module --with-http_auth_request_module
--with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic
--with-http_geoip_module=dynamic --with-http_perl_module=dynamic
--add-dynamic-module=njs-1c50334fbea6/nginx --with-threads
--with-stream --with-stream_ssl_module --with-http_slice_module
--with-mail --with-mail_ssl_module --with-file-aio --with-ipv6
--with-http_v2_module --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'
--with-ld-opt=-Wl,-E



_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Flush access log buffer

oscaretu .
Hello!

If you have installed sysdig, [https://www.sysdig.org/] (a kind of strace but for all the computer, not just for a only process) you can do commands like:

  sysdig fd.name contains .gz

and it will show information about who is accessing any file that contains ".gz" in its name.

root@veve0410:/home/oscar# sysdig proc.name=nginx and fd.name contains access
2828 08:45:18.248862970 1 nginx (28325) > write fd=75(<f>/html/logs/nginx/produccion/portal/access.log) size=331
2829 08:45:18.248867711 1 nginx (28325) < write res=331 data=66.249.79.51 - - [27/Feb/2018:08:45:18 +0100] \"GET /diario/1991/04/10/internacio
15081 08:45:19.538002590 1 nginx (28325) > write fd=75(<f>/html/logs/nginx/produccion/portal/access.log) size=124
15082 08:45:19.538007576 1 nginx (28325) < write res=124 data=104.199.186.40 - - [27/Feb/2018:08:45:19 +0100] \"GET /elpais/portada_america.htm
19211 08:45:19.718872876 1 nginx (28325) > write fd=75(<f>/html/logs/nginx/produccion/portal/access.log) size=332
19212 08:45:19.718877388 1 nginx (28325) < write res=332 data=66.249.79.45 - - [27/Feb/2018:08:45:19 +0100] \"GET /diario/2005/08/23/catalunya/
22775 08:45:20.215718840 1 nginx (28325) > write fd=75(<f>/html/logs/nginx/produccion/portal/access.log) size=330
22776 08:45:20.215723447 1 nginx (28325) < write res=330 data=66.249.79.42 - - [27/Feb/2018:08:45:20 +0100] \"GET /diario/2009/05/23/babelia/12
^Croot@veve0410:/home/oscar#


sysdig can be a great help to watch what is happening in your linux computer.

Here you have other examples of what you can do with sysdig / csysdig (sorry, the explanations are in Spanish):
csysdig    # versión de tipo 'top', desde la que se puede activar la traza de cada proceso
sysdig -h  # ayuda
sysdig -l
sysdig -cl # lista los chisels disponibles. Mira en /usr/share/sysdig/chisels/ los que vienen de serie. Mira en /usr/share/sysdig/chisels/ los que vienen de serie
sysdig -L  # listar los eventos que se pueden capturar
sysdig "proc.name=httpd and evt.type=open and fd.num<0 and evt.dir =<" # comprobar errores al abrir ficheros

sysdig -c spy_ip 10.168.1.100 # Ver la conversación que tiene lugar con esa IP
                              # Si se hace desde un frontal, se ven las peticiones HTTP
                              # hechas por los navegadores y las respuestas del servidor

sudo sysdig -c echo_fds "fd.name not contains /dev/" # Mostrar accesos a ficheros, con cierto filtro adicional
sysdig fd.name contains sitemap                      # Vigilar accesos a ficheros de sitemaps

sysdig proc.name=httpd and proc.pid = 23216
sysdig proc.pid = 23216
sysdig proc.apid = 23216 # procesos cuyo padre sea el proceso de PID 23216
sysdig proc.name=httpd
sysdig -w apache-durante-atasco-nanosleep-al-recibir-SIGHUP.scap proc.name=httpd # está en /html/tmp de veve0223
sysdig -r apache-durante-atasco-nanosleep-al-recibir-SIGHUP.scap                 # reproducir las operaciones guardadas con -w
sysdig -p"%evt.time %evt.arg.name" evt.type=open                                 # mostrar el instante
sysdig -p"%evt.num %evt.arg.name" evt.type=open                                  # mostrar el numero; sirve para luego filtrar un rango por el numero
sysdig -r apache-durante-atasco-nanosleep-al-recibir-SIGHUP.scap  -p"%evt.num %evt.arg.name" evt.type=open # mostrar el numero
sysdig -r apache-durante-atasco-nanosleep-al-recibir-SIGHUP.scap "evt.num > 3362620" | less  # ignorar eventos anteriores a uno dado

sysdig "not evt.type in ('select', 'switch', 'clock_gettime', 'rt_sigprocmask', 'ioctl')" #  es posible que esto no funcione en los servidores, pero sí en mi portátil (versión más reciente)
sysdig proc.name=searchd and evt.type=recvfrom # para que se muestren las IPs y puertos que se conectan al daemon de búsqueda de Sphinx

sysdig -c lsof "fd.type=ipv4"   # equivale a lsof -i que sirve para listar todas las conexiones de red,
                                # aunque con lsof veo que indica si es TCP o UDP. Para separar las que
                                # son TCP o UDP, tienes que ejecutar por separado los dos comandos siguientes
sysdig -c lsof "fd.l4proto=tcp" # Versión restringida a TCP del comando anterior, equivalente a lsof -i tcp
sysdig -c lsof "fd.l4proto=udp" # Versión restringida a UDP del comando anterior, equivalente a lsof -i udp

csysdig -v files                # Ficheros a los que se va accediendo, con refresco de pantalla
csysdig -v file_opens           # Ficheros a los que se va accediento, en modo lista acumulada

Kind regards,
Oscar

On Tue, Feb 27, 2018 at 3:32 AM, [hidden email] <[hidden email]> wrote:
On Fri, 23 Feb 2018 18:54:48 -0800
"[hidden email]" <[hidden email]> wrote:

> On Thu, 22 Feb 2018 18:40:12 -0800
> "[hidden email]" <[hidden email]> wrote:
>
> > When I was using FreeBSD, the access log was real time. Since I went
> > to Centos, that doesn't seem to be the case. Is there some way to
> > flush the buffer?
> > _______________________________________________
> > nginx mailing list
> > [hidden email]
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> I found a flush=x option on the command line. I set it for 1m for
> testing. Note that you need to specify a buffer size else nginx will
> choke.
>
> _______________________________________________

This flush=time option isn't working. I'm at a loss here.

Here is some of a ls -l:
-rw-r----- 1 nginx adm    12936 Feb 27 02:17 access.log
-rw-r--r-- 1 nginx root    4760 Feb 24 03:06 access.log-20180224.gz
-rw-r----- 1 nginx adm  1738667 Feb 26 03:21 access.log-20180226

This is the ls -l on /var/log/nginx:
drwxr-xr-x. 2 root   root       4096 Feb 27 02:11 nginx

I'm not requesting a compressed log, so I assume centos is creating the
gunzip files. Usually the access.log file has content, but sometimes it
is empty and the log data is on the access.log-"date" file, which I
suspect is a roll over from access.log. That is maybe centos rolls it
but doesn't zip it right away.


http {
    log_format  main  '$status $remote_addr - $remote_user [$time_local] "$request" '
                      '$body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main buffer=32k flush=1m;


uname -a
Linux 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

nginx -V
nginx version: nginx/1.12.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib64/nginx/modules
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
--group=nginx --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_dav_module --with-http_flv_module --with-http_mp4_module
--with-http_gunzip_module --with-http_gzip_static_module
--with-http_random_index_module --with-http_secure_link_module
--with-http_stub_status_module --with-http_auth_request_module
--with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic
--with-http_geoip_module=dynamic --with-http_perl_module=dynamic
--add-dynamic-module=njs-1c50334fbea6/nginx --with-threads
--with-stream --with-stream_ssl_module --with-http_slice_module
--with-mail --with-mail_ssl_module --with-file-aio --with-ipv6
--with-http_v2_module --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic'
--with-ld-opt=-Wl,-E



_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Oscar Fernandez Sierra
[hidden email]

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx