Empty error and access log

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Empty error and access log

satay
Dear Team,

I have configured access_log and error_log at https block level and
sometimes(like after log rotation), could see the logs are with 0 bytes
(which means, nothing logged though it serves the requests and application
is accessible).

Any help would be appreciated to figure out the issue and why its occurring
.

Config:

#Log file format
    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" "$gzip_ratio" ';

    access_log /var/logs/access.log main;
    error_log  /var/logs/error.log error;

Log Files:    

-rwxr-xr-x. 1 root root 0 Sep 25 03:13 access.log
-rwxr-xr-x. 1 root root 0 Sep 26 03:34 error.log

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285708,285708#msg-285708

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

satay
Workaround: restarting nginx helps and logs getting generated.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285708,285709#msg-285709

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

Francis Daly
In reply to this post by satay
On Thu, Sep 26, 2019 at 06:22:41AM -0400, krishna wrote:

Hi there,

> I have configured access_log and error_log at https block level and
> sometimes(like after log rotation), could see the logs are with 0 bytes
> (which means, nothing logged though it serves the requests and application
> is accessible).

What, specifically, do you mean by "log rotation"?

If it involves renaming or deleting a file without telling nginx that
the filehandle should be closed and reopened, then it is possible that
nginx is still happily logging to the old filehandle.

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

satay
Hi Francis,

Thanks for the reply.

logrotate has been used to archive the nginx log files on daily basis from
the machine and below is the configuration.
Once it has been executed, the files would get zipped in gzip format and new
files would get created  (as killing the nginx PID, further the nginx log
will be generated).


/var/nginx/logs/*log
{
 daily
 rotate 20
 missingok
 notifempty
 compress
 sharedscripts
 postrotate
 /usr/bin/kill -USR1 `cat /var/pid/nginx.pid 2>/dev/null` 2>/dev/null ||
true #PID file for nginx
 endscript
 }

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285708,285715#msg-285715

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

satay
We recently noticed this on our servers and we were using USR1 for
postrotation which wasn't working as expected.  We changed the post rotation
command to be HUP and this fixed the issues of rotation being borked.

http://nginx.org/en/docs/control.html
It says that USR1 should reopen logfiles after they have been renamed but we
found that this didn't work

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285708,285717#msg-285717

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

Francis Daly
In reply to this post by satay
On Thu, Sep 26, 2019 at 07:25:45AM -0400, krishna wrote:

Hi there,

> logrotate has been used to archive the nginx log files on daily basis from
> the machine and below is the configuration.

>  postrotate
>  /usr/bin/kill -USR1 `cat /var/pid/nginx.pid 2>/dev/null` 2>/dev/null ||
> true #PID file for nginx
>  endscript

That looks like it should work; although the "2>/dev/null" parts will
possibly throw away any useful status message.

http://nginx.org/en/docs/control.html describes what should happen,
on the nginx side.

When I test here, I don't see anything obvious in the error_log that
indicates that USR1 was received.

So from here, it looks to me that if you can describe a repeatable recipe
that causes this unwanted behaviour to happen, then there may be something
within nginx that needs fixing.

(If I do a straightforward

mv logs/access.log logs/access.log.$(date +%s)

then my next request is written to the moved file; and after

kill -USR1 `cat logs/nginx.pid`

then the next request is written to the new access.log.

So a "simple" test here does not fail.)

Cheers,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

Maxim Dounin
In reply to this post by satay
Hello!

On Thu, Sep 26, 2019 at 09:13:20AM -0400, rick_pri wrote:

> We recently noticed this on our servers and we were using USR1 for
> postrotation which wasn't working as expected.  We changed the post rotation
> command to be HUP and this fixed the issues of rotation being borked.
>
> http://nginx.org/en/docs/control.html
> It says that USR1 should reopen logfiles after they have been renamed but we
> found that this didn't work

The USR1 is the right way to ask nginx to reopen log files, and it
is enough for log rotation.  While using HUP is also possible, it
does full configuration reload, and using it for log rotation is
wrong - for example, this may result in accidental use of a
configuration being edited.

If USR1 does not work for you, most likely it is because log files
after log rotation cannot be opened for writing by worker
processes.  Fix is to assing correct access rights to the new
files created during log rotation, and also make sure nginx worker
process are able to access the directory with the log files.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

Maxim Dounin
In reply to this post by satay
Hello!

On Thu, Sep 26, 2019 at 06:22:41AM -0400, krishna wrote:

> Dear Team,
>
> I have configured access_log and error_log at https block level and
> sometimes(like after log rotation), could see the logs are with 0 bytes
> (which means, nothing logged though it serves the requests and application
> is accessible).
>
> Any help would be appreciated to figure out the issue and why its occurring
> .
>
> Config:
>
> #Log file format
>     log_format main '$remote_addr - $remote_user [$time_local] '
>                     '"$request" $status $body_bytes_sent '
>                     '"$http_referer" "$http_user_agent" "$gzip_ratio" ';
>
>     access_log /var/logs/access.log main;
>     error_log  /var/logs/error.log error;
>
> Log Files:    
>
> -rwxr-xr-x. 1 root root 0 Sep 25 03:13 access.log
> -rwxr-xr-x. 1 root root 0 Sep 26 03:34 error.log

These files are only writable by root, hence nginx worker
processes won't be able to open these for writing after log
rotation.  You have to fix your log rotation configuration to
create files which are writable by nginx user.

For example, nginx own packages as available from nginx.org use
the following logrotate configuration
(http://hg.nginx.org/pkg-oss/file/tip/debian/nginx.logrotate):

/var/log/nginx/*.log {
        daily
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 nginx adm
        sharedscripts
        postrotate
                if [ -f /var/run/nginx.pid ]; then
                        kill -USR1 `cat /var/run/nginx.pid`
                fi
        endscript
}

Note the "create 640 nginx adm" line.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

satay
Hello All,

Thanks for the updates.

Figured out that, nginx.pid file doesn't have the correct PID value(it
contains 1)  - which could be valid since nginx is running inside docker
container and we are setting up the log rotation from the host machine where
the log files getting rotated but new files not getting logged further by
nginx worker process.

As per the comments tried to edit the nginx.pid file(with Nginx host PID),
then issued command "kill -USR1 `cat /var/run/nginx.pid`" which works fine
and new log files getting created and logged with data.

Had came across the blogs, they suggested to use nginx reload during post
rotation as below with the logrotate,
 postrotate
     docker exec nginx bash -c "nginx -s reload 2>/dev/null"
 endscript

Kindly let me know, if above is a valid approach can be followed (or)  to
get the actual Nginx PID at host level & initiate kill command with USR1.

Thanks in advance.
Regards,
Krishna

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,285708,285726#msg-285726

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

Francis Daly
In reply to this post by Maxim Dounin
On Thu, Sep 26, 2019 at 06:13:55PM +0300, Maxim Dounin wrote:
> On Thu, Sep 26, 2019 at 06:22:41AM -0400, krishna wrote:

Hi there,

> > -rwxr-xr-x. 1 root root 0 Sep 25 03:13 access.log
> > -rwxr-xr-x. 1 root root 0 Sep 26 03:34 error.log
>
> These files are only writable by root, hence nginx worker
> processes won't be able to open these for writing after log
> rotation.  You have to fix your log rotation configuration to
> create files which are writable by nginx user.

Ah, that was the part that I had missed here too.

Thanks for the extended explanation.

Cheers,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Empty error and access log

Francis Daly
In reply to this post by satay
On Fri, Sep 27, 2019 at 01:36:55AM -0400, krishna wrote:

Hi there,

I do not have the answer for you. But...

> Figured out that, nginx.pid file doesn't have the correct PID value(it
> contains 1)  - which could be valid since nginx is running inside docker
> container and we are setting up the log rotation from the host machine where
> the log files getting rotated but new files not getting logged further by
> nginx worker process.
>
> As per the comments tried to edit the nginx.pid file(with Nginx host PID),
> then issued command "kill -USR1 `cat /var/run/nginx.pid`" which works fine
> and new log files getting created and logged with data.

...what nginx needs is for its master process to receive a "USR1"
signal. It does not care how that is done.

As I understand it, docker includes a "kill" subcommand with a "-s" option
to send a specific signal to a container -- which should mean "to the
one process that is running in the container", which in your case should
be nginx.

> Had came across the blogs, they suggested to use nginx reload during post
> rotation as below with the logrotate,
>  postrotate
>      docker exec nginx bash -c "nginx -s reload 2>/dev/null"
>  endscript
>
> Kindly let me know, if above is a valid approach can be followed (or)  to
> get the actual Nginx PID at host level & initiate kill command with USR1.

I think that there were good reasons explained not to send HUP when USR1
is all that is needed.

nginx writes its pid file to a well-known place. If "something else"
does virtualising or jailing or containerising or namespacing or any
other translation between nginx's idea of its pid and the rest of the
world's idea of the nginx pid, then it is that "something else's" job
to untranslate as well.

Which means: if you use docker to hide nginx from the system, you should
use docker to expose nginx to the system.

Ask docker for the "real" pid; or use docker to avoid having to find
the pid.

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx