blank page cached ONLY for homepage URL on Wordpress when using keyword monitoring

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

blank page cached ONLY for homepage URL on Wordpress when using keyword monitoring

rnburn
Hi all, new to this awesome community, and would greatly appreciate some
help. I love Nginx but for months I've been trying to tackle a very strange
issue where fastcgi is caching a blank page when monitoring tools like
Monitis or Uptime Robot run keyword-based uptime monitors for the root URL
only. In this case the site is http://musikandfilm.com. All child pages do
not experience the blank page issue. It only occurs for
http://musikandfilm.com. (However, child pages do sometimes intermittently
get blank pages cached if I leave off the "/" at the end of the URL field in
my monitor settings, so I think this may have something to do with the
issue.)

When I delete the cached page and visit the homepage in my browser, I see
fastcgi cache the page correctly. But when I delete the cached page and
allow the monitors to hit the site first, a blank home page is cached that
looks like the example below, even though I get a 200 response. I believe
that the Monitors are requesting the homepage in a certain way that is
causing fastcgi to cache the page incorrectly, but I don't know exactly
what's happening. Here's an example of what's cached:


   
`^C^@^@^@^@^@^@^@<88>o^OY^@^@^@^@ÿÿÿÿÿÿÿÿxa^OY^@^@^@^@C\R^N^@^@¯^@á^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
    KEY: httpGETmusikandfilm.com/
    ^A^F^@^A^@*^F^@Content-type: text/html; charset=UTF-8^M
    ^M`

And that's it, there's no body after this point as there would be if
correctly cached.

I've been chasing this problem for so long I will gladly pay someone to fix
this. I'm not even joking! I've checked my NGINX configs a hundred times at
this point and followed all the recommendations from RT, NGINX, and
elsewhere for NGINX+Wordpress setups, and I have failed thus far. I've spent
at least 100 hours researching this, so if anyone is willing to help it
would so so appreciated. I'm running Ubuntu 16.04, by the way. I'll post my
NGINX configs below:

============================================
======= sites-available/default config: =======
============================================

#move next 4 lines to /etc/nginx/nginx.conf if you want to use fastcgi_cache
across many sites
    fastcgi_cache_path /var/www/html/nginx-cache levels=1:2
keys_zone=WORDPRESS:100m inactive=60m;
    #fastcgi_cache_key "$scheme$request_method$http_host$request_uri";
    fastcgi_cache_use_stale error timeout invalid_header http_500;
    fastcgi_ignore_headers Cache-Control Expires Set-Cookie;

    server {
            listen 80;
            listen [::]:80;
           
            server_name musikandfilm.com www.musikandfilm.com;

    access_log   /var/log/nginx/musikandfilm.com.access.log;
    error_log    /var/log/nginx/musikandfilm.error.log;

            root /var/www/html;
            index index.php;

    set $skip_cache 0;


    # POST requests and urls with a query string should always go to PHP
    if ($request_method = POST) {
    set $skip_cache 1;
    }  
    if ($query_string != "") {
    set $skip_cache 1;
    }  

    # Don't cache uris containing the following segments
    if ($request_uri ~*
"/wp-admin/|/xmlrpc.php|wp-.*.php|/feed/|index.php|sitemap(_index)?.xml") {
    set $skip_cache 1;
    }  

    # Don't use the cache for logged in users or recent commenters
    if ($http_cookie ~*
"comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_no_cache|wordpress_logged_in")
{
    set $skip_cache 1;
    }
   
    autoindex off;
     
            location ~ /purge(/.*) {
                    fastcgi_cache_purge WORDPRESS
"$scheme$request_method$host$1";
            }
     
   
            location ~*
^.+\.(flv|pdf|avi|mov|mp3|wmv|m4v|webm|aac|ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$
{
    expires max;
                    log_not_found off;
                    access_log off;
            }
     
            location / {
                    try_files $uri $uri/ /index.php?$args;
            }
     
            location ~ \.php$ {
    try_files $uri /index.php;
    include fastcgi_params;
    include /etc/nginx/snippets/fastcgi-php.conf;
    fastcgi_pass unix:/run/php/php7.0-fpm.sock;
                    if ($request_method = HEAD) {
                   set $skip_cache 1;
            }
    fastcgi_cache_bypass $skip_cache;
                    fastcgi_no_cache $skip_cache;
    fastcgi_cache WORDPRESS;
                    fastcgi_cache_valid 200 60m;
            }
     
            location ~* ^/wp-includes/.*(?<!(js/tinymce/wp-tinymce))\.php$
{
                    internal;
            }
     
            location = /favicon.ico {
                    log_not_found off;
                    access_log off;
            }
     
            location = /robots.txt {
                    access_log off;
                    log_not_found off;
            }
     
            location = /wp-config.php {
                    deny all;
            }
     
            location ~* /(?:uploads|files)/.*\.php$ {
                    deny all;
            }
     
            location ~*
^/wp-content/.*\.(txt|md|exe|sh|bak|inc|php|pot|po|mo|log|sql)$ {
                    deny all;
            }
     
            location ~ /\.(ht|svn)? {
                    deny all;
            }
    }

============================================
=======  /etc/nginx/nginx.conf =======
============================================

    `user www-data;
    worker_processes auto;
    worker_rlimit_nofile 100000;
    pid /run/nginx.pid;

    events {
    worker_connections 4096;
    multi_accept on;
    }

    http {
    ##
    # EasyEngine Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 30;
    types_hash_max_size 2048;

    server_tokens off;
    reset_timedout_connection on;
    # add_header X-Powered-By "EasyEngine";
    # add_header rt-Fastcgi-Cache $upstream_cache_status;

    # Limit Request
    limit_req_status 403;
    limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;

    # Proxy Settings
    # set_real_ip_from proxy-server-ip;
    # real_ip_header X-Forwarded-For;

    fastcgi_read_timeout 300;
    client_max_body_size 100m;

    ##
    # SSL Settings
    ##

    ssl_session_cache shared:SSL:20m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    ssl_ciphers
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    ##
    # Basic Settings
    ##
    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    # Log format Settings
    log_format rt_cache '$remote_addr $upstream_response_time
$upstream_cache_status [$time_local] '
    '$http_host "$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent"';

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types
       application/atom+xml
       application/javascript
       application/json
       application/rss+xml
       application/vnd.ms-fontobject
       application/x-font-ttf
       application/x-web-app-manifest+json
       application/xhtml+xml
       application/xml
       font/opentype
       image/svg+xml
       image/x-icon
       text/css
       text/plain
       text/x-component
       text/xml
       text/javascript;
    ##
    # Cache Settings
    ##
           
    fastcgi_cache_key "$scheme$request_method$http_host$request_uri";
    #add_header Fastcgi-Cache $upstream_cache_status;
    #add_header Fastcgi-method $request_method;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
    }


    #mail {
    # # See sample authentication script at:
    # # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
    #
    # # auth_http localhost/auth.php;
    # # pop3_capabilities "TOP" "USER";
    # # imap_capabilities "IMAP4rev1" "UIDPLUS";
    #
    # server {
    # listen     localhost:110;
    # protocol   pop3;
    # proxy      on;
    # }
    #
    # server {
    # listen     localhost:143;
    # protocol   imap;
    # proxy      on;
    # }
    #}`

============================================
=======  fastcgi_params =======
============================================

    fastcgi_param   QUERY_STRING            $query_string;
    fastcgi_param   REQUEST_METHOD          $request_method;
    fastcgi_param   REQUEST_SCHEME          $scheme;
    fastcgi_param   CONTENT_TYPE            $content_type;
    fastcgi_param   CONTENT_LENGTH          $content_length;
    fastcgi_param HOST $host;

    #fastcgi_param   SCRIPT_FILENAME        
$document_root$fastcgi_script_name;
    fastcgi_param   SCRIPT_FILENAME             $request_filename;
    fastcgi_param   SCRIPT_NAME             $fastcgi_script_name;
    fastcgi_param   PATH_INFO               $fastcgi_path_info;
    fastcgi_param       PATH_TRANSLATED        
$document_root$fastcgi_path_info;
    fastcgi_param   REQUEST_URI             $request_uri;
    fastcgi_param   DOCUMENT_URI            $document_uri;
    fastcgi_param   DOCUMENT_ROOT           $document_root;
    fastcgi_param   SERVER_PROTOCOL         $server_protocol;

    fastcgi_param   GATEWAY_INTERFACE       CGI/1.1;
    fastcgi_param   SERVER_SOFTWARE         nginx/$nginx_version;

    fastcgi_param   REMOTE_ADDR             $remote_addr;
    fastcgi_param   REMOTE_PORT             $remote_port;
    fastcgi_param   SERVER_ADDR             $server_addr;
    fastcgi_param   SERVER_PORT             $server_port;
    fastcgi_param   SERVER_NAME             $server_name;

    fastcgi_param   HTTPS                   $https;

    # PHP only, required if PHP was built with --enable-force-cgi-redirect
    fastcgi_param   REDIRECT_STATUS         200;

    # To fix CGI application vulnerability - https://httpoxy.org
    fastcgi_param   HTTP_PROXY      "";

============================================
=======  fastcgi-php.conf =======
============================================

    # regex to split $uri to $fastcgi_script_name and $fastcgi_path
    fastcgi_split_path_info ^(.+\.php)(/.+)$;

    # Check that the PHP script exists before passing it
    #try_files $fastcgi_script_name =404;

    # Bypass the fact that try_files resets $fastcgi_path_info
    # see: http://trac.nginx.org/nginx/ticket/321
    set $path_info $fastcgi_path_info;
    fastcgi_param PATH_INFO $path_info;

    fastcgi_index index.php;
    include fastcgi.conf;

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274097,274097#msg-274097

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: blank page cached ONLY for homepage URL on Wordpress when using keyword monitoring

Francis Daly
On Mon, May 08, 2017 at 01:59:43AM -0400, seth2958 wrote:

Hi there,

I do not have an answer for you, but I have some suggestions which might
help you identify the problem and solution.

> fastcgi is caching a blank page when monitoring tools like
> Monitis or Uptime Robot run keyword-based uptime monitors for the root URL
> only.

Can you use tcpdump or something similar to identify the exact request
that is made? (Including all of the headers that are sent.) If it reliably
fails, then you should be able to construct an equivalent "curl" command
to make the same request, and see it fail yourself.

>             location ~ /purge(/.*) {
>                     fastcgi_cache_purge WORDPRESS
> "$scheme$request_method$host$1";

I think that that says that you are using nginx-plus, in which case
there may be a better place to send your request for a timely response.

>             location ~ \.php$ {
>                     if ($request_method = HEAD) {
>                    set $skip_cache 1;
>             }

That is "if" inside "location"; it is very easy to go wrong using that
construct. It might be worth considering moving those three lines outside
of all "location" blocks, to match the other similar ones.

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: blank page cached ONLY for homepage URL on Wordpress when using keyword monitoring

rnburn
Thank you for the insights Francis!!

It's too early to tell, but I think moving the "if" statement outside the
location block may have done the trick. I also changed the the statement so
that only GET requests are cached like so:

if ($request_method != GET) {
                set $skip_cache 1;
        }


I haven't pinpointed the exact cause yet, but initial test results are
promising. I set up three times as many keyword monitors and have not seen
an empty cached page yet after deleting the existing cached page and letting
the cached page rebuild. Only the complete page is being cached. Before this
change it was pretty easy for me to break the cache, but now I've been
unable to replicate the problem. My theory is that the monitors are making
some sort of non-GET (but not HEAD) requests which NGINX is for some reason
treating as GET. So hopefully moving the if statement and changing the logic
will be the solution here.

If anyone else is able to confirm the exact issue, I'd still really like to
know.

Oh and another possible effect of these changes...page load time has
decreased by at least 300 milliseconds!

Thanks again!!

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274097,274103#msg-274103

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: blank page cached ONLY for homepage URL on Wordpress when using keyword monitoring

Peter Booth
Seth,

It's actually very easy to reproduce this issue - from a browser request http://musikandfilm.com/?a=b and you will see it. There are a couple of low level tools that expose some possible issues.

If you email me directly I can talk about this in more detail. Try peter underscore booth at me dot com.



Sent from my iPhone

> On May 8, 2017, at 7:17 PM, seth2958 <[hidden email]> wrote:
>
> Thank you for the insights Francis!!
>
> It's too early to tell, but I think moving the "if" statement outside the
> location block may have done the trick. I also changed the the statement so
> that only GET requests are cached like so:
>
> if ($request_method != GET) {
>                set $skip_cache 1;
>        }
>
>
> I haven't pinpointed the exact cause yet, but initial test results are
> promising. I set up three times as many keyword monitors and have not seen
> an empty cached page yet after deleting the existing cached page and letting
> the cached page rebuild. Only the complete page is being cached. Before this
> change it was pretty easy for me to break the cache, but now I've been
> unable to replicate the problem. My theory is that the monitors are making
> some sort of non-GET (but not HEAD) requests which NGINX is for some reason
> treating as GET. So hopefully moving the if statement and changing the logic
> will be the solution here.
>
> If anyone else is able to confirm the exact issue, I'd still really like to
> know.
>
> Oh and another possible effect of these changes...page load time has
> decreased by at least 300 milliseconds!
>
> Thanks again!!
>
> Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274097,274103#msg-274103
>
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: blank page cached ONLY for homepage URL on Wordpress when using keyword monitoring

rnburn
Thanks Peter. I just sent you an email from sethleon at gmail dot com.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,274097,274112#msg-274112

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx