Operation not permitted) while reading upstream - error log is filled with tons of these

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Operation not permitted) while reading upstream - error log is filled with tons of these

Friscia, Michael

I’m stumped and have exhausted pretty much every google search I could come up with on this. I have NGINX setup for caching. I’m using Ubuntu 17.10 and Nginx 1.12.1. Everything appears to be working just fine by the error log is getting filled with errors like this

 

2018/01/20 07:37:44 [crit] 122598#122598: *91 chmod() "/etc/nginx/cache/nginx2/main/a5/d8/e0/72677057b97aef4eee8e619c49e0d8a5.0000000013" failed (1: Operation not permitted) while reading upstream, client: 35.226.23.240, server: *.---.com, request: "GET /search/----.profile HTTP/1.1", upstream: "http://---.---.---.---:80/search/----.profile", host: "---.com"

 

From what I can tell, everything is working. Nginx must have permission, I deleted the cache folder and let it make on restart.

 

The actual setup is that this is on Azure, the storage location /etc/nginx/cache/nginx2 is a mounted file share from Azure storage. I then created the folder /main/ inside that, this is the folder I deleted and on restart nginx created. Which makes me think that there is no permission problem but instead some other underlying problem that is throwing this false error. The cache gzip files are being made and the folder path for 2:2:2 is being created.

 

Has anyone seen/solved this before?

 

___________________________________________

Michael Friscia

Office of Communications

Yale School of Medicine

(203) 737-7932 - office

(203) 544-3282 – mobile

http://web.yale.edu

 


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Operation not permitted) while reading upstream - error log is filled with tons of these

Aziz Rozyev
As usual, try disabling selinux if it’s active. If that helps, investigate audit log..

br,
Aziz.





> On 20 Jan 2018, at 15:47, Friscia, Michael <[hidden email]> wrote:
>
> I’m stumped and have exhausted pretty much every google search I could come up with on this. I have NGINX setup for caching. I’m using Ubuntu 17.10 and Nginx 1.12.1. Everything appears to be working just fine by the error log is getting filled with errors like this
>  
> 2018/01/20 07:37:44 [crit] 122598#122598: *91 chmod() "/etc/nginx/cache/nginx2/main/a5/d8/e0/72677057b97aef4eee8e619c49e0d8a5.0000000013" failed (1: Operation not permitted) while reading upstream, client: 35.226.23.240, server: *.---.com, request: "GET /search/----.profile HTTP/1.1", upstream: "http://---.---.---.---:80/search/----.profile", host: "---.com"
>  
> From what I can tell, everything is working. Nginx must have permission, I deleted the cache folder and let it make on restart.
>  
> The actual setup is that this is on Azure, the storage location /etc/nginx/cache/nginx2 is a mounted file share from Azure storage. I then created the folder /main/ inside that, this is the folder I deleted and on restart nginx created. Which makes me think that there is no permission problem but instead some other underlying problem that is throwing this false error. The cache gzip files are being made and the folder path for 2:2:2 is being created.
>  
> Has anyone seen/solved this before?
>  
> ___________________________________________
> Michael Friscia
> Office of Communications
> Yale School of Medicine
> (203) 737-7932 - office
> (203) 544-3282 – mobile
> http://web.yale.edu
>  
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Operation not permitted) while reading upstream - error log is filled with tons of these

Friscia, Michael
Ok, found it. It was a permission issue. I did not have the uid and gid in the fstab, I set that the the www-data user that nginx is running as and the errors went away. I’m just not sure I understand how it made the cache files/directories in the first place. But clearly this is a problem when the cache is on a CIFS mount. I reverted to have the cache write to a local drive and there was no problem.

So I ended up on this page which led to the solution
https://stackoverflow.com/questions/21318573/permissions-issue-on-cifs-mount-between-ubuntu-and-mavericks

Thank you for the suggestion, that’s actually where I started after you mentioned it to end up where I did.
___________________________________________
Michael Friscia
Office of Communications
Yale School of Medicine
(203) 737-7932 - office
(203) 544-3282 – mobile
http://web.yale.edu <http://web.yale.edu/>
 

On 1/20/18, 8:26 AM, "nginx on behalf of Aziz Rozyev" <[hidden email] on behalf of [hidden email]> wrote:

    As usual, try disabling selinux if it’s active. If that helps, investigate audit log..
   
    br,
    Aziz.
   
   
   
   
   
    > On 20 Jan 2018, at 15:47, Friscia, Michael <[hidden email]> wrote:
    >
    > I’m stumped and have exhausted pretty much every google search I could come up with on this. I have NGINX setup for caching. I’m using Ubuntu 17.10 and Nginx 1.12.1. Everything appears to be working just fine by the error log is getting filled with errors like this
    >  
    > 2018/01/20 07:37:44 [crit] 122598#122598: *91 chmod() "/etc/nginx/cache/nginx2/main/a5/d8/e0/72677057b97aef4eee8e619c49e0d8a5.0000000013" failed (1: Operation not permitted) while reading upstream, client: 35.226.23.240, server: *.---.com, request: "GET /search/----.profile HTTP/1.1", upstream: "https://urldefense.proofpoint.com/v2/url?u=http-3A__-2D-2D-2D.-2D-2D-2D.-2D-2D-2D.-2D-2D-2D-3A80_search_-2D-2D-2D-2D.profile&d=DwIGaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=M4doNhMd3t6V64zd-c0vWeBW7Z5jUH7dBvAM-8w3vDg&s=osyFFwk-WV5Vfm0y8mJYBaHvoETSeWkW5-r3PlhFAPw&e=", host: "---.com"
    >  
    > From what I can tell, everything is working. Nginx must have permission, I deleted the cache folder and let it make on restart.
    >  
    > The actual setup is that this is on Azure, the storage location /etc/nginx/cache/nginx2 is a mounted file share from Azure storage. I then created the folder /main/ inside that, this is the folder I deleted and on restart nginx created. Which makes me think that there is no permission problem but instead some other underlying problem that is throwing this false error. The cache gzip files are being made and the folder path for 2:2:2 is being created.
    >  
    > Has anyone seen/solved this before?
    >  
    > ___________________________________________
    > Michael Friscia
    > Office of Communications
    > Yale School of Medicine
    > (203) 737-7932 - office
    > (203) 544-3282 – mobile
    > http://web.yale.edu
    >  
    > _______________________________________________
    > nginx mailing list
    > [hidden email]
    > https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.nginx.org_mailman_listinfo_nginx&d=DwIGaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=M4doNhMd3t6V64zd-c0vWeBW7Z5jUH7dBvAM-8w3vDg&s=V3XuCla575XErTF3p9z7MMFi6tKt7MrpiCo8rYhuIIU&e=
   
    _______________________________________________
    nginx mailing list
    [hidden email]
    https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.nginx.org_mailman_listinfo_nginx&d=DwIGaQ&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=M4doNhMd3t6V64zd-c0vWeBW7Z5jUH7dBvAM-8w3vDg&s=V3XuCla575XErTF3p9z7MMFi6tKt7MrpiCo8rYhuIIU&e=

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Operation not permitted) while reading upstream - error log is filled with tons of these

Francis Daly
In reply to this post by Friscia, Michael
On Sat, Jan 20, 2018 at 12:47:29PM +0000, Friscia, Michael wrote:

Hi there,

> 2018/01/20 07:37:44 [crit] 122598#122598: *91 chmod() "/etc/nginx/cache/nginx2/main/a5/d8/e0/72677057b97aef4eee8e619c49e0d8a5.0000000013" failed (1: Operation not permitted) while reading upstream, client: 35.226.23.240, server: *.---.com, request: "GET /search/----.profile HTTP/1.1", upstream: "http://---.---.---.---:80/search/----.profile", host: "---.com"

> The actual setup is that this is on Azure, the storage location /etc/nginx/cache/nginx2 is a mounted file share from Azure storage.

How precisely is the file share mounted?

The error indication is that chmod() failed.

Does chmod() work on that filesystem?

Can you change the mount options such that chmod() will return success,
even if it does not actually change the permissions of the file?

https://superuser.com/questions/744384/allow-chmod-on-cifs-mount suggests
that "noperm" in the mount options had the desired effect for one user;
note that that does not seem (to me) to match what the "mount.cifs"
documentation says, so you'll want to do your own investigation and
testing before relying on it.

> Has anyone seen/solved this before?

I haven't. So the above is "questions" rather than "suggestions how to fix".

Good luck with it,

        f
--
Francis Daly        [hidden email]
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Operation not permitted) while reading upstream - error log is filled with tons of these

Friscia, Michael
I tried appling the noperm flag on my test environment, but once I applied the other fix of adding gid/uid to the cifs mount, the chmod() error went away. Now I face a new error that happens infrequently which is rename() fails. I’m guessing the real problem is that without a lot of additional configuration, the nginx cache may not work properly with a cifs mount. My problem is that I’ve forgotten more linux than I remember so I’m stumbling through these errors as they come. But if anyone has seen the rename() error and knows a fix, that would help.

___________________________________________
Michael Friscia
Office of Communications
Yale School of Medicine
(203) 737-7932 - office
(203) 544-3282 – mobile
http://web.yale.edu <http://web.yale.edu/>
 

On 1/20/18, 8:58 AM, "nginx on behalf of Francis Daly" <[hidden email] on behalf of [hidden email]> wrote:

    On Sat, Jan 20, 2018 at 12:47:29PM +0000, Friscia, Michael wrote:
   
    Hi there,
   
    > 2018/01/20 07:37:44 [crit] 122598#122598: *91 chmod() "/etc/nginx/cache/nginx2/main/a5/d8/e0/72677057b97aef4eee8e619c49e0d8a5.0000000013" failed (1: Operation not permitted) while reading upstream, client: 35.226.23.240, server: *.---.com, request: "GET /search/----.profile HTTP/1.1", upstream: "https://urldefense.proofpoint.com/v2/url?u=http-3A__-2D-2D-2D.-2D-2D-2D.-2D-2D-2D.-2D-2D-2D-3A80_search_-2D-2D-2D-2D.profile&d=DwICAg&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=UJ4oKb6qp-Ebegtr5fHmO-kPUVhfiwYjNdejFmU1NXI&s=U1ihyg-PzcrmSmpX_F1LuqGWbuzXE5j2ntOwmfP51Ac&e=", host: "---.com"
   
    > The actual setup is that this is on Azure, the storage location /etc/nginx/cache/nginx2 is a mounted file share from Azure storage.
   
    How precisely is the file share mounted?
   
    The error indication is that chmod() failed.
   
    Does chmod() work on that filesystem?
   
    Can you change the mount options such that chmod() will return success,
    even if it does not actually change the permissions of the file?
   
    https://urldefense.proofpoint.com/v2/url?u=https-3A__superuser.com_questions_744384_allow-2Dchmod-2Don-2Dcifs-2Dmount&d=DwICAg&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=UJ4oKb6qp-Ebegtr5fHmO-kPUVhfiwYjNdejFmU1NXI&s=as7NqjlOXjgBHNCelfury7UdUNuN2k7qLdyly5jwGrw&e= suggests
    that "noperm" in the mount options had the desired effect for one user;
    note that that does not seem (to me) to match what the "mount.cifs"
    documentation says, so you'll want to do your own investigation and
    testing before relying on it.
   
    > Has anyone seen/solved this before?
   
    I haven't. So the above is "questions" rather than "suggestions how to fix".
   
    Good luck with it,
   
    f
    --
    Francis Daly        [hidden email]
    _______________________________________________
    nginx mailing list
    [hidden email]
    https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.nginx.org_mailman_listinfo_nginx&d=DwICAg&c=cjytLXgP8ixuoHflwc-poQ&r=wvXEDjvtDPcv7AlldT5UvDx32KXBEM6um_lS023SJrs&m=UJ4oKb6qp-Ebegtr5fHmO-kPUVhfiwYjNdejFmU1NXI&s=2DD6Xlj8iSkt07h_51z6WbBf1N57_e4fDlAzA3p1STs&e=
   

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx