Prevent direct access to files but allow download from site

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Prevent direct access to files but allow download from site

Olaf van der Spek
Good evening,
I would like to block direct access to files in a folder on my site, but
allow downloading from the site.

Specifically, I want to be able to download a file from the site's html
tag:

<a href="/assets/file/test.txt" download=""> Download TXT </a>

But do not allow direct access and download, using the browser or other
tools such as curl or wget.
So block access to the link:

https://domain.com/assets/file/test.xt

is it possible to obtain such a result?
Thank you

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287297#msg-287297

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Ralph Seichter-2
* MAXMAXarena:

> I want to be able to download a file from the site's html tag [...]
> But do not allow direct access and download, using the browser or
> other tools such as curl or wget.

Public access and restricted access are mutually exclusive. It also
makes nearly no difference what utility is used to access a URL pointing
to the text file you gave as an example, because they'll all send a HTTP
GET request, which is what the web server expects. Attempting to limit
access based on a User-Agent header or similar, to identify the client
type, is easily circumvented.

-Ralph
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

lists@lazygranch.com
You could make it harder to pass around the URL if it is dynamic. That is make the url session related.

You can do a search on "uncrawlable" and then exactly the opposite of what they suggest. That is most people want to be crawled, so their advice is backwards.

One thing to watch out for is Google dorking. Even if Google doesn't crawl, the html/javascript/etc leave clues to file locations.

When I am on a server I will copy what is needed to block wget, curl, nutch, screamingfrog, etc. I have a number of Nginx maps I use to block and flag troublemakers.





  Original Message  


From: [hidden email]
Sent: March 11, 2020 3:57 PM
To: [hidden email]
Reply-to: [hidden email]
Subject: Re: Prevent direct access to files but allow download from site


* MAXMAXarena:

> I want to be able to download a file from the site's html tag [...]
> But do not allow direct access and download, using the browser or
> other tools such as curl or wget.

Public access and restricted access are mutually exclusive. It also
makes nearly no difference what utility is used to access a URL pointing
to the text file you gave as an example, because they'll all send a HTTP
GET request, which is what the web server expects. Attempting to limit
access based on a User-Agent header or similar, to identify the client
type, is easily circumvented.

-Ralph
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
Hello @Ralph Seichter,
what do you mean by "mutually exclusive"?
As for the tools I mentioned, it was just an example.
Are you telling me I can't solve this problem?


Hello @garic,
thanks for this answer, it made me understand some things. But I don't think
I understand everything you suggest to me.

Are you suggesting me how to make the link uncrawlable, but how to block
direct access?

For example, if the user downloads the file, then goes to the download
history, sees the url, copies it and re-downloads the file. How can I
prevent this from happening?

Maybe you've already given me the solution, but not being an expert, i need
more details if it's not a problem, thanks.

I found this stackoverflow topic that is interesting:
https://stackoverflow.com/questions/9756837/prevent-html5-video-from-being-downloaded-right-click-saved

Read the @Tzshand answer modified by @Timo Schwarzer with 28 positive votes,
basically it's what I would like to do, but in my case they are pdf files
and I use Nginx not Apache.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287302#msg-287302

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

lists@lazygranch.com
Answers intermixed below.

On Wed, 11 Mar 2020 21:23:15 -0400
"MAXMAXarena" <[hidden email]> wrote:

> Hello @Ralph Seichter,
> what do you mean by "mutually exclusive"?
> As for the tools I mentioned, it was just an example.
> Are you telling me I can't solve this problem?
>
>
> Hello @garic,
> thanks for this answer, it made me understand some things. But I
> don't think I understand everything you suggest to me.
>
> Are you suggesting me how to make the link uncrawlable, but how to
> block direct access?

If you are going to block access to a file, you should protect it from
the shall we say the less sophisticated user right down to the
unrelenting robots.txt ignoring crawler.

Most of the webmasters want to be crawled, so they have posted what
stops the crawlers from reaching your files. Things like ajax used to be
a problem. Over the years the crawlers have become smarter. But you can
search blogs for crawling problems and then do just the opposite of
their suggestion. That is make your file hard to reach.

In other words, this is a research project.

>
> For example, if the user downloads the file, then goes to the download
> history, sees the url, copies it and re-downloads the file. How can I
> prevent this from happening?
>
> Maybe you've already given me the solution, but not being an expert,
> i need more details if it's not a problem, thanks.
>

In the http block, "include" these files which contain maps. You will
have to create these files.
---------------------
    include /etc/nginx/mapbadagentlarge ;
    include /etc/nginx/mapbadrefer  ;
    include /etc/nginx/mapbaduri;
-------------------------------------------------

Here is how I use maps. First in the location at the webroot:
---

        location / {
        index  index.html index.htm;
        if ($bad_uri)      { return 444; }
        if ($badagent)     { return 403; }
        if ($bad_referer)  { return 403; }
-------

403 is forbidden, but really that is found and forbidden. 444 is no
reply. Technically every internet request deserves an answer but if
they are wget-ing or whatever I hearby grant permission to 444 them (no
answer) if you want.

A sample mapbaduri file follows. Basically place any word you find
inappropriate to be in the URL in this file. You need to use caution
that the words you put in here are not containined in a URL that is
legitimate. Incidentally these samples are real life.
---------------------------------
map $request_uri $bad_uri {
    default                                0;
    ~*simpleboot                           1;
    ~*bitcoin                              1;
    ~*wallet                               1;
}
---------------------------------------------

Next up and more relevant to your question is my mapbadagentlarge file.
This is where you trap curl and wget. There are many lists of bad
agents online. The "" seems to trap those lines with no agent.
-------------------------------------
map $http_user_agent $badagent {
    default                            0;
    ""                                 1;
    ~*SpiderLing                       1;
    ~*apitool                          1;
    ~*pagefreezer                      1;
    ~*curl                             1;
    ~*360Spider                        1;
}

----------------------

The mapbadrefer is up to you. If you find a website you don't want
linking to your website, you make a file as follows:
----------------------------------------
map $http_referer $bad_referer {
    default                                0;
    "~articlevault.info"                   1;
    "~picooly.pw"                          1;
    "~pictylox.pw"                          1;
    "~imageri.pw"                          1;
    "~mimgolo.pw"                          1;
    "~rightimage.co"                       1;
    "~pagefreezer.com"                     1;
}
-----------------------------------------------

Note that as you block these website you will probably loose google
rank.


> I found this stackoverflow topic that is interesting:
> https://stackoverflow.com/questions/9756837/prevent-html5-video-from-being-downloaded-right-click-saved
>
> Read the @Tzshand answer modified by @Timo Schwarzer with 28 positive
> votes, basically it's what I would like to do, but in my case they
> are pdf files and I use Nginx not Apache.
>

Right click trapping is pretty old school. If you do trap right clicks,
you should provide a means to save desired links using a left click. I
don't know how to do that but I'm sure the code exists.  

Don't forget the dynamic url. That prevents the url from being reused
outside of the session. I've been on the receiving end of those
dynamic URLs but never found the need to write one. So that will be a
research project for you.

I'm in the camp that you can probably never perfectly make a file a
secret and yet serve it, but you can block many users. It is like you
can block the script kiddie, but a nation state will get you.

> Posted at Nginx Forum:
> https://forum.nginx.org/read.php?2,287297,287302#msg-287302
>
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Ralph Seichter-2
In reply to this post by Olaf van der Spek
* MAXMAXarena:

> what do you mean by "mutually exclusive"?

I am assuming you have looked up the definition, so I'm not sure in what
way the term could be be misunderstood?

-Ralph
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
In reply to this post by Olaf van der Spek
I would generally say this is not possible in the way you describe it. There
are two ways, however, this could be implemented:

1. You use one-time links to content files: all content retrieval URLs will
get a parameter expires=X (how long the link should be valid) and a
signature (e.g., an HMAC with a secret only known to the NGINX server).
Retrieval won't go through mere file access, but a handler that verifies the
addtional parameters. If they check out, you redirect to an internal
location serving the file.

2. You use a session context: whenever a page validly serving a link to a
certain content is delivered, you set a cookie. Retrievals to files require
the cookie to be present. No cookie, no access.

Cheers,
--j.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287305#msg-287305

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
In reply to this post by Ralph Seichter-2
Hi, thank you for your help, but as I said, being an expert, I have
difficulty understanding certain things. If you know how to solve my
problem, a small example would help me.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287312#msg-287312

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
In reply to this post by lists@lazygranch.com
Thanks for all this information, I try to study and apply what you told me.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287313#msg-287313

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
In reply to this post by Olaf van der Spek
j94305 Wrote:
-------------------------------------------------------
> 2. You use a session context: whenever a page validly serving a link
> to a certain content is delivered, you set a cookie. Retrievals to
> files require the cookie to be present. No cookie, no access.
>
> Cheers,
> --j.

Hi, the second option seem interesting and relatively "simple" solutions,
but I am having some problems.

I put a pdf file in the domain.com/assets/file/test.pdf directory

I created a cookie when a user logs in.
document.cookie = "user_logged = 1";

On Nginx I created this rule:

    location ~ ^/assets/file/ {
    if ($http_cookie ~* "user_logged") {
    allow all;
    }
    root /path/to/root;
    }

I also tried this:

    location ~ ^/assets/file/ {
    if ($cookie_user_logged = "1") {
    allow all;
    }
    root /path/to/root;
    }



But it seems not to work correctly, the user either manages to download from
the direct link https://domain.com/assets/file/test.pdf from the browser,
and from the a href tag of the site, or fails from either side.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287315#msg-287315

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

J.R.
In reply to this post by Olaf van der Spek
Without you being more specific on HOW you want to block direct
downloads and how extreme you want to prevent it, then it's all just a
wild guess what kind of solution you want.

From the example link you gave for stackoverflow, it sounds like you
just want to prevent hotlinking (i.e. downloading without the client
sending a proper referral URL)...

The nginx equivalent of the apache blocking via referral can be found:

http://nginx.org/en/docs/http/ngx_http_referer_module.html

You just set the 'valid_referers' you want, then create a simple 'if'
statement in a location to return a '403 forbidden'...

if ($invalid_referer) { return 403; }
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
Hi, thanks again for the reply.

HOW I want to block I don't know, I am on this forum for this reason.

I thought I was clear, I don't know how to explain it in different words.
I want to prevent the user from downloading the file without being logged on
my site.
The user MUST BE ABLE to download the file from the article pages when
LOGGED.
If the user is NOT LOGGED, he cannot download the file, therefore even
recovering the url, he must receive an error or any other type of block.

I have already tried using the parameter valid_referes, it seems that I have
to specify the domain and also the address of the article pages, but they
are dynamic, the article is not always the same. Maybe I'm doing something
wrong.

If I enter my domain as valid_referer, the user can still download the file
by retrieving the link of the site, from the download history (if he has
previously downloaded the file)

Example of an article address
domain.com/my-first-article

On this page there is a link to download a file located in this path:
domain.com/assets/file/test.pdf

The user does not need to be able to copy and paste this link into the
brower to access the file directly.

I'm continuing to try, but maybe I'm missing some operating logic.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287319#msg-287319

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
In reply to this post by Olaf van der Spek
The key requirement you mentioned now: the user needs to be logged in.

So, the next question is: how do we know the user is logged in. It can't be
just a simple cookie because that could be faked (I could add "LOGGED_IN=1"
without the site authorizing this), and therefore there is no security at
all. Maybe added obscurity :-)

What you need to do is issue a cookie that can only have been created by
NGINX, e.g., something like a string "{user}/{checksum}", where {user} is
the id of the logged-in user, and {checksum} is an HMAC of a secret only
known to NGINX and that user id. In consequence, without knowing the secret,
nobody can fake this cookie. If NGINX gets the cookie, it can determine the
HMAC of the user id and compare against the checksum provided, in order to
check the validity of the request.

The HMAC computation and validation can quickly be done with a few lines of
Javascript, and would even allow for some sort of single sign-on across all
services capable of using these functions (also in other NGINX instances
knowing this secret). Maybe you want to add some form of expiration to such
cookies.

The classical way, however, is to use https and something like a basic
authentication (auth_basic directive), and require valid authentication for
those URIs referring to protected files. If your authentication scheme is a
bit more complex, you may want to use auth_request or some form of initial
login plus a secured session cookie as described above. Essentially, this
principle (with JWT) is also used by more novel schemes such as OpenID
Connect.

Let's see if this helps you further.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287324#msg-287324

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

RE: Prevent direct access to files but allow download from site

Reinis Rozitis
In reply to this post by Olaf van der Spek
> The user MUST BE ABLE to download the file from the article pages when
> LOGGED.
> If the user is NOT LOGGED, he cannot download the file, therefore even
> recovering the url, he must receive an error or any other type of block.

It's rather difficult to achieve that only with a webserver (as typically a webserver itself has no idea about users being logged in or out and just to rely on a cookie is possible but rather weak check).

While you can use the secure link module (https://nginx.org/en/docs/http/ngx_http_secure_link_module.html ) with expiration a more common way would be to implement the download check in the application itself and use the nginx X-Accel-Redirect feature - https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/

Without knowing what kind of app (php/python/js/perl etc) are you running it's hard to give an exact example but the gist of the idea is to:
- place the files outside webroot
- configure the path as an internal nginx location
- the application then checks if the user has an active session, then sends the X-Accel-Redirect header with the particular file to nginx which sends the file to user.

There should be plenty of samples on internet.


rr

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Ralph Seichter-2
In reply to this post by Olaf van der Spek
* MAXMAXarena:

> The user MUST BE ABLE to download the file from the article pages when
> LOGGED. If the user is NOT LOGGED, he cannot download the file,
> therefore even recovering the url, he must receive an error or any
> other type of block.

You describe restricted access, not public access. That differs from
your OP where you wanted to have it both ways. See NGINX docs, section
"Restricting Access with HTTP Basic Authentication". The latter can be
replaced with LDAP or whatever auth mechanism you prefer should basic
authentication not be to your taste.

-Ralph
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
In reply to this post by Olaf van der Spek
I managed to solve using cookies, but as you said, it is not secure.
Although I have no experience, I managed to bypass the control.
Maybe it's not the safest way like I did, in any case it is not recommended
to proceed in this way.

I have experience with auth_basic, but using the terminal to create user and
password and to grant access.

Too many different information in this topic that I have opened, my fault, I
want to simplify it.

I know I previously said I wanted to avoid using Curl, but
I would like to understand the mechanism.

Imagine that the user logs in and i provide him an url, for example:

curl -u {{user.id}}:{{unique_value}}
https://domain.com/assets/file/test.txt

Or

curl -O https://domain.com/assets/file/test.txt?param={{unique_value}}

How can I find out with Nginx if the username and password are real or that
the user/unique_value is still active?
Should I somehow access the database or am I wrong?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287335#msg-287335

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Prevent direct access to files but allow download from site

Olaf van der Spek
MAXMAXarena Wrote:
-------------------------------------------------------
> How can I find out with Nginx if the username and password are real or
> that the user/unique_value is still active?
> Should I somehow access the database or am I wrong?

MAXMAXarena I've just come across this thread looking to answer almost the
same question. In my situation I am running the website on PHP using a
framework called bitweaver. This handles the user login to the dynamic pages
and downloading images and pdf files via the framework, but the thumbnail
images are linked to directly by nginx and can be viewed even if not logged
in.
I've spent the last couple of days playing with http_auth_request_module and
the auth_request entry. I've got it crudely working and I can manually
switch the access on and off using the auth.php script which has access to
the database, but I've hit a snag I'm still trying to crack. The storage
structure is /storage/515/1515/thumbs/ where the second number is the file I
want to access ( the first number just breaks down the storage into smaller
groups of folders ) ... What I'm stuck with is how to get the file number
into auth.php so I can sort out if the current user ID has access to that
file, allowing 'anonymous' users to see as subset of files. You can probably
get away without that bit and just confirm the user ID and at the moment I'd
be happy with just that as well but I'm missing something when nginx runs
auth.php :(

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287297,287559#msg-287559

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx