nginx caching

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

nginx caching

Gabriel Ramuglia
Hi,

I've been using nginx for a while now to proxy flv videos via youtube,
to good effect. However, the site that I'm accessing to proxy videos,
I'm having problems being able to access their site in bulk, so I've
decided to start caching things locally so as not to hit the remote
server so often. I'm doing this in squid, and, for a variety of
reasons, using the url_rewriter function to help out. My urls change
every hour, but the video they refer to does not, so I have to use the
rewriter to make sure the cache stays relevant.

Now, the problem is, squid is using something like 60% of one core of
cpu doing something like 200 megabit / s of bandwidth. This is on top
of the fact that squid has to pass cache misses off to nginx, which is
using another 20% or so (of one core, again) cpu use. Obviously if I
could cut squid out of the equation this would be ideal.

Is there a way in nginx to use the same functionality as the
url_rewriter in squid? Is there a tutorial or other helpful links on
doing caching in nginx?

Thanks,
Gabe

Reply | Threaded
Open this post in threaded view
|

Re: nginx caching

Igor Sysoev
On Sun, Jul 19, 2009 at 02:54:34AM -0700, Gabriel Ramuglia wrote:

> Hi,
>
> I've been using nginx for a while now to proxy flv videos via youtube,
> to good effect. However, the site that I'm accessing to proxy videos,
> I'm having problems being able to access their site in bulk, so I've
> decided to start caching things locally so as not to hit the remote
> server so often. I'm doing this in squid, and, for a variety of
> reasons, using the url_rewriter function to help out. My urls change
> every hour, but the video they refer to does not, so I have to use the
> rewriter to make sure the cache stays relevant.
>
> Now, the problem is, squid is using something like 60% of one core of
> cpu doing something like 200 megabit / s of bandwidth. This is on top
> of the fact that squid has to pass cache misses off to nginx, which is
> using another 20% or so (of one core, again) cpu use. Obviously if I
> could cut squid out of the equation this would be ideal.
>
> Is there a way in nginx to use the same functionality as the
> url_rewriter in squid? Is there a tutorial or other helpful links on
> doing caching in nginx?

http://wiki.nginx.org/NginxHttpProxyModule#proxy_cache

http {
   proxy_cache_path   /spool/nginx/cache levels=1:2
                      keys_zone=cache:10m
                      inactive=3h;

   server {

       location / {
           proxy_pass   ...;

           proxy_cache            cache;
           proxy_cache_valid      200 24h;
           proxy_cache_use_stale  error timeout invalid_header updating
                                  http_500 http_502 http_503 http_504;

           proxy_ignore_headers   Expires Cache-Control;
       }

As to url_rewriter, it's depend on functionality: you may use simple
"rewrite" or complex "perl_set":
http://wiki.nginx.org/NginxEmbeddedPerlModule


--
Igor Sysoev
http://sysoev.ru/en/

Reply | Threaded
Open this post in threaded view
|

Re: nginx caching

Gabriel Ramuglia
thanks, i didn't know that embedded perl could be used this way,
that's a great help.

2009/7/19 Igor Sysoev <[hidden email]>:

> On Sun, Jul 19, 2009 at 02:54:34AM -0700, Gabriel Ramuglia wrote:
>
>> Hi,
>>
>> I've been using nginx for a while now to proxy flv videos via youtube,
>> to good effect. However, the site that I'm accessing to proxy videos,
>> I'm having problems being able to access their site in bulk, so I've
>> decided to start caching things locally so as not to hit the remote
>> server so often. I'm doing this in squid, and, for a variety of
>> reasons, using the url_rewriter function to help out. My urls change
>> every hour, but the video they refer to does not, so I have to use the
>> rewriter to make sure the cache stays relevant.
>>
>> Now, the problem is, squid is using something like 60% of one core of
>> cpu doing something like 200 megabit / s of bandwidth. This is on top
>> of the fact that squid has to pass cache misses off to nginx, which is
>> using another 20% or so (of one core, again) cpu use. Obviously if I
>> could cut squid out of the equation this would be ideal.
>>
>> Is there a way in nginx to use the same functionality as the
>> url_rewriter in squid? Is there a tutorial or other helpful links on
>> doing caching in nginx?
>
> http://wiki.nginx.org/NginxHttpProxyModule#proxy_cache
>
> http {
>   proxy_cache_path   /spool/nginx/cache levels=1:2
>                      keys_zone=cache:10m
>                      inactive=3h;
>
>   server {
>
>       location / {
>           proxy_pass   ...;
>
>           proxy_cache            cache;
>           proxy_cache_valid      200 24h;
>           proxy_cache_use_stale  error timeout invalid_header updating
>                                  http_500 http_502 http_503 http_504;
>
>           proxy_ignore_headers   Expires Cache-Control;
>       }
>
> As to url_rewriter, it's depend on functionality: you may use simple
> "rewrite" or complex "perl_set":
> http://wiki.nginx.org/NginxEmbeddedPerlModule
>
>
> --
> Igor Sysoev
> http://sysoev.ru/en/
>
>

Reply | Threaded
Open this post in threaded view
|

Re: nginx caching

Igor Sysoev
On Sun, Jul 19, 2009 at 04:21:35AM -0700, Gabriel Ramuglia wrote:

> thanks, i didn't know that embedded perl could be used this way,
> that's a great help.

You may use

perl_set  $backend_url  'sub { return "http://youtube_backend/url" }';

proxy_pass  $backend_url;

Also, it better to add

     upstream youtube_backend {
          server ...;
          server ...;
          server ...;
     }

to disable resolving on run-time.


> 2009/7/19 Igor Sysoev <[hidden email]>:
> > On Sun, Jul 19, 2009 at 02:54:34AM -0700, Gabriel Ramuglia wrote:
> >
> >> Hi,
> >>
> >> I've been using nginx for a while now to proxy flv videos via youtube,
> >> to good effect. However, the site that I'm accessing to proxy videos,
> >> I'm having problems being able to access their site in bulk, so I've
> >> decided to start caching things locally so as not to hit the remote
> >> server so often. I'm doing this in squid, and, for a variety of
> >> reasons, using the url_rewriter function to help out. My urls change
> >> every hour, but the video they refer to does not, so I have to use the
> >> rewriter to make sure the cache stays relevant.
> >>
> >> Now, the problem is, squid is using something like 60% of one core of
> >> cpu doing something like 200 megabit / s of bandwidth. This is on top
> >> of the fact that squid has to pass cache misses off to nginx, which is
> >> using another 20% or so (of one core, again) cpu use. Obviously if I
> >> could cut squid out of the equation this would be ideal.
> >>
> >> Is there a way in nginx to use the same functionality as the
> >> url_rewriter in squid? Is there a tutorial or other helpful links on
> >> doing caching in nginx?
> >
> > http://wiki.nginx.org/NginxHttpProxyModule#proxy_cache
> >
> > http {
> >   proxy_cache_path   /spool/nginx/cache levels=1:2
> >                      keys_zone=cache:10m
> >                      inactive=3h;
> >
> >   server {
> >
> >       location / {
> >           proxy_pass   ...;
> >
> >           proxy_cache            cache;
> >           proxy_cache_valid      200 24h;
> >           proxy_cache_use_stale  error timeout invalid_header updating
> >                                  http_500 http_502 http_503 http_504;
> >
> >           proxy_ignore_headers   Expires Cache-Control;
> >       }
> >
> > As to url_rewriter, it's depend on functionality: you may use simple
> > "rewrite" or complex "perl_set":
> > http://wiki.nginx.org/NginxEmbeddedPerlModule
> >
> >
> > --
> > Igor Sysoev
> > http://sysoev.ru/en/
> >
> >

--
Igor Sysoev
http://sysoev.ru/en/