Caching web traffic with Squid27. Jun '10

While developing stuff on my PC, I noticed that some scripts download the same files multiple times. Same applies for my Estobuntu boxes which I want to keep up-to-date and at the moment software packages from APT repositories are downloaded at least twice. I figured out that I need caching proxy server for my LAN. Obvious place to install it was to my DD-WRT router with USB harddisk attached to it.

I tried several packages available in DD-WRT's optware:

  • polipo - Worked great but the developer has no intentions to add support for intercepting/transparent proxy

  • tinyproxy - Had some weird issues, couldn't get it even running

I ended up installing Squid:

ipkg-opt install squid

Next I edited Squid's configuration file in /opt/etc/squid/squid.conf. The sample configuration file was HUGE, so I here are the most important lines:

# Listen to requests from these IP addresses
acl our_networks src
http_access allow our_networks

# Listen on this IP address/port and allow transparent proxying
http_port transparent

# Maximum file size, allow caching bigger files
maximum_object_size 2048 MB

# Run Squid process as this user
cache_effective_user nobody

# Run Squid process as this group
cache_effective_group nobody

# Set hostname, otherwise Squid refuses to start
visible_hostname dd-wrt

# Path where to store cache data (max 32768 megabytes)
cache_dir ufs /opt/var/squid/cache 32768 16 256
coredump_dir /opt/var/squid/cache

Next initialize cache in /opt/var/squid/cache:

squid -z
chown -R 65534:65534 /opt/var/squid/cache

I also created customized startup script in /opt/etc/init.d/squid:

killall squid
echo "Press Ctrl-C..."
sleep 2

if grep -q nobody /etc/passwd; then
    echo "User 'nobody' exists"
    echo "Created user 'nobody'"
    echo "nobody:*:65534:65534:nobody:/var:/bin/false" >>

if grep -q nobody /etc/group; then
    echo "Group 'nobody' exists"
    echo "Created group 'nobody'"
    echo "nobody:x:65534:" >>

# Start Squid

Create custom firewall script for DD-WRT in /opt/etc/rc.firewall:

iptables -t nat -A PREROUTING -p tcp
    --dport 80 -j REDIRECT --to-port 3128

Enable execute bit and set nvram variable to point to this script:

nvram set rc_firewall=/opt/etc/rc.firewall
nvram commit
chmod 755 /opt/etc/rc.firewall

That's it, you can test it by either killing squid process, in this case none of the HTTP requests should go through or try downloading a file and then downloading it again. For example Linux source tarball was downloaded at 300-400kB/s the first time, any of the subsequential downloads were about 3-4MB/s.

Squid caching HTTP