#Howto Filter #HTTPS Traffic with #Squid 3 on #UbuntuServer

This article will tell you how to compile, setup and configure Squid proxy capable of filtering encrypted HTTPS connections using Diladele Web Safety ICAP content filtering server. Being able to look into HTTPS contents greatly increases your ability to control what is allowed and accepted within your network while keeping inappropriate contents away.

Why do We Need to Filter HTTPS

HTTPS protocol was designed to provide secure means of communications between internet browser and remote web servers. In order to achieve this goal HTTPS protocol encrypts data passing through established connections so that it cannot be decrypted in reasonable amount of time thus preventing anyone from sniffing the contents interchanged over this connection. This protocol was primarily invented to enable safe and secure communication between the user and financial sites or government institutions over the insecure medium such as the Internet.

Recently more and more web sites started to use HTTPS encrypted communications to increase online privacy of users. Google who as first enabled HTTPS for all its searches by default probably initiated this trend. Although there are no doubts that HTTPS encryption is a good thing for safety on the wire we must take into account that it also creates several problems for controlled networks typically found at home or offices. The main problem here is the essence of the HTTPS protocol itself – no one except the browser and the web server is able to see and thus filter transferred data. This may not always be desired. Contents that are usually blocked suddenly become immediately accessible by anyone. As an example imagine a school network where minors can see questionable content by just mistyping a search term in Google. Moreover the law often forces administrators in educational institutions to block access to such content (e.g. CIPA for educational environments) and encrypted access to web sites makes it nearly impossible to fulfill such an obligation.

In order to overcome these limitations it is advised to setup HTTPS filtering of web contents with help of SSL bump feature of Squid proxy server and Diladele Web Safety web filter.

How It Works

In order to filter web requests user’s browser needs to be explicitly directed to use the proxy that is deployed in the same network. It is also possible to set the transparent proxy but we are not going to explain how this is done in this tutorial because steps involved are quite different from explicit proxy setup.

When a user tries to navigate to a web site, browser sends the request to proxy server, asking it to get the requested page on his behalf. The proxy establishes a new connection to the remote site and returns the response to browser. If normal HTTP is used then proxy is able to see the original contents of the response and filter it. In case of HTTPS the flow of data is a little different. Browser asks the proxy to establish a virtual tunnel between itself and remote server and then sends encrypted data through the proxy. Domain name to which a virtual tunnel is being established is usually known, so proxy is able to block this virtual tunnel when it finds out that domain name belongs to a prohibited category. Unfortunately this is not a complete solution as there are a lot of sites on the Internet which are general in nature (like Google or YouTube) but allow you to easily navigate to something undesired.

To improve the quality of web filtering and get access to contents in encrypted connections, browsers in the network may be setup to trust proxy to act on their behalf for establishing HTTPS connections, filtering them and passing the allowed data to clients while blocking everything that is not allowed. Although this assumption is too strict to be implemented in public networks, it is easily doable in controlled home, educational or corporate environments where administrators act as sole owners of network devices and may force any trusting rules. After established trust browser is able to ask proxy to connect to a remote site in a safe manner with HTTPS, proxy is able to decrypt the traffic, filter it, encrypt it again and pass it to browser. As browser trusts the proxy it continues working with filtered HTTS without any errors or warnings.

Unfortunately most of the Squid versions included in most common Linux/FreeBSD distributions do not contain compile switches necessary for successful HTTPS filtering. Proxy administrator needs to recompile Squid proxy, reinstall and reconfigure it with additional list of options. Although this process is not very complex, it is complex enough and it would be helpful to have all the necessary steps exactly described. We will provide such exact instructions for the latest version of one of the most popular Linux distributions – Ubuntu Server 13.10.

Build Squid with SSL Bump and ICAP Client

Before compiling it is considered a good practice to bring the operation system to a most recent state. This can be done by running the following commands in the terminal.

sudo apt-get update && sudo apt-get upgrade && sudo reboot

In order to build the Squid from source we need to install some build tools and fetch the sources of Squid and various dependent packages from Ubuntu repository. This does not need to take place on the production server, it is possible to build Squid on one machine and install the resulting binaries on others.

sudo apt-get install devscripts build-essential fakeroot libssl-dev

sudo apt-get source squid3

sudo apt-get build-dep squid3

Running the following command unpacks Squid source package together with all system integration scripts and patches provided by Ubuntu developers.

dpkg-source -x squid3_3.3.8-1ubuntu3.dsc

Sources are unpacked into squid3-.3.8 folder. We need to set this folder as current and modify configure options in debian/rules to include compiler switches (–enable-ssl and –enable-ssl-crtd) necessary for HTTPS filtering.

patch squid3-3.3.8/debian/rules < rules.patch

The rules.patch file should look like this.

--- rules 2013-11-15 11:49:59.052362467 +0100
+++ rules.new 2013-11-15 11:49:35.412362836 +0100
@@ -19,6 +19,8 @@
 DEB_CONFIGURE_EXTRA_FLAGS := --datadir=/usr/share/squid3 \
 --sysconfdir=/etc/squid3 \
 --mandir=/usr/share/man \
+ --enable-ssl \
+ --enable-ssl-crtd \
 --enable-inline \
 --enable-async-io=8 \
 --enable-storeio="ufs,aufs,diskd,rock" \

One file in source code of Squid Proxy needs to be adjusted too (src/ssl/gadgets.cc). This change is needed to prevent Firefox error sec_error_inadequate_key_usage that usually occurs when doing HTTPS filtering with latest Firefox browsers. If you use only Google Chrome, Microsoft Internet Explorer or Apple Safari this step is not required.

Download patches here

patch squid3-3.3.8/src/ssl/gadgets.cc < gadgets.cc.patch

Where gadgets.cc.patch looks like this.

--- gadgets.cc 2013-07-13 09:25:14.000000000 -0400
+++ gadgets.cc.new 2013-11-26 03:25:25.461794704 -0500
@@ -257,7 +257,7 @@
 mimicExtensions(Ssl::X509_Pointer & cert, Ssl::X509_Pointer const & mimicCert)
 {
 static int extensions[]= {
- NID_key_usage,
+ //NID_key_usage,
 NID_ext_key_usage,
 NID_basic_constraints,
 0

Then we build the package using the following command. After a while this command builds all required *.DEB packages.

cd squid3-3.3.8 && dpkg-buildpackage -rfakeroot -b

Install Diladele Web Safety

SSL Bumpng feature alone is not enough to block questionable web content. We also need the filtering server that could be paired with Squid. We will use Diladele Web Safety (DDWS) formerly known as QuintoLabs Content Security for the filtering and blocking part. It is an ICAP daemon capable of integrating existing Squid proxy and providing rich content filtering functionality out of the box. It may be used to block illegal or potentially malicious file downloads, remove annoying advertisements, prevent access to various categories of the web sites and block resources with explicit content.

We will use version 3.0 of qlproxy that is in release candidate state and will probably be finally released this month. It was designed specifically with HTTPS filtering in mind and contains rich web administrator console to perform routine tasks right from the browser.

By default, DDWS comes with four polices preinstalled. Strict policy contains web filter settings put on maximum level and is supposed to protect minors and K12 students from inappropriate contents on the Internet. Relaxed policy blocks only excessive advertisements and was supposed to be used by network administrators, teachers and all those who do not need filtered access to web but would like to evade most ads. Third policy is tailored to white list only browsing and the last group contains less restrictive web filtering settings suitable for normal web browsing without explicitly adult contents shown.

In order to install Diladele Web Safety for Squid Proxy, download package for Ubuntu 13.10 from Diladele B.V. web site athttp://www.quintolabs.com using browser or just run the following command in terminal.

wget http://updates.diladele.com/qlproxy/binaries/3.0.0.3E4A/amd64/release/ubuntu12/qlproxy-3.0.0.3E4A_amd64.deb

Administration console of Diladele Web Safety is built using Python Django framework and is usually managed by Apache web server. To install packages required for correct functioning of web UI run the following commands in the terminal.

sudo apt-get install python-pip

sudo pip install django==1.5

sudo apt-get install apache2 libapache2-mod-wsgi

Install the DEB package and perform integration with Apache by running the following commands.

sudo dpkg --install qlproxy-3.0.0.3E4A_amd64.deb

sudo a2dissite 000-default

sudo a2ensite qlproxy

sudo service apache2 restart

Please note you may need to uncomment the #Require all granted line in /etc/apache2/sites-available/qlproxy.conf if you get Access Denied errors trying to access Diladele Web Safety’s Web UI. This is due to the fact that Apache configuration settings have changed between Ubuntu 12 and Ubuntu 13. Luckily has to be done only once.

Configure Squid for ICAP Filtering and HTTP Bumping

The Squid packages we have compiled previously need to be installed on the system. To perform installation run the following commands.

sudo apt-get install ssl-cert

sudo apt-get install squid-langpack

sudo dpkg --install squid3-common_3.3.8-1ubuntu3_all.deb

sudo dpkg --install squid3_3.3.8-1ubuntu3_amd64.deb

sudo dpkg --install squidclient_3.3.8-1ubuntu3_amd64.deb

In order to recreate original SSL certificates of the remote web sites during HTTPS filtering Squid uses a separate process named ssl_crtd that needs to be configured like this.

sudo ln -s /usr/lib/squid3/ssl_crtd /bin/ssl_crtd

sudo /bin/ssl_crtd -c -s /var/spool/squid3_ssldb

sudo chown -R proxy:proxy /var/spool/squid3_ssldb

Finally, modify Squid configuration file in /etc/squid3/squid.conf to integrate it with Diladele Web Safety as ICAP server. Due to the size of the patch file it’s text not included into the article directly but is part of the download archive.

sudo cp /etc/squid3/squid.conf /etc/squid3/squid.conf.default

sudo patch /etc/squid3/squid.conf < squid.conf.patch

sudo /usr/sbin/squid3 -k parse

From now on Squid is capable of HTTPS filtering and we may continue filtering adjustments from Web UI of Diladele Web Safety.

Navigate to http:/// and login with default name root and password P@ssw0rd. Select Settings / HTTPS Filtering / Filtering Mode. Diladele Web Safety may either filter specific HTTPS sites or all of them with exclusions. Total filtering is more tailored to providing very safe network environments.

Select the desired mode, click Save Settings, add target domains or exclusions as you like and then restart ICAP server by clicking on the green button in the top right corner as indicated on the following screenshots.

 

Run the following command in terminal on the proxy.

sudo service squid3 restart

Navigate to google.com and see that HTTPS filtering is indeed active. The following warning shows that Squid was able to bump the SSL connection, filtered it and encrypted in again using Diladele Web Safety’s generated certificate.

In order to get rid of these warnings, we must install the myca.der certificate file from into the browser and mark it as trusted. Again navigate to http://. Select Settings / HTTPS Filtering / Certificates and select the one that matches your operating system or device. Instructions on how to install the certificate in each operating system or device is slightly different, the following screens show how to install the DER file in the Apple iPad and Microsoft Internet Explorer. For other devices please take a look at Online Documentation of Diladele Web Safety.

Reopen your browser, navigate to Google and make sure the certificate warning is away. If you click on the lock icon in the internet address box then it clearly indicates the google.com was signed by proxy’s certificate and not by original certificate by google.

If you try to search Google with some adult only terms (e.g. NSFW) Diladele Web Safety blocks the access to explicit contents showing its denied page.

Please be sure to change the default certificates that come with installation package of Diladele Web Safety to something more unique for your network. For instructions on how to regenerate your own certificates for this purpose consult Online Documentation of Diladele Web Safety.

Resume

Now we have HTTPS web filtering up and running and our network environment become a little safer for those who need protection at most. Next steps would be direct all clients browsers to use Squid proxy, regenerate the default proxy certificates, setup authentication and authorization to get user specific reports in Diladele Web Safety, integrate it with e.g. Active Directory using Squid’s support for Kerberos authentication and optionally setup transparent HTTPS filtering. It is also advisable to setup the caching DNS server on Squid proxy to further increase speed of connections.

Link Reference

•  Diladele B.V. web site at http://www.quintolabs.com
• Online Documentation of Diladele Web Safety
• Squid Proxy Wiki on SSL Bumping
• Ubuntu Server 13.10.

source