With the recent announcement of the UK’s Investigatory Powers (IP) Bill I decided to theorize about the kind of stuff an ISP may implement to be compliant with the legislation. Putting aside the political conversations or my own personal views on the subject I wanted to evaluate how ISPs may implement logging to meet the requirements and the weaknesses these methods may have and how a user may attempt to evade these. However, please note, I am not sure how ISPs are designing implementations to meet the needs of the bill and this is purely speculation.
The IP Bill essentially requires UK ISPs to:
- Retain logs of each website each subscriber visits for an entire year, although it does not have to be granular to the page level.
- Allow police to request access to these logs in a targetted fashion without a warrant.
- Have a legal obligation to assist in the interception of data.
- Enforce existing legislation that an ISP must have the ability to remove encryption applied by the ISP.
- Provides local government some access to the data for investigatory purposes.
- Make it a criminal offense for somebody working for an ISP to reveal that data has been requested.
First off let’s think about the logging, there are three potential methods I can think of that an ISP may implement here. We can start by looking at the simplest form of monitoring which sites a subscriber visits, DNS inspection. With the DNS inspection method an ISP would essentially monitor DNS traffic flowing across its network and attribute each request to a subscriber. As DNS is not an encrypted protocol it is feasible that an ISP could simply perform a dump on all traffic running over port 53 (the standard DNS port) and hence capture DNS lookups not only for the ISP’s DNS servers, but also any third party DNS servers a subscriber may be using. Of course this has some weakness as a user may not utilize traditional DNS to lookup names, and may instead use a service such as Google Public DNS’ DNS over HTTPS, also caching has an important part to play in DNS so potentially the logs timings may not be incredibly accurate if a subscriber has a long DNS cache period on his local machine or indeed a caching DNS server on his home network.
Next up is performing a complete dump of all user’s traffic and extracting metadata. This gives a much clearer view of what the subscriber is up to, although has some downsides too. With this method all traffic from the subscriber is dumped and analyzed, in this form we get to see the source, the destination and the port, as well as the full contents of the data providing encryption is not used. In the case of HTTP the site being requested could easily be extracted from the headers and other useful information could be taken from other protocols even if it is simply which protocol is being used. However with encrypted payloads the ISP cannot see what is going on right?
Now let’s assume the third method is an extension of the second method but enhanced for inspecting encrypted traffic also. In the case of SSL it is possible that during a request the ISP performs a man in the middle attack on the traffic. Essentially you would send your request which is intercepted by a silent proxy, this acts as the end web server decrypting your request and performing metadata recording before making the actual request to the end web server on your behalf, once the response is provided by the end server the proxy then decrypts this, performs metadata recording and then forwards the data on to you resigning it with a certificate authority controlled by the ISP on the way. Obviously this has some flaws, the certificate the client sees is the certificate signed by the ISP, not the original certificate so the client may become suspicious of this however realistically how many average internet subscribers actively check SSL certificates authenticity. The next issue is more security savvy developers may use certificate pinning to ensure that the endpoint they are calling actually responds with the certificate they expect if it does not this would indicate there has been some foul play during the connection.
Which methods do I think the ISPs may be implementing I hear you ask… Well, we cannot be certain which methods ISPs are using, potentially they may not even be using the methods I have considered and may be doing something completely different, although I suspect it’s probably a mixture of DNS inspection and real-time metadata extraction from a subscriber’s unencrypted traffic. I suspect that the overheads and implications of meddling with encrypted traffic is probably too much for ISPs to consider as a normal day to day activity, after all it isn’t transparent to subscribers, it may stop some stuff working where advanced certificate validation techniques are in place and it is expensive as it would need to rely on many HSMs to handle ISP scale resigning of content, and most importantly even with encrypted traffic the obligations to log the sites a subscriber visits but not anymore in depth data can be satisfied with DNS inspection.
Now let’s consider the ISP’s legal obligation to assist in the interception of data… This one is actually pretty easy to achieve if an ISP has implemented the metadata extraction from the subscriber’s traffic. Instead of simply extracting metadata when a request has been made to intercept data an ISP essentially just needs to set a flag on the account to retain all data for a particular subscriber rather than just the metadata, of course in extreme cases this could also incorporate the silent SSL man in the middle technique, although again this may alert a more savvy subscriber to the potential interception.
It’s probably a good idea to talk about how you may evade these techniques, first off it is important to understand that you are not going to stop your ISP from recording some data about your activities. You’ll always leave some form of metadata, however you can probably (again we cannot be sure as maybe there is something being implemented we are not aware of) greatly reduce the amount of data that they are able to collect. The first advice is to use a VPN, a VPN basically tunnels all of your content from your PC / mobile / VPN’d device to a VPN service provider, this is encrypted and isn’t subject to man in the middle attacks like HTTPS traffic. When doing this we can assume all your ISP sees is a connection from you, the subscriber, to an IP address somewhere and the size of the encrypted data stream flowing, they probably cannot extract any meaningful data from this, although they are probably aware that this is a VPN connection. If you choose your VPN provider carefully you can probably find somebody in a strange country like Panama where there are no laws surrounding log retention and they are quite happy to pipe the VPN logs directly to /dev/null (the bin). Another point to consider is that even when using a VPN some traffic may still go out over your normal connection, in some cases to reduce VPN traffic VPN clients still send DNS requests out over the normal connection, obviously in the case of DNS inspection this would give out what you were surfing to over your VPN connection (well the top level of the site anyway), so before you start going online check to make sure your VPN is set to “tunnel all traffic over VPN” including DNS requests. Also a good tip is to use a VPN client which is very vocal about events like disconnections / interruptions, in the case that your VPN client disconnects you may suddenly start sending traffic out directly over your ISP again without realising if your VPN client is the quiet type that hides away in the notification area.
Will we ever know for sure how ISPs are doing this? I suspect not, the ISPs are going to keep this close to their chests until we see the UK’s equivalent of Ed Snowden pop up it’s unlikely we will really know how this stuff works, it is purely just speculation. If you are not up to anything dodgy online it is probably safe to say you will not notice any of the new IP Bill stuff going on around you, however for other’s it is discomforting to think that so much data is being collected about their online activities. Let me know your thoughts on the UK’s Investigatory Powers Bill and how you suspect ISPs may achieve the requirements of the bill, I’d be interested to hear your opinions.