Security on the Internet sucks, and it is only getting worse. The problem is systemic, with security researchers and developers not producing viable ways for the average user to live on the Internet in a secure fashion without excessive paranoia.
The story of Apache’s Infrastructure
The Apache Software Foundation runs about 40 machines, with varying access policies, but some have upwards of 2300 shell accounts, one for every commiter. In the last year, there have been three major incidents in this infrastructure:
- The first attack was in August 2009 was caused by misconfiguration of our backup procedures, and is detailed in this downtime report.
- The second attack was a persistent DDoS attack against issues.apache.org in October 2009.
- The third attack started this week, was a directed attack against the Apache JIRA instance, targeting individual Apache Infrastructure team members. Full details have not yet been posted online about this attack, but you can see the initial email from Joe [gpg signature]. Hopefully later this week, we will get up a blog post with full details.
As a mostly volunteer organization, it is difficult to implement draconian security policies, but the ASF has avoided running most dynamic webapps — the vast majority of our websites are static HTML. Maybe this has saved us from untold other security issues, but even with our believed limited exposure, we still got hacked.
The ASF is by no means perfect, it has half-implemented some of the best practices we know we need to do, but I believe overall the ASF is more secure than most big companies. It has some of the best sysadmins I have known, but it still has issues. Maybe we can just blame that on having too many users, but I believe fundamentally, Internet security is a failure.
I believe there are four major facets around our insecure Internet:
- Identity and Authentication
- Transport Security
- Secure Software and Operating Systems
- Law Enforcement
Identity and Authentication: Failed.
If there was one thing I would change, it would be to stop everyone in the world from using Passwords. Individuals might pick good ones, but on a whole, they pick bad passwords. They also use the same password across a multitude of services.
The problem is most attackers collect these passwords, and then use them to escalate privileges to more services.
Wait a minute you might ask, you just combined Identity with Authentication, but they are different! And yes, you are right, but for the common user, they don’t know the difference. To solve both on a wide scale, I believe their issues are joined at the hip, as authentication depends on identity in most important use cases.
There are many ways you can avoid using passwords, but they are all too difficult for the average user and widespread adoption.
OpenID was one of the first real innovators in this area, and much credit is due to Brad for it. Even though most people on the internet likely have a provider, very few use it on a daily basis. Between the user experience issues and phishing problems, I do not believe OpenID will ever be a real replacement for passwords for all websites. It has solved many problems like how to comment on a blog — which is great, I hate blog spam — but it isn’t the end of Identity and Authentication.
OAuth is taking a different approach, and solving a different problem, which is great for my twitter account. It is still too early to know if OAuth will really improve the wide-scale security of connected web services, but it has been three years since the project started and real-world use cases are still limited. The standard still changing quickly certainly isn’t helping adoption.
Both Amazon Web Services and PayPal let you use multi-factor authentication easily, and I applaud them for this, but most websites and services do not, notability for things like email, which today is the primary identify of most people on the Internet. I believe more services should adopt SMS based multi-factor authentication, and products like Twilio’s SMS API make this easier than ever. I still can count on a single hand the services I have ever logged into using MFA though — I still can’t login to my bank with it, nor my email. Companies like YubiCo are also providing open stacks to improve security, but again most people don’t own a token.
You can find limited cases of SSL Client Certificates being useful and working, but on the whole they are still painful with many sharp edges. I used client certificates extensively at Joost, and I never ever want to repeat that experience, and I am a fairly technical user. The difficulties are not just on the clients and users, but also on running a Certificate Authority correctly with the right policies, revocations and security models.
It isn’t just the users that have problems — providers like DreamHost are unable to authenticate their own users, letting attackers take over accounts mostly via social engineering.
Transport Security: Failed.
As part of the TLS protocol, you need to establish trust between various parties, and so for the most common configurations on the Internet, SSL/TLS depends upon Certificate Authorities.
Trusting Certificate Authorities has turned into an oxymoron. With Certificates being shipped that no one even knows how they got in the trusted list, to the threat of man in the middle attacks from valid certificates, to off the shelf devices for sale to attack it, TLS has failed.
In addition the problems of the SSL renegotiation attacks don’t help the situation, and it will take years before everyone has upgraded their SSL software to prevent this exploit.
I believe while issues in the TLS protocol itself are going to be rare overall, the problem of the CAs will not go away. I don’t know how to solve the trusted CA problem — distributed trust systems are one of the hardest problems to solve for the average end user. As a normal user, at some point you will need to trust a large company to make trust decisions for you, but this process is still too opaque to provide real trust for most people. I personally have doubts that the Extended Validation Certificates are a good thing, in fact I believe it might be providing an illusion of being more secure. We are still trusting the same Certificate Authorities that have almost zero business motivation to provide good security.
Secure Software and Operating Systems: Failed.
Do your Linux servers have an uptime of over 30 days? Then it is very likely they have a local root kernel exploit. It used to be funny to make fun of Windows exploits, and there have been many remote ones which is terrible, but Linux and most open source alternatives have not truly improved security for the average server. The problem isn’t just that the operating system kernels are insecure, it is that privilege escalation is far too easy, and far too common.
You should design software around expecting a local user to be compromised, and not to pick on projects like WordPress, but they have seen a rash of severe security issues over the years, with a relatively small code base — and most webapps, open source or not have similar records. The problem is once an attacker can execute local code, in almost all situations it means with a little work, they can also gain root.
On the user’s side, browsers and their plugins, like flash, have had a similarly abysmal track record. Real innovation has come from Google Chrome, and most other browsers are copying these methods. This is a very good thing. Hopefully it will reduce the size of botnets in the future, but today most users are vulnerable to a multitude of remote attacks.
Law Enforcement: Failed.
In most cities, crime isn’t a major problem anymore. You still lock your doors, take basic precautions with your bike, but the truth is, if someone really wanted to steal something from you, they probably could, but crime is not rampant. You have an expectation that law enforcement will help you.
While law enforcement can sometimes turn a blind eye to a class of crimes, often victimless ones, they have on the whole turned a blind eye to Internet hacking. As long as an attacker doesn’t go after Sarah Pallin’s email, there are rarely any consequences for most incidents.
Inside Apache, we have discussed going to the FBI several times, but the conclusion every time is it would be a waste of our time. The FBI doesn’t care about our problems, because we aren’t a political candidate, nor do we have millions of credit cards. They have their Internet Crime Complaint Center (IC3), but I believe its just a synonym for ‘circular file‘.
Obama’s White House has published their Cyberspace Policy Review (PDF), and it talks about many great points, but it does not actually bring change to the Internet in any measurable fashion.
I don’t want to lock up 12 year old kids for the rest of their lives because they defaced some website, but there must be a better framework and structure for prosecuting attackers world wide. No matter the improvements made to software, users, or best practices, with attackers essentially taking zero risk of ever getting caught today, they have no motivation to stop.
What now?
People are working on making the Internet a better place, but it isn’t enough. Everyone, in every part of the stack must care about their security. Providers, both big and small, software developers, open source and proprietary, users both advanced and novice, they all live in a difficult world, and most of them live in an insecure one.
We won’t all switch to OpenBSD. We won’t all switch to Chrome. We won’t all stop using passwords. And the government can’t save you either.
I wish I had a single answer, I dream that it was a solvable problem. As a technical person, I am more scared of having my own identity stolen, than of any terrorists attacks.
Right now, the mission is on the individual to make smart choices, and do their best, but the only way the world will truly be a better place is if there is a systemic shift, to caring about security of the average human on the Internet, and maybe it will be big companies like Google or Microsoft that end up conquering this problem, but I hope we can learn form existing open source patterns, and find a better distributed way.
