Every year since Red Hat Enterprise Linux 4 was released we've published a risk report where we look at the state of security of the distribution. We investigate the key vulnerabilities, metrics on vulnerability counts, and how users could have been exploited by them. The Six Years of Red Hat Enterprise Linux 4 report (PDF) covering Feb 2005-2011 was published today.
"Red Hat knew about 51.5% of the security vulnerabilities that we fixed in advance. The average time between Red Hat knowing about an issue and it being made public was 23 days (median 10 days).... A default installation of Red Hat Enterprise Linux 4 AS was vulnerable to 20 critical security issues over the first six years. "
The data we publish is interesting to get a feel for the risk of running Enterprise Linux, but isn't really useful for comparisons with other distributions, or operating systems. One important difference is that it is Red Hat policy to count vulnerabilities and allocate CVE names to all issues that we fix, including ones that are found internally. This is not true for many other vendors including folks like Microsoft and Adobe who do not count or disclose issues they fix which were found internally.
It came as no surprise when Microsoft admitted to quiet security patching. We knew many years ago that they did this: not counting extra vulnerabilities that were found internally or by researchers contracted to work for them. For closed source, single vendor software, this isn't too big of a deal - it's not like the user has a choice if they need to update some application to address one critical vulnerability or 20.
When you look back, before they admitted to this practice, Microsoft actively used vulnerability counts in reports as a tool to discredit the security of open source distributions. Famously even Steve Ballmer participated in counting vulnerabilities using candy.
In other news, the Red Hat Enterprise Linux 4 risk report we release each year has been published (PDF). This whitepaper looks at the state of security for the first five years of Red Hat Enterprise Linux 4 from its release on February 15th, 2005. It includes metrics, key vulnerabilities, and the most common ways users were affected by security issues.
"Red Hat knew about 52% of the security vulnerabilities that we fixed in advance of them being publicly disclosed. The average time between Red Hat knowing about an issue and it being made public was 22 days (median 10 days).... A default installation of Red Hat Enterprise Linux 4 AS was vulnerable to 14 critical security issues over the entire five years. "
Secunia collect some very interesting information about the patch state of Windows systems. Their results from 20,000 machines published yesterday were that over 98% of PCs were insecure, having at least one out-of-date application installed.
Actually this isn't surprising and is exactly what I'd expect; it's all down to third party applications.
Let's say you're browsing the web. It's more than likely that at some point you'll want to view some PDF files, watch some Flash content, or play a Java game. Those tasks are all dealt with by third party applications, although to the end user it's all part of the browser experience. Since your system is only as secure as its weakest link, you need to manage security updates for those third party applications just as carefully as you manage security updates for the rest of your system. That's why Adobe Reader, Java, Flash, and all the myriad of other applications you've installed in order to make your system useful have their own update mechanisms. Some applications on Windows will 'phone home' when they are run and check to see if they need to be updated, others deploy services that sit in the background looking for updates from time to time, others even check every time your system starts. Many don't get automated updates at all.
How do you deal with all that risk? I believe it's possible by providing an OS distribution which includes all the bits you'll likely need to make a useful computing environment, thereby taking away that update uncertainty. Red Hat ship several PDF viewers in our distributions for example, but we also ship (in an Extras channel) Adobe Reader. Our Security Response Team are monitoring for security issues in everything we ship, all the third party applications, and providing a single point of contact, a single notification system, and a single way to get the updates.
If Microsoft knew that say 25% of all their users installed Firefox, wouldn't they be better bundling it and providing their centralised automated updates for it, to reduce their customers overall risk? They do already bundle some third party applications, although it's been with mixed success as we found 3 years ago when they didn't provide security fixes for bundled Flash (ZDNet coverage).
This is, in part, why you've not seen me respond recently to the Vista security reports which compare vulnerability counts. In these reports they use a cut-down minimal Red Hat Enterprise Linux installation in order to make it look more like Windows for the comparisons. But this is completely backwards -- the fact that we're including and fixing the flaws using a common process in so much third party software is actually helping reduce the risk and protect real customers. For example we could easily cut our vulnerability count by shipping only one PDF viewer instead of four. But if we know that these other viewers are going to get installed by the customer anyway all we've done is to hide the vulnerability count elsewhere, and you've made the customers overall risk increase.
So it may seem counter-intuitive but we should ship as much third party applications (that we know people use) as we can, because a single managed security update and notification process will decrease a users overall risk. The fewer third party applications that users have to get from elsewhere and install and manage for themselves the better in my opinion.
Remember all those reports which compared the number of security vulnerabilities in Microsoft products against Red Hat? Well researchers have just uncovered proof and an admission that Microsoft silently fix security issues; in one case an advisories states it fixes a single vulnerablity but it actually fixes seven.
Whilst you could perhaps argue that users don't really care if an advisory fixes one critical issue or ten (the fact it contains "at least one" is enough to force them to upgrade), all this time the Microsoft PR engine has been churning out disingenuous articles and doing demonstrations based on vulnerability count comparisons.
Last year I wrote about how both Red Hat and Microsoft shipped the third party Flash browser plugin with their OS and whilst we made it easy for users who were vulnerable to get new versions, Microsoft made it hard. With another critical security issue in Flash last week, George Ou has noticed the same thing.
The Washington Post looked at how quickly Microsoft fix security issues rated as Critical in various years.
For 2005, Microsoft fixed 37 critical issues with an average of 46 days from the flaw being known to the public to them having a patch available.
For 2005, Red Hat (across all products) fixed 21 critical issues with an average of 1 day from the flaw being known to the public to having a patch available. (To get the list and a XML spreadsheet, grab the data set mentioned in my previous blog and run "perl daysofrisk.pl --distrib all --datestart 20050101 --dateend 20051231 --severity C").
(The blog also looks at the time between notification to the company and a patch, whilst daysofrisk.pl currently doesn't report that, the raw data is there and I just need to coax it out to see how we compare to the 133 days for Microsoft)
Last weekend a number of security issues (heap buffer overflows) were found in the Macromedia flash plugin, first reported as affecting Windows only. However we were able to verify yesterday that the issues do affect Linux too. Red Hat shipped the vulnerable flash plugin in an Extras channel (not part of the main distribution, used for such third-party software) for users of Enterprise Linux 3 and 4. Microsoft shipped the vulnerable flash plugin as part of Windows XP SP1 and SP2 (according to their blog.)
- Red Hat Enterprise Linux customers who installed flash just use up2date or the Red Hat Network interface in the usual way and will get their flash update along with a email notification if they need it. Or with automatic updates they'd have it by now.
- Microsoft customers are on their own. Maybe they read the MSRC blog or realise that they have Flash installed and go to the Macromedia site to get their update. Meanwhile being vulnerable to an issue where a malicious web site could run arbitrary code on their system.
One of the top reasons that machines fall foul to security exploits is when they are not kept up to date with security issues. So it follows that to protect users a vendor needs to make security updates as easy and painless as possible. At conferences I highlight that one of the important things a Linux distribution gives you are updates across your entire stack - you don't need to use one system to grab your OS updates, another to get updates to your office application, the built-in update system in your Money tool, a manual update for Flash, and so on.
Mike Nash of Microsoft has repeated his Red Hot demonstration where he compares the number of Windows Server 2003 vulnerabilities to those in Red Hat Enterprise Linux 3. Windows has 30ish and Red Hat has 200ish. I'd normally ignore such terrible manipulations; it's the things that Mike doesn't say that are more important. For example Red Hat Enterprise Linux contains several office suites, money management tools, several PDF viewers, various instant messaging tools all of which don't get counted in the Windows Server 2003 stats. But anyone who has ever used a Linux distribution knows that, so let's ignore the obvious flaws and look at what issues matter the most.
Out of all those Red Hat Enterprise Linux vulnerablities, only 2 were critical based on the Microsoft severity scale. That means only 2 vulnerabilities could have potentially allowed a worm to spread without interaction. Out of the Microsoft vulnerabilities there are 8 critical.
So whilst it might be harder to hold 200 sweets in your hand without dropping a few, I'd rather be holding 200 sweets and 2 ticking timebombs than 30 sweets and 8 ticking timebombs.
Back in March I wrote about a Role Comparison Report from Security Innovation which was published without involving Red Hat. Since that report they contacted and supplied their dataset in which we were able to correct some mistakes. This week Security Innovation released another report from the data, this time looking at the role of a Database Server.
Despite the report's claim to incorporate a qualitative assessment of vendor reactions to serious vulnerabilities, the headline metrics treats all vulnerabilities as equal, regardless of their risk to users.
Their headline figure is 61 days of risk for a Red Hat Enterprise Linux 3 minimal installation with the addition of MySQL server from Red Hat Enterprise Linux Extras.
That sounds like a lot of days of risk - but if you filter their dataset by severity, using the Microsoft scale for determining the severity of each issue you find the following:
** Critical issues: 3 total issues. All fixed on the same day as first public disclosure, therefore having 0 days average risk.
** Critical plus Important: 49 total, with 34 average days of risk
Red Hat prioritise all vulnerabilities and fix first those that matter the most. We publish our raw data and metrics at https://people.redhat.com/mjc/
Days of risk statistics only tell a small part of the story: studies show consumers take some time to apply patches even after a vendor has produced a security update. At Red Hat we continue to work on ways to help people keep their machines up to date. Last year we added Exec-Shield to Red Hat Enterprise Linux 3 which included support for processor EDB (execute disable bit) and NX (no execute) technology. Earlier this year Red Hat Enterprise Linux 4 shipped with Security Enhanced Linux turned on by default. These technology innovations are designed to reduce the risk of security issues.
Today a "Role Comparison Report" from Security Innovation was published which has a headline that we fix security issues less than half as fast as Microsoft.
Red Hat was not given an opportunity to examine the "Role Comparison Report" or it's data in advance of publication and we believe there to be inaccuracies in the published "days of risk" metrics. These metrics are significantly different from our own findings based on data sets made publically available by our Security Response Team.
Despite the report's claim to incorporate a qualitative assessment of vendor reactions to serious vulnerabilities, the headline metrics treats all vulnerabilities as equal, regardless of their risk to users. The Red Hat Security Response Team publish complete data sets allowing calculations to be made taking into account the severity of each flaw. Red Hat prioritise all vulnerabilities and fix first those that matter the most.
For example out of the dataset examined by the report there were only 8 flaws in Red Hat Enterprise Linux 3 that would be classed as "critical" by either the Microsoft or Red Hat severity scales. Of those, three quarters were fixed within a day, and the average was 8 days. A critical vulnerability is one that could be exploited to allow remote compromise of a machine without interaction, for example by a worm.
With the current threat landscape it is no longer sufficient for operating system vendors to just respond to security issues. As part of our overall security strategy Red Hat is continually innovating to create new technologies that proactively help reduce the risk of unpatched or as yet undiscovered vulnerabilities.
Data set and perl script to let you run your own metrics from the Security Response Team
