Moving my email archives to Ponymail went well
One feature I forgot that zoe had was how it indexed some attachments. If you have an email with a PDF attachment and that PDF attachment had plain text in it (it wasn't just a scan) then you could search on words in the PDF. That's super handy. Ponymail doesn't do that, in fact you don't get to search on any text in attachments, even if they are plain text (or things like patches). Let's fix that!
Remember how I said whenever I need some code I first look if there is an Apache community that has a project that does something similar? Well Apache Tika is an awesome project that will return the plain text of pretty much whatever you throw at it. PDF? sure. Patches? definitely. Word docs? yup. Images? yes. Wait, images? so Tika will go and use Tesseract and do an OCR of an image.
Okay, so let's add a field to the mbox index, attachmenttext, populate it with Tika, and search on it. For now if some text in an attachment matches your search query you'll see the result, but you won't know exactly where the text appears (perhaps later it could highlight which attachment it appears in).
I wrote a quick Python script that runs through all emails in Ponymail (or some search query subset), and if they have attachments runs all the attachments through Apache Tika, storing the plain texts in the attachmenttext field. We ignore anything that's already got something in that field, so we can just run this periodically rather than on import. Then a one-line patch to Ponymail also searches the attachmenttext field. 40,000 attachments and two hours later, it was all done and working.
It's not ready for a PR yet; probably for Ponymail upstream we'd want the option of doing this at import, although I chose not too so we can be deliberately careful as parsing untrusted attachments is risky
So there we have it; a way to search your emails including inside most attachments, outside the cloud, using Open Source projects and a few little patches.
Whenever I need some open source project my first place to look is if there is an Apache Software Foundation community with a project along the same lines. And the ASF is all about communities communicating over Email, so not only is there an ASF project with a solution, but that project is used to provide the web interface for all the archived ASF mailing lists too. "Ponymail Foal" is the project and lists.apache.org is where you can see it running. (Note that the Ponymail website refers to the old version of Pony Mail before "Foal")
Internally the project is mostly Python, HTML and Javascript, using Python scripts to import emails into elasticsearch, so it's really straightforward to get up and running following the project instructions.
So I can just import my several hundred thousand email messages I have in random text mbox format files and be done? Well, nearly. It almost worked but it needed a few tweaks:
Managing a personal email archive can be a daunting task especially with the volume of email correspondence. However, with Ponymail, it's possible to take control of your email archive, keep it local and secure, and search through it quickly and efficiently using the power of ElasticSearch.
Red Hat Enterprise Linux 5 is coming up to its fifth year since release, and is supported for another five years, until 2017.
The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server if you had installed 5.7, up to and including the 5.8 release, broken down by severity. It's split into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve quite a bit of manual effort to select every one). For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what packages you have installed or removed.
So, for a default install, from release of 5.7 up to and including 5.8, we shipped 42 advisories to address 118 vulnerabilities. 4 advisories were rated critical, 13 were important, and the remaining 25 were moderate and low.
Or, for all packages, from release of 5.7 up to and including 5.8, we shipped 71 advisories to address 177 vulnerabilities. 7 advisories were rated critical, 16 were important, and the remaining 48 were moderate and low.
The 7 critical advisories addressed 20 critical vulnerabilities across 4 different packages:
Updates to correct 19 out of the 20 critical vulnerabilities were available via Red Hat Network either the same day or the next calendar day after the issues were public. The update to krb5 took 2 calendar days because it was public on Christmas day.
Overall, for Red Hat Enterprise Linux 5 since release until 5.8, 98% of critical vulnerabilities have had an update available to address them available from the Red Hat Network either the same day or the next calendar day after the issue was public.
Although not in the definition of critical severity, also of interest during this period were a couple of remote denial of service flaws that were easily exploitable:
In addition, updates to Firefox, NSS, and Thunderbird were made to blacklist a compromised Certificate Authority.
To compare these statistics with previous update releases we need to take into account that the time between each update release is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:
This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other major versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.
See also: 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, and 5.1 risk reports.
We often wondered how the same effect would look if rendered with video. With video you've got the extra element of time, each segment of the mosaic can be running from a different starting point, with a different speed, and even a different direction. In addition the segments themselves can move over time. Would this end up with an effect that was just too much of a mess? Or would it give an effect that helps visualise the consequence of spacetime?
We started by taking several videos at three different locations over the period of a year with a Kodak Zi8 camera. A motorway bridge over the M74, just outside the Buchanan shopping center in Glasgow, and a bench in Strathclyde park. Lining up the images was done roughly by using lines drawn on acetate stuck over the camera screen.
The software to do the mosaic effect was hand-written. We used a simple scripting language, Perl, and the image library GD. On a relatively modern Linux PC running Fedora 16 we can render near real-time 720p HD even when handling 300 segments of mosaic. A simple language controls which parts of the screen come from which video, and the first half of the music video uses this with simple effects having just a few boxes overlayed:
Later in the video things get more complicated, using randomisation to pick the location and movement of each segment:
We used our scripts to create a number of ~13 second segments, then put them all together using kdenlive. The intro and outro were taken from a different video from a hotel room in London Victoria; the intro using a 'miniature' effect, and outro using the randomised segments applied to a single video.
The Perl script and a 5 frame example is available to download: 2011-sonik-vid-example.tar.bz2 (1.4M)
Watch the full video, or click through to YouTube to see it in HD:
We've now created a page to give acknowledgements to the companies and individuals that report issues in our online services, such as finding a cross-site scripting flaw in a Red Hat web site, or a vulnerability in OpenShift.
So let's take a look at the most common times and days we push advisories for Red Hat Enterprise Linux 4, 5, and 6 (including Supplementary) using a heatmap:
The more advisories pushed for a given date and hour, the darker that section of the graph is. So the most popular times for pushing advisories are Tuesdays at 10am and 2pm Eastern US time, Fridays are pretty light for pushes, and there was nothing during the weekends. The spread of the graph shows that we push advisories when they are ready, rather than waiting to a fixed day and time, in order to reduce the risk to users.
All the data used to create this graph is available as part of our public metrics. Thanks to Sami Kerola for the R code from which I based my graph generation.
The table below lists all vulnerabilities which have a CVSS score of 7 or more ('high'), that we fixed in any product during calendar year 2010.
Most common CWE were:
CVE | CWE | 2011 top 25? | CVSS base score | Fixed in |
---|---|---|---|---|
CVE-2007-4567 | CWE-476 | no | 7.8 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2009-0778 | CWE-770 | no | 7.1 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2009-1385 | CWE-191 | no | 7.1 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2009-3080 | CWE-129 | no | 7.2 | Red Hat Enterprise Linux 3, 4, 5, MRG (kernel) |
CVE-2009-3245 | CWE-252 | no | 7.6 | Red Hat Enterprise Linux 3, 4, 5 (openssl) |
CVE-2009-3726 | CWE-476 | no | 7.2 | Red Hat Enterprise Linux 4, 5, MRG (kernel) |
CVE-2009-4005 | CWE-127 | no | 7.1 | Red Hat Enterprise Linux 4 (kernel) |
CVE-2009-4027 | CWE-362 | no | 7.8 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2009-4141 | CWE-416 | no | 7.2 | Red Hat Enterprise Linux 5, MRG (kernel) |
CVE-2009-4212 | CWE-191 | no | 10.0 | Red Hat Enterprise Linux 3, 4, 5 (krb5) |
CVE-2009-4272 | CWE-764 | no | 7.8 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2009-4273 | CWE-78 | yes | 7.9 | Red Hat Enterprise Linux 5 (systemtap) |
CVE-2009-4537 | CWE-120 | yes | 7.1 | Red Hat Enterprise Linux 4, 5, MRG (kernel) |
CVE-2009-4895 | CWE-362 | no | 7.2 | Red Hat Enterprise MRG (kernel) |
CVE-2010-0008 | CWE-606 | no | 7.8 | Red Hat Enterprise Linux 4, 5 (kernel) |
CVE-2010-0291 | CWE-822 | no | 7.2 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2010-0738 | CWE-424 | no | 7.5 | JBoss Enterprise Application Platform 4.2, 4.3 |
CVE-2010-0741 | CWE-20 | no | 7.1 | Red Hat Enterprise Linux 5 (kvm) |
CVE-2010-1084 | CWE-120 | yes | 7.2 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2010-1086 | CWE-20 | no | 7.8 | Red Hat Enterprise Linux 4, 5 (kernel) |
CVE-2010-1087 | CWE-362 | no | 7.2 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2010-1166 | CWE-823 | no | 7.6 | Red Hat Enterprise Linux 5 (xorg-x11-server) |
CVE-2010-1173 | CWE-120 * | yes | 7.1 | Red Hat Enterprise Linux 4, 5 (kernel) |
CVE-2010-1188 | CWE-416 | no | 7.8 | Red Hat Enterprise Linux 3, 4, 5 (kernel) |
CVE-2010-1436 | CWE-120 | yes | 7.2 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2010-1437 | CWE-362 | no | 7.2 | Red Hat Enterprise Linux 4, 5 (kernel) |
CVE-2010-2063 | CWE-823 | no | 7.5 | Red Hat Enterprise Linux 3, 4, 5 (samba) |
CVE-2010-2235 | CWE-77 | no | 7.1 | Red Hat Network Satellite Server 5.3 (cobbler) |
CVE-2010-2240 | CWE-788 | no | 7.2 | Red Hat Enterprise Linux 3, 4, 5, MRG (kernel) |
CVE-2010-2248 | CWE-682 | no | 7.1 | Red Hat Enterprise Linux 4, 5 (kernel) |
CVE-2010-2492 | CWE-805 | no | 7.2 | Red Hat Enterprise Linux 5, 6 (kernel) |
CVE-2010-2521 | CWE-805 | no | 8.3 | Red Hat Enterprise Linux 4, 5, MRG (kernel) |
CVE-2010-2798 | CWE-476 | no | 7.2 | Red Hat Enterprise Linux 5 (kernel) |
CVE-2010-2962 | CWE-823 | no | 7.2 | Red Hat Enterprise Linux 6, MRG (kernel) |
CVE-2010-3069 | CWE-129 | no | 8.3 | Red Hat Enterprise Linux 3, 4, 5, 6 (samba) |
CVE-2010-3081 | CWE-131 | yes | 7.2 | Red Hat Enterprise Linux 3, 4, 5, 6, MRG (kernel) |
CVE-2010-3084 | CWE-120 | yes | 7.2 | Red Hat Enterprise Linux 6 (kernel) |
CVE-2010-3301 | CWE-129 | no | 7.2 | Red Hat Enterprise Linux 6 (kernel) |
CVE-2010-3302 | CWE-120 | yes | 7.1 | Red Hat Enterprise Linux 6 (openswan) |
CVE-2010-3308 | CWE-120 | yes | 7.1 | Red Hat Enterprise Linux 6 (openswan) |
CVE-2010-3432 | CWE-805 * | no | 7.8 | Red Hat Enterprise Linux 4, 5, 6, MRG (kernel) |
CVE-2010-3705 | CWE-788 | no | 8.3 | Red Hat Enterprise Linux 6, MRG (kernel) |
CVE-2010-3708 | CWE-77 | no | 7.5 | JBoss Enterprise Application Platform 4.3, SOA Platform 4.2 |
CVE-2010-3752 | CWE-78 | yes | 7.1 | Red Hat Enterprise Linux 6 (openswan) |
CVE-2010-3753 | CWE-78 | yes | 7.1 | Red Hat Enterprise Linux 6 (openswan) |
CVE-2010-3847 | CWE-426 | no | 7.2 | Red Hat Enterprise Linux 5, 6 (glibc) |
CVE-2010-3856 | CWE-426 | no | 7.2 | Red Hat Enterprise Linux 5, 6 (glibc) |
CVE-2010-3864 | CWE-362 | no | 7.6 | Red Hat Enterprise Linux 6 (openssl) |
CVE-2010-3904 | CWE-822 | no | 7.2 | Red Hat Enterprise Linux 5, 6 (kernel) |
CVE-2010-4170 | CWE-88 | no | 7.2 | Red Hat Enterprise Linux 4, 5, 6 (systemtap) |
CVE-2010-4179 | CWE-862 | yes | 7.5 | Red Hat Enterprise MRG (cumin) |
CVE-2010-4344 | CWE-120 | yes | 7.5 | Red Hat Enterprise Linux 4, 5 (exim) |
* - in both these cases the outcome is not a buffer overflow as the possible overflow is detected and instead converted into an abort (DoS)
See also our 2010 analysis
The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server if you had installed 5.6, up to and including the 5.7 release, broken down by severity. It's split into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve quite a bit of manual effort to select every one). For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what packages you have installed or removed.
So, for a default install, from release of 5.6 up to and including 5.7, we shipped 27 advisories to address 109 vulnerabilities. 3 advisories were rated critical, 12 were important, and the remaining 12 were moderate and low.
Or, for all packages, from release of 5.6 to and including 5.7, we shipped 58 advisories to address 172 vulnerabilities. 4 advisories were rated critical, 20 were important, and the remaining 34 were moderate and low.
The 4 critical advisories addressed 34 critical vulnerabilities across just 2 different packages:
Updates to correct all of the 34 critical vulnerabilities were available via Red Hat Network either the same day or the next calendar day after the issues were public.
Overall, for Red Hat Enterprise Linux 5 since release until 5.7, 97% of critical vulnerabilities have had an update available to address them available from the Red Hat Network either the same day or the next calendar day after the issue was public.
Although not in the definition of critical severity, also of interest during this period were a couple of flaws that were easily exploitable:
In addition, updates to Firefox and NSS were made to blacklist a number of compromised SSL certificates.
To compare these statistics with previous update releases we need to take into account that the time between each update release is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:
This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other major versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.
See also: 5.5 to 5.6, 5.4 to 5.5, 5.3 to 5.4, 5.2 to 5.3, 5.1 to 5.2, and 5.0 to 5.1 risk reports.
The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server if you had installed 5.5, up to and including the 5.6 release, broken down by severity. It's split into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you have installed or removed.
So, for a default install, from release of 5.5 up to and including 5.6, we shipped 57 advisories to address 206 vulnerabilities. 10 advisories were rated critical, 27 were important, and the remaining 20 were moderate and low.
Or, for all packages, from release of 5.5 to and including 5.6, we shipped 80 advisories to address 300 vulnerabilities. 12 advisories were rated critical, 34 were important, and the remaining 34 were moderate and low.
The 12 critical advisories addressed 49 critical vulnerabilities across just 3 different packages:
Updates to correct 48 out of the 49 critical vulnerabilities were available via Red Hat Network either the same day or the next calendar day after the issues were public. The update to fix Exim took 3 calendar days from the date of the report to the Exim developers.
Overall, for Red Hat Enterprise Linux 5 since release until 5.6, 97% of critical vulnerabilities have had an update available to address them available from the Red Hat Network either the same day or the next calendar day after the issue was public.
To compare these statistics with previous update releases we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:
This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other major versions, distributions, or operating systems -- for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.
See also: 5.4 to 5.5, 5.3 to 5.4, 5.2 to 5.3, 5.1 to 5.2, and 5.0 to 5.1 risk reports.
Hold on a second. It might be important. I'd better go and read it. Oh it's just a note confirming some meeting for next week. Deleted. Now, what was I working on?
A few years ago, when I was analysing where my time was going, (and why I was working 60+ hour weeks), I figured out that the context switching caused by being unable to concentrate on a task for more than a few minutes was a major productivity drain.
It's hard to resist a new email. My new cellphone takes great delight in having 'push' email and would really like to beep on each new message I receive. The web is full of gmail notifier applications designed specifically to interrupt you to some important new mail. Even my favourite command-line email client, Alpine, likes to ping you about new mail arriving in your inbox even if you're busy in some other mailbox or composing a mail.
Alpine ought to have some sort of "don't notify me" option, but in the meantime I apply the brute-force patch below to disable it.
This 5-minute patch has saved me several hours of task switching every week, and although this means it can sometimes be an hour or two between me checking my inbox, no one has really noticed.
--- alpine-2.00/pith/newmail.c.orig 2010-07-19 16:47:01.127480500 +0100 +++ alpine-2.00/pith/newmail.c 2010-07-19 16:47:35.657602347 +0100 @@ -680,7 +680,7 @@ } format_new_mail_msg(folder, number, e, intro, from, subject, subjtext, sizeof(subject)); - +#if 0 if(!for_new_mail_win) q_status_message5(SM_ASYNC | SM_DING, 0, 60, "%s%s%s%.80s%.80s", intro, @@ -706,6 +706,7 @@ #endif } #endif +#endif if(pith_opt_icon_text){ if(F_ON(F_ENABLE_XTERM_NEWMAIL, ps_global)