• SecurityManual
• IncidentReports

Incident Reports

This wiki page contains security incident reports in chronological order.

1. 20140407 Heartbleed OpenSSL.

• Arbitration a20140408.1

• blog post including English, German version.

• report to board

• bug-1265 bugtracker entry for mail-script

• CAcert-PM-Heartbleed-EN.pdf press release English

Overview of actions

iang reports Heartbleed bug on cacert-sysadm mailing list
https://lists.cacert.org/wws/arc/cacert-sysadm/2014-04/msg00001.html

CAcert critical team checks OpenSSL version on webdb and signer and confirms that these critical machines are not vulnerable
https://lists.cacert.org/wws/arc/cacert-sysadm/2014-04/msg00002.html

golem reports about bug

CAcert personnel gets aware that something was up with OpenSSL

start to seriously discuss the issue

As it was already checked that the root key could not have been affected and the central systems were safe as well, it was agreed that a double tracked strategy should be followed, as two things were likewise important:

1. recovery strategy for CAcert systems
   

   • investigate what infrastructure systems were affected
     

   • update them
     

   • get new keys for them
     

2. informing our members
   

   • to prepare information about root keys being safe, but member probably were likely to be affected, anyway, and what the status on our systems is
     

   • via blog post
     

   • via mail to all members (this was later narrowed down)
     

   • for this the authorization of an Arbitrator was needed
     

   • maybe other information strategies

At this time already was done:

• risks analysed
• most relevant systems fixed (and upgraded certs)
• blog post sent
• tweet sent
• main cacert mailinglist informed
• members of the following teams had been involved:
• infrastructure, arbitration, critical, support, software
• the following teams were informed (enough to be able to react)
• infrastructure, critical, board, audit, pr, support, software
• (correct documented) arbitration ruling to inform members per automated mail

CAcert critical team reports detailed versions of OpenSSL for all CAcert critical machines and confirms that they are not vulnerable
https://lists.cacert.org/wws/arc/cacert-sysadm/2014-04/msg00004.html

At this time was already done

• only two non-critical systems fixed waiting for new cert
• automated mail send to members who either have subscribed "general announcements" or had active keys since the bug was introduced

This was mostly done by a handful of people - during their free time (at night).

Timeline

(all times of when systems were fixed should be added)

2. 20140828 CAcert website showing incomplete names

• https://lists.cacert.org/wws/arc/cacert-systemlog/2014-08/msg00019.html

• https://lists.cacert.org/wws/arc/cacert-systemlog/2014-08/msg00021.html

• https://bugs.cacert.org/view.php?id=1301

Overview of actions

CAcert critical sysadmin team completes upgrade of CAcert chroot application environment to Debian Wheezy on the main webserver, after having installed the patch for https://bugs.cacert.org/view.php?id=1297, deemed to be the last blocking issue for this upgrade. The webserver is started in the new upgraded environment, but a backup of the old environment is kept online just in case.

A well-known CAcert user sends e-mail to support@cacert.org, reporting that the last name and username in the account of this user are showing up as empty. The user is requesting clarification, and is hinting that the problem might have to do with the recent change by the critical sysadmin team.

Support@cacert.org opens case s20140828.90 - Last name gone

Support@cacert.org tries to phone critical sysadmin teamleader on his mobile phone, but gets no response. A voice mail is left. (Unfortunately, no attempt was made to phone the critical sysadmin teamleader on his landline -- which would have resulted in a response in this case --, or to phone another critical sysadmin team member).

Support @cacert.org sends e-mail to critical-admin@cacert.org stating "we do have a problem with the database. All entries with special characters are no visible.", and including a copy of the e-mail of the original problem reporter.

Critical sysadmin teamleader checks mobile phone voice mail, and starts investigating the reported problem. His initial suspicion is that some php5 environment settings concerning character encoding may have changed between the old and new CAcert chroot application environment. After some tests on the cacert2 test server (which has a copy of the production software environment), this turns out not quite to be the case. But further investigation reveals that the php5 function htlmentities() has different behaviour between PHP 5.3 (used in the old environment) and PHP 5.4 (used in the new environment) when its "encoding" parameter is left unspecified. This function is used by the CAcert application function sanitizeHTML(). It turns out that invoking this function with a string containing non-ascii characters results in an empty string as the return value.

Critical team shuts down the main webserver, and re-activates the old CAcert chroot application environment which was a kept as a backup after the upgrade on 2014-08-28.

Main webserver is running again with Debian Squeeze-based CAcert chroot application environment. The "dropping names" issue is resolved.

Critical team reports bug #1301 to software development, including a detailed analysis of the problem, and including a suggested bug fix (i.e. explicitly specifying the "decoding" parameter for htmlentities()).

Problem is picked up by software development. Further progress can be monitored in https://bugs.cacert.org/view.php?id=1301.

Impact analysis

From 2014-08-28 17:02:53 CEST until 2014-08-29 09:41 CEST the production server has been running with a software configuration causing incomplete names to be displayed to website users. Two groups of users were affected by this:

1. users with non-ascii characters in one or more of their name fields
2. users trying to assure a user in group #1

For the first group of users we can state that the only damage done will have been some confusion for the user -- suddenly seeing (part of) his/her name dropped makes a bad impression. However it cannot result in bad values being recorded in the CAcert database. Note that only the display of these names was affected, the name values used internally were not affected at all.

For the second group of users, the issue can be more severe: an assurer may have been presented with an incomplete name of the user which he/she was attempting to assure. In the worst case this may have caused an assurer to refuse to perform such an assurance. We have investigated how many assurances have actually been recorded in the system in the vulnerable time period. This resulted in the identification of some 11 assurances. For these assurances the true names have been checked. Fortunately, none of these true names contained any non-ascii characters, so they were not affected by the bug.

Summarizing we think that we can conclude that the integrity of the CAcert database, and particularly the assurance part, has not been damaged by this incident.