Team Reports

Team Leaders were encouraged to present a report for their team. 16th January. Closed.

Teus

Critical System Administrator team report July 2008 - June 2009

In May 2008 a final plan for migrating CAcert's critical services (web/db, signer) from Vienna (Austria) to Ede (Netherlands) was drawn up. Wytze van der Raay offered to give a helping hand for the transition period in July/August. Lack of documentation and handover prevented a rebuild of the services from scratch on the available equipment in Ede, so the intended data migration did not take place. Instead, a complete copy of the Vienna server disks was made end of September 2008 by Philipp Guehring, and transported by car to Ede. At the start of October 2008, Mendel Mobach and Wytze van der Raay brought up the services in Ede with the help of Philipp, and took responsibility for managing the critical services.

Ian Grigg and Teus Hagen initiated work on establishing a security policy and security manual to govern the service, and this work was augmented by contributions from Mendel and Wytze. Thus the critical server administrator team got kicked into existence. The two main objectives of the team are: Wytze

In January 2009, Stefan Kooman was recruited as the third team member, but untested background check procedures delayed his actual coming on board until May 2009. A fourth person was interviewed in May 2009 to become member of the team as well, but he had to decline due to work commitments. The work on Security Policy and Security Manual, and two visits by (then auditor) Ian Grigg in March and May 2009 led to the creation of a number of technical documents describing various procedures for critical systems management. Formal logging (to a public mailing list) of all configuration and security management activities was also initiated in this period. From this logging the visits to the hosting facility between Oct 2008 and June 2009 can be learned:

Mendel

Hans

Plans for the coming year include:

Wytze van der Raay, 20091229

Education Team

Ted

Besides doing a bit of support for the Assurer Training Events (ATEs) in spring/summer the main job of the Education team in 2009 was CATS ("CAcert Training System") maintenance. A few numbers:

2009

2008

Number of tests made

6088

4721

Number of passed tests

2519 (41%)

2072 (44%)

Number of different "users" (Certificate IDs)

2800

2032

Number of tests made in english

2779 (46%)

2992 (64%)

Number of tests made in german

3289 (54%)

1711 (36%)

Assurers with passed test according to CAcert statistics page

3175

1375

Assurer Candidates (100 points but no test)

9900

10100

Number of PDF "Certificates of Achievement" requested

371

387

Number of printed "Certificates of Achievement" requested

53

60

Distribution of countries of printed certificates: DE: 32, NL: 6, US: 2, IT: 2, CH: 2, PL: 1, IL: 1, CZ: 1, AU: 1, AT: 1

CATS is running quite smoothly, there are a few bugs open in mantis, but none of them is considered severe.

On the downside, work on anything other than maintenance has almost stalled. I wanted to transfer the current set of questions from the development system to the production system but could not find a quiet hour for some time. Translation of the questions to dutch language has stalled, though most work has already been done. No other translations have been started.

Jens

Some improvements on CATS would be nice but currently noone is working on them:

Other things outside of CATS where work is lurking:

Ted ;)

20100101

Events Team

Uli

I took over this job from Mario around CeBIT time this year (March 2009) by losing the game "Volunteers forward."

OK, since than, I have been helping event organizers to organise their events and coordinate requests for events. I have tried to address the responsibility of managing event reports by first introduced the signaling at the PastEvents wiki page (that has now also been adopted by the ArbitrationCases overview). It helps to get an overview of the current and actual state of the event reports. The response to requests for event reports is very, very slow ... recurring reminders do not solve this problem ...

I have managed 37 events up to now, some did not happen, most events with success. The question we've heard each time is:

After becoming aware that we still needs helping hands everywhere within CAcert, I introduced a Recruitment Campaign at ATEs, as initiated by Daniel. A plan for expanding CAcert through Europe this year has not had as much results as expected. An attempt at expansion to Belgium did not happen (ATE Lummen is still on the Queue). Andreas Buerki is working on expansion to France, but this still needs time because of cultural differences. I have initiated some expansion to the North and the East with the CBLOS Flensburg event, close to the border to Denmark. Still needs more activities with Assurers from northern Germany. Probably one hop to Kopenhagen, then Malmoe (Sweden) is possible, but it needs some time. Contacts were made after Linuxtag Berlin. Expansion to Eastern Europe are still on the wish list but had no results yet.

In the meanwhile we are trying to develop event procedures we can use in CAcert deserts to create a nucleus of new CAcert groups that can grow, now that the TTP program and the Super Assurer program have been frozen. These initiatives are time-consuming as they mean doing individual educational presentations (relating to the co-Audited assurances). Probably a reduced ATE presentations program will be helpful, first used at mrmcd beginning Sept. But individual, one-on-one educations are also needed despite the ATE presentations, involving lots of Assurance in Practice material.

Support for non-European events is quite a problem. I can send information to requestors for Events, but if they are unable to handle events by themselves, nothing happens. Sending out lots of information does not help, we get no responses, no results.

Assurer Training Events (ATE) Team

Henrik

In relation to CAcert's audit, Ted (Education Officer) and myself created the concept of Assurer Training Events (ATE's). This became the solution for un-blocking the Audit-over-Assurance Audit issue. The concept of co-auditing by Senior-Assurers helps CAcert move forward in the audit process.

The collection of results of the co-audited assurances reports exhibits the same problem as the events reports. Slow, very, very slow. Currently, I collect these reports to get them compiled for the co-Audit-report (i) and the project of 'experience points increase' for ATE attendees (ii). Once finished, I send these co-Audit reports to the Assurance Officer Sebastian, and the 'Experience Points' list to the Education Officer Ted. The response after each of the ATEs is generally: "Helpful, Great, Continue."

Many arbitrations are initiated after the ATEs surrounding the issue of Names, so the arbitration work-load has peaked. We have included the Recruitment campaign into the ATE concept, because we met active Assurers at these events, and saw positive results. We have now got some new helping hands in Arbitration, Sysadmins, Developers.

regards, uli ;-)

Lambert 20090908

Arbitration Team Report 2009

Munich Minitop 2009-05-17

Arbitrators. We need more. Ted is busy. Sebastian has asked someone but not yet, person busy. Should an Arbitrator be a Senior Assurer? Probably. Arbitration is good, it is working, but it is too slow. Need faster tools for simple actions.

Arbitration needs better support, need more support engineers, if you want better support, Arbitration should expedite the support engineer background checks

Arbitrated Background Checks where started but never finished until Guillaume's resignation.

Recruitment

At the ATEs the Events- and ATE team meets active assurers, did interviews with many of them. The result was a list of arbitrators that were nominated in August 2009 by the board

Backlog in Arbitration

Starting the ATEs had the intention to train the assurers and help them by their daily work. One of the results was, that now, with better educated assurers many of "old" assurances problems popped up the runs into dispute filings. Therefore the Arbitration team count was too low. The backlog starts.

Relation to Support

Arbitration can only work if support works. With no working support no dispute filings gets thru. No execution requests can be handled. This problem still continued till mid Nov 2009.

Nick

Statistics

2008/2009

2007/2008

Total

47

6

closed

34

6

running

13

0

2008-III

2008-IV

2009-I

2009-II

Total

1

0

8

38

closed

1

0

6

27

running

0

0

2

11

Arbitrations2009-2008-year.jpg

Arbitrations2009-2008-quarters.jpg

our first Arbitrator compiled 2010-01-06 from

Cases by Topics (state 2008-2009)

Account deletion / Assurer Accounts

3

Account deletion / Non-Assurer Accounts

5

Data matching / Name mismatching

9

Data matching / Name order

2

Data matching / Additional names

0

Data matching / Name Modifications Requested

9

Data matching / Date of Birth

4

Other Assurer errors

5

System operations / Arbitrated Background Checks

3

System operations / System Tasks

6

Others

4

External

0

Arbitrations2009-2008-topics.jpg

compiled 2010-01-06 from

regards, uli ;-)

20100106

BirdShack Team

Mario

During CeBIT 2009 discussions between PD, Ian and Mario discovered that the current CAcert software (libressl) might be hard to maintain. For doing more investigation on this, these three plus Alejandro met up in Innsbruck for a week for digging into it.

Analysing the current software revealed that the software had grown by time and is not well designed and possibly would not stand an audit [1].

A complete rewrite of the CAcert software had been proposed and a design was worked out known as BirdShack [2].

The main target of BirdShack was to be auditable and secure. Therefore a three tier architecture was developed.

The main features of the new architecture [3]:

Additionally there needs to be a software assessment team build up to meet Security Policy. The board was asked to start Arbitrated Background Checks on several candidates [4]. This is necessary as well for the current software and for BirdShack.

Mit freundlichen Grüßen / Best regards Mario, 20090106

Assurance Team Report 2009

Sebastian

The Assurance Team draws from Education, Events and ATE teams. Our first appearance as a core team happened at CeBIT March 2009 when Sebastian was appointed Assurance Officer and Ulrich was appointed Events Officer.

Assurance Training Event

Together with Ted of the Education team, we created the Assurer Training Event (ATE) concept, presentations and team. The ATE concept called for 2 parts in each event. First was training and presentations, then followed by assurances which were closely tested from a checklist. Each test in checklist was covered in presentations.

We did this ATE in many events in Germany, and it was also used several times in the Assurance Auditing spring tour.

Report on 'Spring Tour' Audit of Assurance -- Evidence Gathering

After many ATEs the assurance team met for a general team meeting the MiniTop Munich May 17th 2009 1.

A clear signal was set to the community by the Future of Assurance:

Assurance needs to be much stronger. Therefore, the conclusion is that Assurance needs to audit itself. The Assurance team Mission is set:

Response to Findings

To meet this mission we Assurance team did this:

  1. Audited or co-Audited Assurance were made by Experienced or so called Senior Assurers over 100s of assurers. A proposed definition has been documented and will be used for 2010.
  2. The CAcert Assurers Reliable Statement (or CARS) was proposed as a way to get a reliable statement from the co-auditor over the ATE reports, and other issues like Assurer over CAP form, criteria auditing, systems reports or any similar reliable need. We need a symbol to show this, like "Fred, CARS" where CARS stands for CAcert Assurer Reliable Statement. CARS is now in the Handbook, Arbitration and training and is spreading through the Assurer network.

  3. Long discussions about the CAP forms find an end in one Arbitration a20090303.1. It rules

    • the english common law position on contracts (that is, documents with legal effect) is that as long as the document carries all the elements of a good contract, it is a good. That is, form is not important.

    • Therefore, there is no 'official' CAcert CAP form. Every form that includes the elements that are listed clearly in the Assurance Policy, section 4.5. is valid. Invalid documents (e.g. old ones missing the essential CCA agreement clause) can be modified by manually adding this clause.
  4. The CAP form that can be printed from the main website has been updated by dirk mid June 2009 to include the CCA agreement clause.
  5. Notify the remaining Assurers. After installing CATS, and implementing a patch about Apr/May 2009, old assurers that have not passed CATS test, are no longer assurers. A mailing has been sent out to these assurers around May/June 2009.

Mark

Our work in late 2009

Many results from the Munich MiniTOP:

Sub-Policies under AP

After the CCA was approved at TOP september 2007, then ratified by both policy group and association AGM, the problems that relates to exceptions continues. We organised a Hamburg MiniTOP in December 2009 to finish off discussions on exceptions from Munich:

Philipp

Policy Group's Year

Policy kicked off the year by bringing Assurance Policy to DRAFT, p20080712.1 and then to POLICY five months later. This major effort created the formal framework for all assurance, and has stood the test of time well, with no outstanding calls for rewrites! What it did leave aside were the exceptions, and only slow progress was made there. Work on Tverify was beaten by the end-of-life of Thawte's Web of Trust. A policy for Juniors has been through several iterations but still not received consensus. TTP likewise has seen several versions, none of which gathered more than a few supporters.

Security Policy was a great success, taking a first cut framework and effort from Pat Wilson and filling it out. This went to DRAFT in March, and allowed the systems audit to kick-off. However briefly. What is significant about this document is that it is all ours, and the best example of a community process: Pat, Teus Hagen, Philipp Dunkel, Wytze van der Raay and Iang all made significant contributions.

And, in fine style, the CPS, the granddaddy of all CA document, went to DRAFT in July of 2009. This document took over 3 years to write! And in the process, we found it much more convenient to kick out all of Assurance, all of Security, and all the agreements as well.

Other notable events include a combined OA sub-policy for Europe, a new regime for IDNs, and more methods for checking domain control.

For the future: the priority remains for us to finish the Audit set, fill out the Assurance Exceptions, and then look at Organisation Assurance with fresh eyes. Philipp

Philipp Dunkel, documents officer 20090115

Sonance Team

Sonance.net engineers Matthias Gassner and Matthias Šubik supported the shutdown of the CAcert critical servers in Vienna on 30th September 2008. Šubik took charge of the backup disks, storing them in the secure safe of Okto.tv, our community TV station. Gassner rode shotgun over the critical disks with Philipp Guering and Iang from Vienna to Ede, for the handover to the new team.

In February 2009, we got together for a combined assurance and disk destruction event. Philipp Dunkel took the angle grinder to the old CAcert critical platters, and the shreds were distributed at random locations from Vienna to Ede!

Late in 2009, we got the go-ahead to put together a 2nd machine for hosting VMs. We will share our 2 machines between Sonance.net and CAcert's infrastructure team. That 2nd machine is now on the bench, virtualised and is receiving its apps & data. Hopefully, VMs are up and delivered to CAcert for February.

Support Team

In 2009 the support team faced it's own crisis, although not a financial one it led to serious changes.

The Crisis

In the beginning of 2009 the support team consisted of only three members (Guillaume Romagny, Alejandro Mery Pellegrini and Philipp Gühring) who were more and more occupied by other tasks within and outside of CAcert. Although their call for help led to an ABC (Arbitrated Background Check) over Werner Dworak in May, this arbitration didn't progress. In August only Alejandro was left to answer the plenty of requests that are sent to support@cacert.org and it became obvious to him that the circumstances wouldn't change so he downed tools, Guillaume took over and ran support on limited operation. In November Guillaume and Alejandro finally resigned and Ian Grigg was appointed Temporary Support Officer and left with the task to build a new team and get support into full working state again as soon as possible.

Guillaume

The Resurgence

Like in the financial crisis the face of the support crisis made things happen that were not possible before:

The Present

Since November we have been building up and improving support:

Having more SEs and Triage helping them to concentrate on the work that needs to be done will hopefully prevent that support gets burned out again. It also means that we still have time for other things (optimise our processes, improve and update documentation and do other community work).

Right now we are switching from a shared IMAP inbox and a mailing list to a proper issue tracker (OTRS) which Mario Lipinski set up. He also helps us polishing off a few rough edges in it. Using OTRS will hopefully make coordination within the support team easier.

The Future

Although we really got somewhere in these few months we still have things to work on:

Michael Tänzer, Support Engineer

Infrastructure Team

Daniel

The year of 2009 started with CAcert becoming more open. System documentation became public. All of a sudden the cacert-board list became publicly viewable at about the same time we migrated from mailman to Sympa which had a X509 authentication regime. Mail lists became externally archived at gmane and became searchable by search engines. This was not without controversy and highlighted a lack of policy around privacy which hasn't been totally fixed with new policies.

We had a great staffing influx in July which has spread the workload however further work is still needed to improve documentation and manage consistency and change.

Offers of infrastructure came and died out. Eventually Adfinis came through with a production server and test machine, power, bandwidth and IP addresses all for free. Big thanks to Mathieu, Ernie Schwob, Andreas Bürki and all the Adfinis staff who made it possible. This is going to make the move of non-critical infrastructure out of BIT which will make our critical systems easier (or possible) to audit.

Following our x509 authentication on the email lists and irc came blog and other systems are still in the works.

A request for help in July was answered by:

Other staff volunteered further down the line

Philipp Dunkel wrote a board motion tracking system that has helped out the board a lot.

A big thank you for volunteering.

All these new staff created the need for email alias so for the convenience of all here how to contact an admin for a system.

A new architecture for access systems by systems administrators was put in place to increase logging and make it easier to manage access thanks to Wytze and Mendel of the critical systems admin team. Stricter firewall rules were also places around CAcert's infrastructure limiting outbound traffic for the first time.

Evaldo

Systems have had a few changes over the 2009 year:

Daniel Black Infrastructure Administrator

Software Team Ad Hoc Formation and Report

Dirk

  1. in late 2009, Teus created a mega-patch to do a sophisticated CAP-Form layout. It had problems due to language support, international characters, PHP version incompatibility and overall completeness. It was not completed, but installed as newcap.php for availability. This first effort provided a first signal to others: small patches only, mega-patches are too hard.
  2. At CeBIT 2009 we had some talks about the CCA, including which CAP-Forms to use etc.
    1. Dirk detected that it is not necessary to set the CCA-Checkmark to get an account. So he tried to setup a test-environment on his machine, which failed, and also tried to write a patch, where the state of this checkbox was used to decide if an account can be created or not (read: checkbox checked or not).
    2. This patch was submitted with a warning that it was untested, and was installed in a broken fashion on the production machine. This was an early warning that the process was not robust.
    3. Some weeks later Dirk managed to set up a test-environment and then was able to write several patches. This slowness in setting up was a warning that the process of development was not easy.
  3. Several patches from Alexander Prinsier and Dirk found their way to production:
    1. CAP-form adds "I Agree to the CCA" clause, installed June, inspired by Munich MiniTOP meeting
    2. a speed-up of SQL-queries
    3. additional functions for Support
  4. In July, Dirk sent CCA-Patches in three mails in one week into cacert-devel and installed them on test1.
    1. However, test1 had a software-state from March or April, another forewarning.
    2. For many months, a test team of half a dozen people was formed and did lots of testing, but found it very tough to test all the combinations. This repeated the signal that mega-patches are too hard.
    1. In November, Dirk removed the CCA-patches from test1 since testing the entire patches proved too hard.
    2. Dirk then requested that test1 should be put in a state compatible with the software-state of CAcert.org so further tests could be run. However this has not happened, and is a further warning that the software development process is not good.
  5. In January, Jan helped to get test1 running again after problems with disk space. This involved configuration changes to apache and mysql, which was necessary and useful, but again the system deviates further from the production system. Therefore it is not easy to do reliable tests of new patches on test1. Another warning.
  6. Conclusion is that that it is a problem for one person only to participate in the development, assessing of patches, installation of the patches into the prodcution system, as well as manage the test systems to provide robust and accurate environments.
  7. Andreas Baess has taken on the task of re-developing the software development process. First step is to build a repository for patches and then a good test-environment. Then, patches will be installed and tested by the test team before being presented for production use.
  8. Ulrich convened a week of meetings in Germany and that included a full day with all the new software developers in Essen. As Birdshack was moving (too) slowly, we decided it was urgent to do something. All were happy with the basic plan presented by Andreas, and gave it enthusiastic thumbs-up.

Ian

Contribution from ex-Auditor

Critical Systems! Audit opened the financial year (FY08/09) with serious pressure on the Critical Systems. An early 2008 plan had collapsed, governance controls had failed on temporary hosting, and temperatures were running high.

This led to Auditor insisting on an entirely new team, and an end-of-2008 deadline. Teus Hagen invited a new critical systems team with Wytze and Mendel, and the Board confirmed the decision m20080624.1 m20080901.1 to move the servers, come hell or high water. The Vienna team of Sonance and CAcert members prepared the machines and disks for transport, and Matthias Gassner and Philipp Guering drove the disks from Vienna to Netherlands. In the event, the transition was smooth and painless, with the systems coming up by midday on the 1st October. The prior planning paid off.

With the systems moved and a new critical systems team in place, a work-through period was required. They were guided by a new security manual from Pat, which took the place of a checklist for work required. This document settled into its ultimate Security Policy form p20090327 after many reviews, and the team declared itself ready for review. The first (and only) formal audit visit was conducted early April as the Security Policy became effective in DRAFT, and reviewed the uptake of the policy, the physical infrastructure and facility, and roots.

Assurance. Assurance Policy kicked off the year's work by going to DRAFT p20080712.1, and with the CCA in place this provided the policy foundation for review of the Registration Authorities or RAs. However, the new policy did not get rolled out to the Community in any cohesive form, and it wasn't until CeBIT in February 2009 that a group formed to get AP moving via training and testing of the Assurers.

The Assurer Training Events and the concept co-auditing was then rolled out across Germany and wider Europe, with the combined results sufficient to be termed a review or audit over Assurance. This base would have supported a formal audit report over the Assurance part of CAcert (known as Registration Authority audit) but this opportunity was not grasped.

Policy. Other policy work moved forward, primarily the CPS which received continuous improvement over the year, to go to DRAFT p20090706. A big gap in policy work was identified, being the Assurance Exceptions. Because these were optional, they were left out of scope of the audit. Also, concerns over the gap between Organisation Assurance Policy and the practice observed caused that to be moved out of scope as well.

Software. The slowness or absence of patching to make necessary changes demanded by audit, policy and the board led to rising concerns about software. Audit inspired a casual review of the source code by CAcert community members in February 2009, and concluded it was difficult to support. Because of skepticism of this result, a formal camp was organised at Innsbruck in April 2009 to investigate the code. This effort reported it as substantially unmaintainable and started a new project called BirdShack.

Although a very good start, the dual-track approach was barely sufficient to appease Audit, and as the SGM process rose up, Birdshack stalled. Consequently, CAcert's software has regressed while most other teams have advanced in leaps and bounds.

Rasika

Admin. The review-proper of the systems and Assurance only got into high gear in early 2009, and this caused a substantial increase in the demands for changes and work on the Community. The Board was not capable of responding to this work, and as work dragged out with little change in sight, and as funds ran low, with schedule running about a year behind, Audit terminated in June 2009.

This crisis sparked an SGM which resulted in a new team of management, including the now ex-Auditor. From the inside, it quickly became clear that the previous Board was a victim of the myth of who was "doing the audit"; it was not the Auditor, nor the Board. Rather it is the Community who progresses the audit work. Since then, we have engaged in a marketing and PR campaign to ask the Community how their contribution feeds into the audit process.

The FY's audit activities was heavily funded by NLnet Foundation, and Audit spent most of 2 phases of NLnet funding of 18,000 euros. Additionally, Audit was funded in cash and in kind by contributions from many members of the Community.

Ian Grigg, ex-Auditor.

20100112

Other Teams

Add your teams here! Your one chance for fame & glory! Eternal long life in words and pictures!

AGM/TeamReports/2009 (last edited 2011-02-08 12:47:31 by UlrichSchroeter)