• Arbitrations
• a20130712.1

• Case Number: a20130712.1
• Status: closed
• Claimants: Tim E
• Respondents: CAcert

• Case Manager: PhilippDunkel

• Arbitrator: UlrichSchroeter

• Date of arbitration start: 2013-07-12
• Date of ruling: 2013-07-16
• Case closed: 2013-07-24
• Complaint: Account Hijacked?
• Relief: TBD

Before: Arbitrator UlrichSchroeter (A), Respondent: CAcert (R), Claimant: Tim E (C), Case: a20130712.1

History Log

• 2013-07-12 (issue.c.o) case [<ticketing system casenumber>]

• 2013-07-12 (A): added to wiki, nominated PhilippDunkel as CM

• 2013-07-12 (A): contacting (C), CC: (Support), (CM), (DRO)
• 2013-07-12 (A): Intermediate ruling #1, sent to (C), (Support), (Critical team), (CM)
• 2013-07-12 (A): sending followup init mailing to (C), with request for accepting CCA/DRP
• 2013-07-12 (A): questions to (Critical team) about a "password problem" issue discovered after a system upgrade
• 2013-07-12 (Support): response to original dispute filing in public support mailing list
• 2013-07-12 (A): response to (Support) regarding response to (C) in public support mailing list, CC: to (C), (CM)
• 2013-07-12 (A): found irc log #sap channel dated 2013-04-17, conversation between Mendel (Critical team) and Ulrich (SAP team) about password login problems after a system upgrade (relates to "magic quotes" settings on production server as depricated in php 5.3 and removed in php 5.4)

• 2013-07-12 (Support) moved 3 tickets to Dispute queue s20130712.1, s20130712.11, s20130712.17

• 2013-07-12 (A): merging tickets s20130712.11, s20130712.17 into s20130712.1

• 2013-07-12 (Support) [s20130712.14]: provides infos about (C)'s account following intermediate ruling #1, CCA acceptance by (C) is given by creating account #2 at 2013-07-12
• 2013-07-12 (Critical team): response to intermediate ruling #1, secured/copied all logs (2 GB) from Nov 2012 until today
• 2013-07-12 (Critical team): response to "password issue regarding system upgrade"
• 2013-07-12 (C): response to intermediate ruling #1 request, forwarding emails from CAcert
• 2013-07-12 (C): clarification about 2 existing accounts, #1 created in 2012 (?), #2 created 2013-07-12
• 2013-07-12 (A): running test scenario on CAcert testserver (discovery of email triggers)
• 2013-07-12 (A): intermediate ruling #2 with order to (Support) to unlock user account #2 of (C) (account created 2013-07-12), to (Critical team), to investigate the IP address that triggers activities on account #1 around 2012-11-20 01:46
• 2013-07-12 (A): details order following intermediate ruling #2 to (Critical team)

• 2013-07-13 (A): questions to (Critical team) how we can freeze the currently set password on the account in question for later compares, so we can receive an evidence that the "password issue by system upgrade" was a problem or if the "password issue" wasn't the cause for (C)'s problem accessing his account (-> technical procedure?)

• 2013-07-13 (A): questions to (C) to identify difference in original state and current state of (C)'s account #1
• 2013-07-13 (C): response with questions answered
• 2013-07-13 (A): further question regarding domain #2 in account #1 of (C) to (C)
• 2013-07-13 (C): 2 further responses regarding domain #2 in account #1
• 2013-07-13 (A): question to (C) if he has an email address under domain #2 ?
• 2013-07-13 (C): confirms, that there exist an email account under domain#2 and that he has access to it
• 2013-07-13 (Support): [s20130712.93] executed the intermediate ruling #2 and unlocked the account #2 of (C)
• 2013-07-13 (A): further email interview communication with (C) regarding email account regarding domain#2, who created the mailbox under domain #2?
• 2013-07-13 (Critical team): intermediate ruling #2 report
• 2013-07-13 (A): summary and response to (Critical team) report with question wether saved and secured log files needs to be removed asap
• 2013-07-13 (C): response regarding addtl. email alias created under domain #2
• 2013-07-14 (A): summary of current findings to (C) with request for a statement by (C)
• 2013-07-14 (Critical team): [removal of copied logfiles] Not needed right now, we can handle this for a while. I don't expect your investigation will take months ...
• 2013-07-14 (Critical team): response #2 to addtl. questions regarding addtl. investigation options
• 2013-07-15 (C): statement to current findings
• 2013-07-15 (A): email accounts b. + c. cross checking
• 2013-07-15 (C): response to email account crosschecking with email addr b - email access by (C) confirmed
• 2013-07-15 (C): response to email account crosschecking with email addr c - email access by (C) confirmed
• 2013-07-16 (A): Intermediate ruling #3, sent to (Support), (C) with order requests and request for exec reports
• 2013-07-16 (Support): [s20130715.104] unlocked account#1. Both account are open now. (intermediate ruling #3, step #1, step #5 processed)
• 2013-07-16 (C): I was able to get into my account again (intermediate ruling #3, step #4, step #5 processed)
• 2013-07-16 (A): details instructions for intermediate ruling #3, step #3 to (C)

• 2013-07-16 (C): Steps 1 thru 5 all worked without trouble. <Email address A> is now registered with my permanent account. (steps #2, #3 finished)

Original Dispute, Discovery (Private Part)

• Link to Arbitration case a20130712.1 (Private Part)

EOT Private Part

Intermediate Ruling #1

• Friday, 2013-07-12 04:21

I hereby give following intermediate ruling:

@Support,

1. please forward the dispute filing that Claimant has posted on the public support mailing list under the subject "Account seems to have been hijacked." to the Arbitration disputes queue immediately (for completenes of documentation)

2. Please hijack the named account

3. Make a printout of - proposed to separate each page to one PDF

1. printout of the main account page
2. current state of _all_ active and expired certificates
   1. client certs
   2. server certs
   3. PGP/GPG keys
3. list of current assurances received
4. list of current assurances given
5. list of secret answers and questions to document overall current account state. please keep this printouts on a save place under Support control until further orders. no sending to Arbitrator, nor CM at the moment except the printout of the main page (Name, DoB, Emails, Domains added to account) (see point 4)

4. Please provide me with the following infos about current account state:

1. Full name
2. DoB in account
3. primary email
4. secondary emails
5. domains added
6. account state
7. certs overview
   • addtl. following numbers:
8. count of assurances received
9. count of assurances given

5. After discovery and collection of infos under 3+4 please lock the account.

6. Please search support queue about Password reset tickets starting November 10 2012 for the primary email address and also secondary email addresses of the account named under 2.

@Critical team, please copy the system logs of the critical system starting November 2012 (as long there are logs available) until today temporarely to another save place so we have a chance for current running arbitration case to collect more informations out of the logs for evidence gathering purposes.

@Tim, can you please forward all emails you've received from support@cacert.org that you can find in your mailbox(es) ?

Frankfurt/Main, 2013-07-12

Discovery

• [s20130712.14]

  • Account referenced in intermediate ruling #1 has been created 2013-07-12

  • Support did not hijacked the account, but locked it.
  • Support identified a 2nd account. This account matches the infos given by (C)'s dispute filing.
• 2013-07-12 (Critical team): response to intermediate ruling #1, secured/copied all logs (2 GB) from Nov 2012 until today

  I've created copies of all the available logs between Nov 2012
  and today, both on the production webserver and on the internal
  log server.
  
  Looking forward to a quick resolution of this issue, so we can
  get rid of the additional 2 GB of logdata as soon as possible.
  
  Regards,
  -- wytze

• 2013-07-12 (Critical team): response to "password issue regarding system upgrade"

  The migration of the webserver took place on April 3, 2013.
  From that point on the magic_quotes functionality was disabled,
  until Mendel patched it on April 17, 2013. The "official" patch
  was made and reported on April 24, 2013. See the attached log
  messages that were sent out to cacert-systemlog@lists.cacert.org.
  
  Regards,
  -- wytze

• 2013-07-12 (C): response to intermediate ruling #1 request, forwarding emails from CAcert
  • Email #1

    Date: Tue 2012-11-20 02:46
    
    Header info:  Subject: [CAcert.org] Email Notification
                  Date: Mon, 19 Nov 2012 18:46:17 -0700
    
    Recipient: users account #1 email address (created before Dec 2012)
    
    Email Notification
    You receive this automatic mail since you yourself or someone else looked up your secret questions and answers for a forgotten password.

    • Email addresses and IP addresses references CAcert's systems that are common for such system notification messages
  • Email #2

    Date: Tue 2012-11-20 02:47
    
    Header info:  Subject: [CAcert.org] Default Account Changed
                  Date: Mon, 19 Nov 2012 18:47:20 -0700
    
    Recipient: users account #1 email address (created before Dec 2012)
    
    Default Account Changed
    You are receiving this email because you or someone else has changed the default email on your account.

    • Email addresses and IP addresses references CAcert's systems that are common for such system notification messages
• Testing scenario on cacert1.it-sls.de testserver that triggers the emails that have been received by (C)

  Starting with account #1, email #1
  
  to change password -> change pass phrase requires old + new pass phrase
  and triggers a notification:
    "You are receiving this email because you or someone else has changed the password on your account.
     That has probably not been received by (C)"
  
  Website access:  My Details - View/Edit - View secret questions
  triggers  Email notification
    "You receive this automatic mail since you yourself or someone else looked up
     your secret questions and answers for a forgotten password."
  
  adding 2nd email #2
  confirm new email under different (!) email domain
  
  login to first account
  switch primary email
  triggers notification to previous primary email #1
     "You are receiving this email because you or someone else has changed the default email on your account."
  
  Email #1  results in error message:  Incorrect email address and/or Pass Phrase.
  Email #2  results in login

• (C)'s Account clarification:

  • Account #1	Account #2
    created 2012 (?)	created 2013-07-12
    	(C) accepted CCA in account creation
    is the hijacked account	is a new account
    currently locked	currently locked
    known 2nd email is not primary email	known email is primary email
    has 2 different email domains confirmed
    has 2 domains confirmed

Intermediate Ruling #2

• Friday, 2013-07-12 22:44
• Discovery revealed, that user account #2, that was subject in intermediate ruling #1 is not the account that probably has been hijacked.
• So therefor (Support) response on intermediate ruling #1 not to hijack account #2 is accepted and confirmed. Reasons given:
  1. (C) created the account last night 2013-07-12 as new account #2
  2. Hijacking account #2 doesn't reveal any more infos what did happen with (C)'s account #1
• So hereby I give the following orders to (Support):
  1. please unlock (C)'s account #2
  2. please keep (C)'s account #1 (as identified in intermediate ruling#1 response) locked until further data discovery about account #1 has been finished.
• I hereby order to Critical team:
  1. please provide me with the info that can be discovered from the saved logs from around 20 Nov 2012 01:46:17 UTC) about (C)'s account #1 activities (actions taken: view details, add addtl. email address, confirm new email address, switch primary email address). Here the IP address of the user who triggered these requests is of special interest.

Frankfurt/Main, 2013-07-12

Discovery II

• 2013-07-13 (C): response with questions answered

  1. only one email registered
  2. two domains registered. both domains matches current domains listed under account #1
  3. Account creation by mid of 2012
  4. list of potential IP addresses that are used by (C)

• (A): account creation date + 6 months matches with expire date of certs
• 2013-07-13 (A): question to (C) if he has an email address under domain #2 ?
• 2013-07-13 (C): confirms, that there exist an email account under domain#2 and that he has access to it
• (C) confirmed, that he is the only one administrator for domain#2 who has also an email box configured under this domain #2
• domain #2 email address has been used in mentioned event in Nov 2012, to be set to new primary email
• 2013-07-13 (Critical team): intermediate ruling #2 report

  From our logs we can see that an email was delivered for <email addr of (C)>
  at 02:46:21 CET, to a google mail server (MX for the domain).
  
  > ...
  
  And the delivery of this email we can also see in our logs at 02:47:23 CET.
  
  > Please provide me with the info of the IP address from which
  > these actions have been triggered.
  
  From the Apache webserver log we can deduce that all accesses were triggered
  by someone accessing the web interface from:
  
          ...comcastbusiness.net
  
  Note that the numeric IP was already translated to this DNS name at the time
  of the access (Nov 20, 2012). But according to the current DNS this does
  correspond to #.#.#.#, both forward and reversed. Rather unsurprisingly,
  this is the IP that the user in question mentioned as his business IP address.
  So I'd be inclined to conclude that the accesses were made from the user's own
  office. Depending on the local office situation, they may have been made by
  himself or some other user in that office ...
  
  The whole session lasted from 02:25:52 CET to 02:47:36 CET, and a total of 98
  Apache web requests have been logged in that time frame (many just simple
  image references of course). Which exact application actions have been taken
  is rather difficult to derive, but someone with deep insight into all the
  application codes may be able to determine that from the logfile which I have
  attached.

• 2013-07-13 (A): summary and response to (Critical team) report

  Intermediate summary:
  This currently is enough, to conclude, that
  a. we didn't had a general attack
  b. if it was an attack on the users account its an "internal" company attack (if there ever was a hijacking) more related to his own office one of the users colleagues? or actions taken by his own and later forgotten ?!?
  
  I keep track of the addtl. 2 GB of logfiles, to not require it any longer then required, but currently there are still some open questions I'm trying to investigate in the interview with (C).
  
  Maybe I need some more infos from the logfiles from July 12th this year, but from current state of the investigations, required logs focusses on the November 20, 2012 and July 12th, 2013 logs each date +/- 1 day log before and 1 day log after
  
  Please leave me a note if its helpful, to remove the "other" logs to dramaticly decrease the "secured" logfiles size
  Then I can release the requirement to keep these logfiles secured in a new intermediate ruling ...

  • 2013-07-14 (A): response to (Critical team)

    thanx for your offer ...  but I'll think, I'll try first to get some infos
    from the user before starting with the time consuming investigations

• 2013-07-14 (Critical team): response #2 to addtl. questions regarding addtl. investigation options
  • password change investigation through binary logs, we only have them starting Dec 26, 2012
  • workaround verification of password change through testserver: possible, but requires (C) to trust our testserver
  • password hashing method doesn't change in system upgrade April 2013 (despite the fact we had password problems reports in April 2013, but they are solved)
• 2013-07-15 (C): confirms (A) summary of current findings
  1. (C) has 3(+2) working email addresses
     1. the email address with that this dispute has been started and that is the primary email address of (C)'s CAcert account #2. This account is currently activated.
     2. a second email address with the same domain as the email addr under a. This email address is secondary email address in (C)'s disputed account #1. This email address moved to the primary email address in communications. Account #1 has two domains added and veryfied. Domain #1 is the domain that is also used in email addresses a. + b. Domain #2 relates to the email address under c.
     3. Number 3 email address is one of three (therefor +2) of (C)'s office email addresses where the email address c. is an alias of his primary office email address. This office email domain is added as domain #2 under (C)'s account #1
  2. statement from (C)

     •  [referenced email-addr c.]
       > > That is correct, that alias is on my primary account at the office.
       > 
       > > That isn't the email address the account was
       > > changed to is it?
       > 
       > yes it is!
       
       That is terribly embarrassing.
       
       > > I ran a search on that email account before
       > > starting this process and didn't find any emails from CaCert.
       > 
       > there were 2 emails sent to <email-addr b.>
       > the 2 mails you've still forwarded.
       > Another email to confirm the new email address has been sent
       > to <email-addr c.> with a link to confirm this
       > addtl. email address.
       > 
       > So you or someone else clicked this link and therefor confirmed
       > this new email address.
       > The next step was to switch this new email address
       > to become the new primary address on the CAcert account.
       
       This would have had to be me. 
       
       > One question we still not concluded:
       > 
       > Who did sitting behind the keyboard on November 20, 2012 around  02:46
       > UTC
       > sending the modification requests to the account ?
       > This timestamp relates to the timestamps in the email headers
       > you forwarded to me, that you've received under your
       > <email-addr b.> mailbox.
       > 
       > 
       > > I don't have any
       > > documentation in my password manager to indicate this.
       > 
       > As you told me in an earlier reply, you are the only sysadmin who
       > also administer the email system.
       > 
       > So please, start rethinking what may did happen on November 20, 2012
       > Do you've started the change by yourself, gets bothered by a phonecall
       > or a colleague and then forgotten to write down the new account
       > settings into your password manager later ?
       
       This is the most likely explanation. It is not uncommon for urgent requests
       to come in that require my immediate attention. 
       As there is no evidence that my email has been compromised, other than this
       event, I would have to conclude that someone or something diverted my
       attention while I was switching the account to my work email.
       
       I suppose I could have deleted the confirmation email that was sent after I
       accepted the change. Not my standard practice, but again documenting such
       changes is, and that didn't happen either.
       
       
       > One more thing you can do, is to research for the emails sent
       > to the email account  '<email-addr c>'
       > on November 20, 2012 - around 02:47 UTC
       > You can also search for the senders email:  support@cacert.org
       > that will be used in such notifications, where the reply-email
       > is set to returns@cacert.org
       > 
       > If you'll find such emails, please forward me a copy (as you've did
       > with the 2 emails to recipient address  '<email-addr b.>')
       > as an attachment in reply to this email.
       
       I checked this again and found only certificate expiration emails, along
       with an email probe on 20120710.
       
       > Please reply to this email with your statement about current findings.
       
       At this point all I can do is offer my sincerest apologies for wasting so
       many peoples time on this. 
       
       What are my options at this point? Can the first account be re-activated so
       I can try the password I have on file with the new email address that I
       apparently changed?
       
       Thanks,
       
       Tim

Intermediate Ruling #3

• by current findings (C) confirmed that he have access to the email addresses a.-c. in question.
• Discovery and investigations did not reveal any hijacking attempts to (C)'s account #1
• there was no malfeasance of any type alleged or found
• (C) made plausible, that account #1 is under (C)'s authority by cross checking both email addresses defined under account #1
• To unlock account #1 and leave the account #2 still open is known temporarly issue under arbitration authority (user has more then 1 active accounts running) and is an acceptable condition in an account recovery scenario.
  1. So therefor I order to (Support) to remove the lock on (C)'s account #1 with current primary email addr C. so that (C) can continue using his account #1.
  2. The current state of account #2 shall be kept open until the account recovery and a transfer of email addr A. over to account #1 is finished upto 7 days.
  3. (C) shall start an email dispute for email addr A. via the CAcert website (menu Disputes - Email dispute) over from account #2 (this results in an automatic closure of account #2) within the next 7 days (deadline set: 2013-07-22 23:59)

  4. In the case (C) has problems recovery access to his account #1, (C) shall follow one of the Lost Password Or Account procedures defined.

  5. I hereby request from (Support) and (C) exec reports of intermediate ruling #3 steps finished.

Frankfurt/Main, 2013-07-16

Intermediate ruling #3 exec steps

• 2013-07-16 (A): Intermediate ruling #3, sent to (Support), (C) with order requests and request for exec reports
• 2013-07-16 (Support): [s20130715.104] unlocked account#1. Both account are open now. (intermediate ruling #3, step #1, step #5 processed)
• 2013-07-16 (C): I was able to get into my account again (intermediate ruling #3, step #4, step #5 processed)
• 2013-07-16 (A): details instructions for intermediate ruling #3, step #3 to (C)

• 2013-07-16 (C): Steps 1 thru 5 all worked without trouble. <Email address A> is now registered with my permanent account. (steps #2, #3 finished)

Ruling

• The case that started with an assumption of a potential "hijacking" of a users account revealed in the discovery and investigation process, that no general hijacking attack nor a unique hijacking attempt over the original users account did happen.
• Evidence gathering includes critical system logs analyze, account state infos received from (Support), several interviews with (C) that the disputed case was a series of unfortunate circumstances, that did happen by the time, the user started an account update (adding addtl. email address to his account and was hindered to document it). More than a half year later, forgotten this essential event, as from this time on, the user did loose access to his account (primary email address switched) and the system still reports: account or password wrong. From the users point of view, the account was correct, so the password must have been changed.
• In summary: there was no malfeasance of any type alleged or found
• At (Support) we've received a couple of Password problem issues back in April this year after a system upgrade. In a first analysis of this case, this was an addtl. possible event, preventing access to the users online account. In the discovery process this issue had been concluded not to be an issue, as the system changes did affect only users trying to login in the first half of April this year. With a system flag re-enabled, the password problems disappeared completely. So its verified, that the password login problem in July didn't relate to the password login problems in early April this year.
• In the intermediate ruling #3 execution steps, claimant has gained access to his original account again and has transfered the email address over from the temporarely created 2nd account, so this account is now closed down by the automated email dispute process at CAcert's main website. (all 5 exec steps of intermediate ruling #3 therefor finished)
• The remaining tasks are the cleanup of secured account states and system logs
• So I hereby order to (Support) to delete all saved printouts from intermediate ruling #1 (if any printouts made)
• To (Critical team) I overrule the order from intermediate ruling #1 to keep a secure copy of the system logs starting November 2012 until today. The secured copy of the system logs can be destroyed. The data is no longer needed under current arbitration as all open questions have been answered. This ruling doesn't affect the regular logs storage and retention.

• To the claimant: I refrain from a fine as I didn't find any malfeasance caused except more work to Arbitration, Support, Critical team. We in CAcert follows the Principles of the Community. Before we set a fine, we train our users. So all what I can give are following 4 recommendations:

  1. to be attentive in doing maintenance tasks with your online account, especialy its about switching the primary email address, as this is the real cause identified in this running case to be the problem (once changed, then forgotten)
  2. to search for Assurers to become assured, at least for upto 50 Assurance points, to gain a higher level of security (you can issue certificates that are valid for 2 years instead of 6 months)
  3. create a client cert and enable cert login, to login securely to your account, also in the case you've forgotten your password.
  4. The assumption that the account was hijacked revealed to be false. But also shown above, the warning message was a bit misleading in current case, as probably the password used was correct, but the account name not. Mostly users entering a wrong password and mostly all users thinks while receiving the message "account or password wrong" that they've entered a wrong password (instead of a wrong username). The CAcert system allows many email addresses to be entered into the online account, but only allows one primary email address, that can be used in connecting to the account. This is not a bug, its a feature .... and results in this running case. So therefor I'll try to encourage you, in case of any trouble in any future case with your account, try contacting (Support) directly by emailing them first. In most cases they can assist in problem situations. Also the online contact form has 2 sections, the private section (write to our Support Engineers) or the public section - write to the public Support mailing list. If you want help for using certs in a certain program, the public support mailing list is the best option, as more readers watching this list and can give answers. But in any account issue, the public support mailing list isn't a good idea. No one other except a Support Engineer can have a look over your current account status and to identify any problem. By "publishing" your assumption, that the account can be hijacked in a public available support mailing list, leaves no other option then to transfer this case to Arbitration, to start investigations, what did happen to this case and to deliver a full open and transparent ruling. So also to be attentive with assumptions like "Account seems to have been hijacked" while using public mailing lists ...

Frankfurt/Main, 2013-07-16

Execution

• 2013-07-16 (A): sent ruling and exec order to (C), (Support), (Critical team), (CM)
• 2013-07-17 (Critical team): exec report - removed secure copy of logs
• 2013-07-21 (A): reminder to support about exec report
• 2013-07-23 (Support): exec report - no account states saved, so therefor nothing to delete
• 2013-07-24 (A): all exec steps finished, case closed, notification to (C), (Support), (Critical team), (CM)

Similiar Cases

• None before

• CategoryArbitration

• CategoryArbCaseAccountCleanup

• CategoryArbCaseEmailDispute

• CategoryArbCaseSystemTasks

• CategoryArbCaseOthers