Matrix.org Security Incident

Matrix.org Security Incident(matrix.org)

74 points by Perceptes 7 years ago | 15 comments

zigara 7 years ago |

The attacker seems to have responded:

https://github.com/matrix-org/matrix.org/issues/357 edit: just saw the rest: https://github.com/matrix-org/matrix.org/issues?utf8=%E2%9C%...

"[SECURITY] SSH Agent Forwarding

I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out.

Complete compromise could have been avoided if developers were prohibited from using ForwardAgent yes or not using -A in their SSH commands. The flaws with agent forwarding are well documented."

iancarroll 7 years ago |

Did the blog get hacked (again?) in between this being posted and now? It has what looks like password hashes and `uname -a` from every(?) server in their infrastructure.

This is about as bad as IR can get: you realize you got hacked, you re-build your entire infrastructure and publicly say it's fixed, and then you get popped again...

snailmailman 7 years ago | |

i think so. here is a wayback machine link to the previous blog post https://web.archive.org/web/20190412000400/https://matrix.or...

it stated "Having fully flushed out the attacker [...]" which i guess turned out to be false :-/

also im getting invalid HTTPS certs on the blog now. for some reason im getting a cert that looks like its for github.com ?

edit: now im getting a lets encrypt cert on matrix.org, but a cloudflare SSL error page when i go to www.matrix.org ? the lets encrypt cert looks like it was just issued about an hour ago.

edit2: i guess both with and without www. are lets encrypt, but the with www. cert was issued back in february (and gives a cloudflare SSL error page), while without www. was issued today. (and gives the current hacked message)

deepwell 7 years ago | | |

Here is the message of the hacker, after the matrix.org admins allegedly cleaned everything up and wrote the original blog post: https://web.archive.org/web/20190412055614/https://matrix.or...

ge0rg 7 years ago |

The most favorable reading of the current defacement page is that the attackers still controls the DNS, but no other parts of the infrastructure.

Otherwise, the page probably wouldn't run off github.

irgeek 7 years ago |

Assuming the GitHub issues are from the actual attacker -- and I see no reason to doubt they are -- this is very troubling:

https://github.com/matrix-org/matrix.org/issues/363

Compromise began well over a month ago

Yikes. That's a long time for a compromise to go unnoticed.

ygjb 7 years ago | |

Not really, the average in the industry seems to be floating between 70 and 400 days depending on the source of your stats on the topic (different vendors and reports use different stats for this)

mkj 7 years ago |

> As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history

That seems like a fairly bad usability/security design?

localhostdotdev 7 years ago |

content before it gets fixed:

    Time for actual transparency.

    [list of servers, uname -a for each]

    root@[name]:/var/lib/postgresql# df -h
    [list of partitions]

    $ cat users.txt | grep [name] | head -n1
    @[name]:matrix.org|[hash]
    $ wc -l users.txt
    [~6M users]

    See you soon.

(affects whole site, even https://matrix.org, site is on jekyll BTW)

l2dy 7 years ago | |

Here's the source of that page: https://github.com/matrixnotorg/matrixnotorg.github.io/blob/....

localhostdotdev 7 years ago | | |

account got banned :) (it's on archive.org though)

arunc 7 years ago |

Why was Jenkins running on a production server?

deepwell 7 years ago | |

it was not, read again

Perceptes 7 years ago | | |

But it does seem to be the case that the same SSH key pair that was used to access Jenkins also provided access to the production infrastructure. Unless I'm misunderstanding the nature of the attack.

Time for actual transparency. [list of servers, uname -a for each] root@[name]:/var/lib/postgresql# df -h [list of partitions] $ cat users.txt | grep [name] | head -n1 @[name]:matrix.org|[hash] $ wc -l users.txt [~6M users] See you soon.