Post-mortem and remediations for Apr 11 security incident

On April 11th we dealt with a major security incident impacting the infrastructure which runs the Matrix.org homeserver – specifically: removing an attacker who had gained superuser access to much of our production network. We provided updates at the time as events unfolded on April 11 and 12 via Twitter and our blog, but in this post weâll try to give a full analysis of what happened and, critically, what we have done to avoid this happening again in future.

Apologies that this has taken several weeks to put together: the time-consuming process of rebuilding after the breach has had to take priority, and we also wanted to get the key remediation work in place before writing up the post-mortem. Firstly, please understand that this incident was not due to issues in the Matrix protocol itself or the wider Matrix network – and indeed everyone who wasnât on the Matrix.org server should have barely noticed.

If you see someone say âMatrix got hackedâ, please politely but firmly explain to them that the servers which run the oldest and biggest instance got compromised via a Jenkins vulnerability and bad ops practices, but the protocol and network itself was not impacted.

This is not to say that the Matrix protocol itself is bug free – indeed we are still in the process of exiting beta (delayed by this incident), but this incident was not related to the protocol. Before we get stuck in, we would like to apologise unreservedly to everyone impacted by this whole incident. Matrix is an altruistic open source project, and our mission is to try to make the world a better place by providing a secure decentralised communication protocol and network for the benefit of everyone; giving users total control back over how they communicate online.

Source: matrix.org