The entire protocol is build under the assumption that you do not need to trust the servers. Let the NSA have then, it doesnt matter. On the other hand 95% of Matrix users are hosted on Matrix.org which was not only hacked several times, but would be an ideal target for any agency to compromise. Its naiive to belive the big Matrix hosts arent compromised. The only effective defense is to build your system around the assumption that the server is compromised, which is what Signal did.