I have never had a household router that had a fan in it. Fanless mini pcs do exists they are rarer and usually more expensive and weaker.

The rebooting problem comes from micro interruption in the power grid. Yes you can add a UPS, but then these will become the main reason why the internet is down (I have a whole stack of APC branded UPS with failed batteries)