Move the damned router Pepega
No really, that was all it took.
I just moved it like 10cm, and it just started working perfectly.
We also called the ISP, he just said that the wifi problems probably came from interference. I thought that wasn't possible, as we didn't have this problem at first.
However then it turned out (for me at least, everyone else already knew), that we got neighbors recently, right next to our apartment.
My router is in the living room, and neighbor's router is in the living room. And guess what, the two of us only have a single wall separating the living rooms.
So just moving the router by 10cm was all it took to fix it.