I ran Furmark for longer than usual and nothing happened so I assumed my card's OC is stable, then I went on to Intel Burn Test and passed the standard test as usual, but when I set it to high (more RAM testing) I failed. So I go into BIOS and it looks like my sticks were clocked way too high. I ended up underclocking them and passed the test.
So I loaded up the part of the game that gave me the most trouble - a huge battle in the rain with lots of magic effects and other fancy stuff. No freezing! A few hours later I finished the game without any issues