It turns out it was the drivers. I knew that I needed to have the newest ones and I thought I had already downloaded and installed them, but it looks like I only downloaded a "patch" driver at the time over the old drivers. After redownloading the full newest driver things ran a little more smoothly when gaming
I then tried the driver sweeper and it turns out there were lots of registries and things from my nvidia card that needed to be cleaned up, and this increased my performance a little bit too I think.
I'm now running all settings on ultra just fine I think, only dipping to about 30 fps at extreme loads
Thanks for the help!