Do the above. Also, is this is a legit, fully updated version of Witcher 3? The game can crash the display driver. The driver typically recovers but will run at its default clock which can then be incorrectly reported by monitoring software, like MSI Afterburner. If restarting the PC resolves the issue temporarily, then this is usually the case. Instead of overclocking the GPU, you can underclock it a bit to try to avoid the issue.
This is an occasional issue that I get from time to time with Witcher 3. It was pretty bad at launch with multiple crashes and similar bugs that have since been mostly ironed out.
Another thing, in the most recent versions of Witcher 3, they included the ability to set a timer on how often the game autosaves. You can change the timer and see if there's any correlation between its timer and when the game runs steady at 30, 10 briefly, and then back up to 30. If this is a cause of one of the issues, you could disable this autosave feature, and save manually and when the game does it at key points.
Control Panel > Administrative Tools > Event Viewer: You can browse the Windows Logs to verify if the nvidia driver is crashing. I can't give you an exact example, but most of their files start with the letters 'nv' so it should be apparent.
Like others above mentioned, I'd suggest cleaning out the nvidia driver and associated software with DDU (Display Driver Uninstaller) and doing a clean install of the nvidia driver software to start with.