AMD has done some weird things with the RX graphics card drivers, including their tendency to throttle based on power limit very, very early. After using DDU to wipe drivers and then reinstalling them, if the problem doesn't go away, use AMD Wattman, built directly into the Crimson interface, to max out the power limit on global settings. It's like free performance at the cost of clicking and dragging on one slider, which is still free. If the problem still persists, it's probably time to up the fan speeds.
I think this is a problem of power limit throttling, combined with thermal limit throttling (not thermal throttling, which slightly different).