Do you have multipul monitors hooked up to this card? If so with multipul monitors these cards will not down clock i dont know why and it has not been fixed yet to my knowlege.
They run at higher clocks with 2 or more monitors because AMD found some time ago that some cards would produce flickering because of not having enough power to render the pixels of the both monitors, they realiced it was solved completely by raising the clocks when at idle mode so they made it default in their drivers since 10.5, so it's actually intended to be like that and it NOT an issue.
Is your card a factory OC version from gigabyte or just the normal one? some of the custom ones are known for not going down to idle clocks completely, because there are slight differences to the card's BIOS and fan control due to different PCB design, this gets solved by installing the drivers from Gigabyte's site instead of the ones from AMD.
Try getting the drivers from the gigabyte site instead.