I think not. Hasten to add, it was a long time ago that I used AGP though....
From what I recall, 2 of the AGP standards that I used when I upgraded early on, were the same voltage, and I've gut feeling it was 4x and 8x, meaning you could (motherboard permitting) move from 4x up to 8x.
Long time ago now though, but double check it. Just google your motherboard and check the spec for the AGP slot.
it should not pose a problem, 4x requires a higher voltage than the 8x, and the mobo should recognize and provide it
there is a small issue with the other way though
the first 8x cards/boards ran 8x at the same voltage as 4x, the next "version" ran 8x at a lower voltage
so if you have the first version of boards and the second version of card your card would recieve a higher voltage at 8x than it should, depending on the board this would result in either the card running at 4x to save itself, or the card blowing out
seeing youre running it the other way, there should be no issue