Yeah it is definately holding you back some if you're getting 38FPS in Skyrim. I run a constant 60FPS with my HD7870 on max settings but I have an i5-2500K.
You're not going to hurt the card any so go ahead and get your game on buddy. The world of Skyrim isn't going to save itself!
nice one -