Is you card connected to the backpane or is it just loose? Should not matter how heavy the card is as long as you have it screwed in properly.
What's this moving your hand around the computer thing you are talking about? Are you randomly just hitting stuff inside the case to try to get it to turn on? Sounds like you need someone to take a look at your build and re-seat/re-install the components.