The motherboard may be defective, probably is for that matter. Whether or not its an MSI or some other brand doesn't matter. If it doesn't do what you want, then it will never please you. Worse, your's doesn't work well even when it isn't overclocked, so that indicates a big problem.
I had an MSI board once. It didn't do what I wanted, so I bought an ASUS that did. Asus is one of the better boards for overclocking, but not always. You have to check out each board for its characteristics. If you have the experience, DFI makes one of the best boards, IMO, for overclocking. It does take more work in that every step has to be done in setting it up, rather than having standard settings to cover everything. It all depends on what you're looking for.