... I've used Nero to make a bootable CD ...
This is your problem. The .iso image file contains a "picture" of the whole CD. In Nero, use the File/Open command to open the .iso file. That will load all the info into Nero, so then just burn that image. You'll end up with a CD that will automatically boot into memtest86+.
As for your RAM, remember that RAM (especially OCZ RAM) is often sold according to its manufacturer-sanctioned overclock rating, which normally requires manually setting the DIMM voltage to a higher value. Here's some info on that: http://forumz.tomshardware.com/hardware/modules.php?name=Forums&file=viewtopic&p=1249881#1249881