I doubt it's your OS. Your RAM isn't even close to the same specs. You can't just grab whatever random RAM you have lying around and expect them to work together. You should still be able to run Memtest86+ because it runs outside of the OS. You burn the ISO image to a CD and boot from the CD. This screams a RAM problem IMO.
You can most certainly change the RAM settings in the BIOS of an GA-EP45-DS3P. Just press Ctrl + F1 when you first get into the BIOS to unlock all the options. You'll need to set the RAM settings to the specs of the lowest RAM. It sounds like maybe DDR2 667 at 1.9v. You didn't list the timings. Those are important to set correctly also.