I have 8 gigs of this(kingston hyperx ddr3 1800 cas8) in this . (gigabyte GA-790FXTA-UD5). It is not stable with recommended settings, what setting should I change first? Also is there a way to know which setting to change based on where in memtest it gives me errors?
Assuming, you started running memtest86+ on the RAM at the stock settings, no errors are acceptable. Isolate your bad RAM and RMA it. If you end your testing/re-testing at stock with two good sticks, you could begin OCing it incrementally with increases to the bus speed. When you get to faster settings that won't POST, increase the volts in small increments to get to POST. After POST, load to memtest86+ using a USB stick, CD, or floppy. If no errors, try higher freqs (speed) on the RAM until you get the RAM freq stable (no errors in memtest86+) at the speed you want.
If you're using all four slots, set the voltage to 0.05-0.1V over spec if you haven't tried that yet. A lot of times it will take slightly more juice with all four slots filled.
Also, it looks like there may be a weird quirk with your motherboard when trying to run at higher speeds. It says "To reach DDR3 1866MHz or above, you must install two memory modules and install them in the DDR3_3 and DDR3_4 memory sockets."
Now, I don't know whether they mean you'll be OK at anything up to 1865MHz but limited to two slots at 1866MHz, or whether they mean you'll be limited to two slots if you try to do anything over the default 1333MHz. If it's the second one, you either need a new motherboard or just have to be resigned to only operating the RAM at 1333, or operating at 1800 but only 4GB.