Two issues, maybe system incompatibility issues?
1. My Video Card is not recognized by the Mobo, do I need to activate the PCIe 3.0 slot somehow?
2. When I add 12gb of ram using 3 sticks, the system does not boot, turns on for 5 seconds and then shuts down, bad ram or can't use odd number of sticks? All the ram is the exact same.
In order the slot to run at PCI-E 3.0 all you have to do is have a 3rd generation Intel® Core™ i5 or Intel Core i7 processor. That it isnt being recognized by the motherboard leads us to one of two possibilities. Sounds like the board is bad or the video card is bad.
Now with this second issue; while the board is a dual channel board you should should be able to put 3 sticks of memory on the board and have it run in single channel mode. That you are experiencing issues on this speaks to the board being bad.
So I would try the video card on another system to make sure and then give ASRock call to see about an RMA.