New Graphics Card PROBLEMS!!! Please Help

Hey so I built my own pc about 2 years ago and I recently for Christmas got a new psu and gpu. I got the new RX480 and a 600watt psu. The parts list below is my exact pc currently. Before I got the new psu and gpu I was running a 430watt and a GTX 750ti. The problem is that with the new RX480 it works like trash. I have ran multiple benchamrks and testing the gpu on games and I get worse results and frames than I would with the gtx750. I used heaven 4.0 benchmark and testing my fps on League Of Legends & H1Z1 (Results shown below). When I would try to test the Heavan benchmark on high or ultra the entire pc would sort of crash and and go into a form of sleep mode but I couldn't turn the pc off with the button I would have to go to the back and switch the psu on and off. This would happen basically everytime I would do ultra or high settings on the Heaven benchmark and even a few times just randomly while using the pc. The pc would also lag in a way when I would be web browsing moving my mouse and scrolling around would be laggy at times. Its not the powersupply I don't think because I used my gtx750 with it and a gtx760 with it and no problems whats so ever. I also wonder if it is the GPU because when I put the gpu in my brothers pc with an i7-4770K and the gpu seemed to be working well while doing the Heaven benchmark and playing H1Z1/League and would be closer to his gtx1060 in terms of results with about 1k less score on Heaven compared to the gtx1060, and a little less frames on both games but still close. I can't put the 1060 in my pc to make sure if its my parts because of size of my motherboard and case. If anyone can please help me out, right now my thesis is that it is my motherboard but again idk please help. Thank you. The ultra Heaven benchmark test shows as if the gpu is good but it did crash right after that result and doing test on low and medium would get lower results than the gtx760 and even the gtx750.(All results and information talked about above will be listed below)

My PC Parts List: https://pcpartpicker.com/list/7zyFYr

Benchmarks Results [Heaven 4.0]
GeForce GTX 751 TI
Quality: Ultra
FPS: 46.6
Score: 1175
min FPS: 18.3
max FPS: 84.2

GeForce GTX 760
Quality: Ultra
FPS: 57.7
score: 1454
min FPS: 21.0
max FPS: 124.5

AMD Radeon RX 480 (on my pc)
Quality: Ultra
FPS: 89.6
Score: 2258
min FPS: 17.8
max FPS: 159.0

AMD Radeon RX 480 (on brothers pc)
Quality: Ultra
FPS: 94.4
Score: 2378
min FPS: 23.9
max FPS: 186.6