I have an old dell optiplex gx520 and im looking to get it a graphics card, but it does not have a pci-e slot so i was just going to upgrade the motherboard. Im not sure what kind size motherboard to get. I would like to keep it in the same case though.
You do not need a pci-e slot for a graphics card. There are a number of pci based low profile cards available. They are not strong, perhaps as much as a 9500GT. http://www.newegg.com/Product/Product.aspx?Item=N82E168...
But your psu is not strong enough to power a stronger graphics card anyway.
BTX was supposed to be a more efficient form factor, but it never caught on. You will be hard pressed to find one new today.
Your PC is near obsolete today, I would not invest much into it.