if its the full size its a q45 chipset and if its an optiplex 780 it has the following
Expansion Slots:
1 full height PCIe x16
1 full height PCIe x1
2 full height PCI
DIMM Slots: 4
If its small form factor it has
Expansion Slots:
1 low-profile PCIe x16
1 low-profile PCI
DIMM Slots: 4
If its ultra low profile it has
Expansion Slots:
N/A
DIMM Slots: 2
So if its full size pull off the side panel and look to see if the pcie is open
If its the smaller size do the same but you will have to look for a low profile pcie card.
If its the ultra small your out of luck.
This is the power available in your system which will also have to be considered for a replacement card
Power
Mini-Tower:
305W Standard PSU or optional 255W 88% Efficient PSU; ENERGY STAR® compliant, Active PFC
Desktop:
255W Standard PSU or optional 255W 88% Efficient PSU; ENERGY STAR® compliant, Active PFC
Small Form Factor:
235W Standard PSU or optional 235W 88% Efficient PSU; ENERGY STAR® compliant, Active PFC
Ultra Small Form Factor:
180W up to 90% Efficient Internal PSU, ENERGY STAR® 5.0 compliant, Active PFC
It looks as if it has a build in video already. You will have to deselct the on board if you add a pcie card.