A bigger heatsink will not help. The oil absorbs the heat just fine, so to get temperatures down, you need to cool the oil. if you pump the oil through an external radiator and back in, it should lower temps. The plastic/plexi enclosure is a better insulator than heat conductor. Puget Systems has examples of this. http://www.pugetsystems.com/submerged.php
I have a cooling system which is a MagiCool Xtreme Nova 1080 radiator and a 360 mm fan.
It is connected to a Koolance PMP-450 high flow pump.
I know that my fan/radiator is a bit to close from the tank wich blow on it hot air... and maybe that the fan isn't really good. I choose it because it was quiet and the air flow is good but it doesn't cover the complete surface of the radiator...
The cooling performance are good for my general use of the computer,
But I like to know if 76 oC at the maximum frequency is to hot ?
As I said I get this temp only after 2 hours of prime95 no stop...
topping out at 76 is OK, but not great. Think about what is generating heat. Power Supply, Motherboard, CPU, Memory, and video devices. stress the video card at the same time and see what happens, the extra heat may push it further.
You have found that the amount of heat you are generating overall balances with the amount of heat you are dissipating when your temperatures level off at 76C, but you have to push the rest of your system.
Additionally, what do your other sensors read at? or better yet, put a thermometer into the oil, what temperature is it at max load?