Task manager may give you misleading information.
If you have a single thread app that is entirely cpu bound, windows will spread it out across all available cores.
With 4 active threads, but only two that are heavily utilized, you may average 50% cpu when you are actually cpu bound.
It is unclear how well windows can detect what needs to be dispatched on the main cores vs' the hyperthreads.
Playing civ 5 on a 4790K with hyperthreads, I see about 25% utilization on alternating cores. not much on the others.
Civ 5 is largely a single thread game.
You might try two experiments:
1. disable hyperthreading. Theory is that windows is dispatching BF4 to the slower hyperthreads.
2. Reduce your oc. If this results in worse fps, then you are likely sensitive to individual core speeds.
BF4 should not be very cpu sensitive in single player. In multiplayer it can be and 4 fast cores will help a lot.