It's kind of obvious you'll be getting better performance in DX9 mode compared to DX11 mode. DX9 mode is older technology that your relatively newer 5850 will run fine. DX11 mode is the latest tech that your card can handle, but it's more taxing since it does more things, reducing the FPS output to your screen.
If it's bothering you that much, wipe the drivers off your 5850 and re-install a fresh set of the latest ones. Also be sure to download the latest application profiles, it could just be that AMD has not yet brought out an optimised set of application profiles for Dragon Age 2 since it's only a demo, you may see different results in the real thing.