That's aliasing. Sort of normal.
If I remember AC3 correctly, it has 3 antialiasing options. Low is FXAA, high is FXAA+4xEdgeAA, and ultra is TXAA.
TXAA is blurry jagged garbage, don't bother with it. I recommend the antialiasing one step below TXAA in AC3. And make sure you're running at your monitor's native resolution.