Yes shadows maxed out kills just about everything. I have also heard it was something in the coding for the shadows. I just leave them turned off and max out everything else. I was able to do that with a Core 2 E8190 @ 3.2ghz and an HD 3870 @ 1680 x1050, so it kinda depends on your resolution. Pretty much any modern dual core or better (core 2, Athlon II, and up) with a midrange or better graphics card can play maxed out with shadows off. I currently play @ 2048 x1152 with my current system, maxed out / shadows off, with no issue.
So for minium I would say Athlon II of around 2.8-3.0, 4gb ddr3, and a 5670/5750/5770 1gb would do just fine.