I have a GTX580 and see huge fluctuation in FPS with Diablo (via the in-game FPS counter). That said, it never drops below a very playable 60 FPS, so I haven't worried much about it. I see the same thing on my 6970M mobile card.
I think this is the nature of the game. It seems more steady when I am running solo and fluctuates a LOT more when 3 other peeps are in group.