The only time I ever really get frustrated with it is on the netbook with a wireless connection.
On my main rig the pages are pretty snappy.
The adverts cause more slowdown than the sigs but again, thats another story.
The thing is we, as a team seem to have settled with this style of sig so we won't keep asking for it to be changed.
I understand that you don't want to open up the doors for changing the rules every 5 mins but this is a minor alteration that surely can't have that much effect on the general flow of the website.