Depends what kind of music you listen to. For example, classical music has up to 60dB peaks. 60dB = 64 times the power required. So if your listening level averages to 1W, you need ~70W/channel amplifier to avoid clipping.
If your amplifier clips repeatedly it can damage your speakers, as the voltage surge does not allow the voice coils to cool properly.