In my opinion, no. You probably won't hear much (if any) difference going from an Audigy2 to an X-Fi. The Audigy2 is still a very good card. I have an Audigy2 ZS in my old computer but when I built my new one, I went with the X-Fi just because it was the latest and greatest and I didn't notice any difference at all except the X-Fi drivers were more fussy.
It's really all about the speakers attached that is where you will get the difference. My 5+ year old Cambridge Megaworks 550 5.1 speakers are still kickin' and work great for me but might have a different opinion if I had a set of 7.1 speakers.