There is not really much difference if your talking about the dual cores, just more cache and Turbo boost. Also an excellent feature is the Hardware Encryption Acceleration, it improves encryption performance unbelievably. However this is not really required for most people.
If your talking about the i5 quads then they are a bit more different. They don't include hyper threading and feature 4 cores, they also don't include the features I listed for the dual core i5s.
Yes, it depends to the Core i5's you want to compare to the i3's. If you're comparing the i3 to one of the i5-6xx series, then there is no big difference (the i5-6xx are duals with hyper threading, just like the i3's), same amount of L3 cache etc.
But if you're comparing to the i5-7xx CPU's (there's actually two of them, the i5-750 and 760), then there is a bigger difference. The i5-7xx have 8MB L3 cache, opposite to those 4MB L3 on the i3's and i5-6xx's, and are true quad cores, thus giving them a bigger advantage in multithreaded apps.