Not really, the letdown people speak of is of how little increase there is in performance. 5-10% performance increase over Ivy Bridge is/was underwhelming, plus Haswell runs hotter so can't OC as high on similar cooling, however, the slight increase in performance basically makes them even when OC'd. (i.e. 4.5ghz Haswell performs about the same as 4.8ghz Ivy)
If starting a fresh build or doing a major generational jump (as you are), I'd say to go with Haswell, although 5-10% isn't huge, its still a performance increase. Those running Sandy or Ivy bridge have no reason to upgrade to Haswell, its just not worth the money for them.
If you want to overclock the CPU as much as possible, go with Ivy Bridge.