What's the hardware difference between hyperthreaded and non-hyperthreaded CPUs?

Let's say you are gaming on an i5, and all of you 4 cpu cores are utilised by the game. Suddenly windows 10 decides to run some irrelevant bloatware out of the blue, this new software wants some CPU time but all your 4 cores are occupied. What happens is that the CPU starts switching (very fast) between one of your game threads and this new software. This will create the illusion that the game and the bloatware are running at the same time.

All seems to be ok right? In theory, you should just lose a few fps at most especially if the bloat software isn't that CPU intesive. However, what happens in reality is different. Each time the CPU switches between one of the the game threads and the bloatware, it has to copy and keep track of a lot of stuff called "program context". This in turn introduces a latency called context switching cost, which wastes CPU time. One benefit of hyperthreading is that it reduces this cost for a fixed number of thread. The other optimisation is that when two threads want a core, it allows a secod thread to use the part of the core that is not utilised by the first thread (threads of a program do not necessarily use all satges of a core at the same time).

So while the computational power of the CPU should theoretically be the same, hyperthreading makes your CPU more efficient at running programs and increases the utilisation. So in practice, the bloatware is less likely to cause hiccups and stutters when you have more threads. Change the blaotware in this example with any other program you might run while gaming, encoders/recorders/streaming/etc. Or if the game you are running can utilise more than the number of physical cores that you have, threads of the same game can benefit from hyperthreading and allow your game to better utilise the CPU.

This was of course, a very simplistic explanation, but it should give an idea of why hyperthreading is useful. This is why I always suggest Ryzen 5 over i5 especially to gamers. In a real world scenario you rarely run just the game in a clean install of windows, and this will result in performance hiccpus not reflected in online reviews. If you have a couple of more threads(or even better, cores) available, you won't have to worry about other processes running while you game.

Intel unfortunately disables the hyperthreading on chip for i5, they disable it on hardware level on the chip itself so bios flashing and the like doesn't help. Recently they stopped disabling it for some pentium chips and others, but for i5 they still disable it on purpose. This is to purely to make more money, they literally damage the chips they are selling for cheaper to drive the sale of the i7 chips for more expensive.

Fortunately, AMD has caught them with their pants down and took the Ryzen out. This might make intel stop disabling the hyperthreading for i5. Unless people just all buy their 7700k, in that case they still won't do it.

It's cheaper for Intel to build a single chip and disable various parts of it to fill all of the different market niches - Core i5's, (quad) i7's and 1151 Xeons are all the same CPU. i5's have some cache and ECC disabled, i7's have ECC disabled, and the Xeons (usually) have their integrated graphics disabled. All CPUs are capable of being fully unlocked, but only "K" models have that enabled. Intel is also able to "harvest" some dies - that is, a certain number of dies cut from the wafer will have a defective portion in their iGPU, making them trash for use as a desktop CPU, but fine as a Xeon, where that's disabled anyway. Ditto with the cache.

I'm a bit vague on the hardware specifics of SMT, but my excuse is that modern CPUs are hideously complex.

Would you prefer Intel only sold fully enabled CPUs, likely for no less than current i7's (because harvested dies save everyone money), and had no options for lower end parts? Or, that all of them were more expensive because Intel had to design a unique hardware configuration for each model, and build a manufacturing line for each?

The cost to build each CPU is actually very small. Most of the costs come from the design of the CPU itself and the development of new process nodes to build them on.

To make a bad car analogy, this is very much like how a Toyota and Lexus are basically the same car underneath. Toyota might retune the engine for slightly higher output or add some extra plastic inside to make it look nicer, maybe slightly better seats or suspension, but mostly they're made of the same parts and the cost to build isn't much different. The fact that they share design makes both of them cheaper, and that a Lexus can be sold at a higher profit margin means you can get a Camry for much less than you could if it were otherwise designed from scratch. This is one of the central pillars of industrial design and manufacturing.