Do you know how small 20-60 nanometers is? That's the kind of dies and circuits being used in even crappy processors.
I don't doubt people could make some kind of processor at home, but it's going to be about as powerful as a computer from the 1950s with dozens of circuits instead of millions.
An Intel Atom circa 2008 has 47,000,000 circuits in a small 1-2" socket. You think you can do that at home? Heck, a 286 from the 80's has 134,000 circuits. That can't even run a gui and I doubt you could do that by hand in even the size of a refrigerator.