No one book is going to get you there. There are a few concepts you need to really understand what's going on. There were three classes I had in college that addressed this kind of stuff.
Predicate Logic, which forms the basis of binary systems. Pretty easy stuff, could probably knock it out in an afternoon or two with the relevant Wikipedia articles.
Digital Logic Circuits, which is about building systems out of logic gates (ultimately how CPUs are designed and built). Much less easy stuff if you do more than just read the Wikipedia page. There are some free programs out there you can use to design and test virtual systems you built out of the various logic gates. Just building something like a simple 8 bit adder can teach you a surprising amount about how this all works.
Finally, Computer Architecture, which is about the actual problems involved in producing complex, efficient processors. Senior level class, one of the hardest I've ever had. Teaching yourself this is about on par with teaching yourself Calculus, I'd say. Not impossible, but most people who claim to have done so are full of crap or think reading a couple Wikipedia articles and remembering one or two terms counts as knowing the subject.