In my opinion, I'd say that a Celeron D could work, and given how marginal any increase to a Pentium 4 would be, it would not be worth the effort.
Either way, that says you best the minimum requirements, which is good. I've found that it runs great (mostly) with my CPU, an Athlon64 2800+, overclocked from 1.8GHz to 2.0GHz. At best, that might only be half again as good as a Celeron D, so... At any rate, the main part of the game that's CPU-bound is towns; however, very little of the game's combat takes place in towns, so you needn't worry too much about performance here.
Another big question is RAM. Contrary to what the others have said, the game is actually quite playable, and even smooth, with as little as 512MB of RAM. I still recommend all you can get, though; loading times can be painful. However, they're not too much different from what I've seen for loading times for Morrowind on the original Xbox. Also note that if you plan on doing editing on the PC using the Elder Scrolls Construction Set, 1GB of RAM is highly recommended if you want to do actual terrain/dungeon editing, and 2GB is highly recommended if you want to use everything without fear of slow performance or even crashing.
Perhaps the big question that got largely ignored is your graphics card; the vast majority of graphics cards out there today (not necessarily found in gaming computers) will not play the game; SM 2.0 support is a must, though programs such as Oldblivion can allow it to run on cards as old as SM 1.1. 256MB of video RAM is recommended if you want to run with maxed textures, and if you plan on using mods to improve textures further, 512MB isn't insane. Likewise, the game is extremely heavy on pixel-shaders, so I'd recommend making sure your card is powerful in that area; GeForce FX cards, with only 2-4 pixel shaders, are laughable in performance compared to the Radeon 9500 through 9800 cards, with have 4-8 pixel shaders.
Most areas of the game, particularly in the wilderness, are dependant on the graphics card most for performance; for instance, if you happen across an oblivion gate in the middle of the forest, that CAN actually bring your framerates into the single-digits. The wilderness is also the harshest-performing part of the game, and where a good chunk of the combat will take place. So make sure you get a good video card, or plan on adjusting the settings to compensate.