I'm not familiar with Photomodeler at all, but I'd think the key here would be to determine what the program is designed to use. In other words, does it actually utilize the CPU to do it's calculations, or does it prefer to use a GPU accelerator to do that. The processing units in GPU's work very differently, and theoretically can compute things faster (this is why Folding @ Home uses GPUs).
If the program does indeed use GPU capabilities, make sure you don't get something with integrated graphics (which many store bought PCs use).
I'd say get an LGA1366 i7. ith it's high memory bandwidth, the memory will take care of itself. Then, if the program is written to take advantage of CUDA, get the most powerful GTX you can afford. If the video card only going to drive the monitor, the video card that you choose will not be that important.
Core i7 is a mammoth is the CPU market. What you buy depends on what you can spend. i7's are expensive but Intel's Core 2 Duo are almost nearing its end. So its your choice, or you can go towards the AMD side.
The price is not so important. I just need the CPU that fits best to my application, as this computer will be dedicated to that. And I don't know if what I need is intel i7 or intel core 2 duo. My main question is: being Photomodeler non multi-threaded, is it better to use 4 cores with lower clock frequency (i7) or 2 cores with higher clock frequency (core 2 duo)?
For 3D applications and multitasking definitely a Quad would be good. I suggest Core i7 or i5, especially since budget isn't a problem for you. If you game, then you wont see much difference in performance than the Core 2 Duos. But for apps, definitely the i line-up of Intel is a good choice.
If they claim that it is not multithreaded, the extra cores won't help you. An i7 is faster on single threaded tasks than a Core 2 as well though. I'd go with an i7-860 if you don't want to overclock - it has excellent turbo modes to help with single threaded tasks, and is awesome at multithreaded stuff too if you ever need that.