You can only use DDR3 memory if your motherboard specifically supports it. The CPU/processor will work fine with DDR3 as long as it is supported by the motherboard. What make and model of mobo are you using?
P35 is the Intel chipset that your Asus mobo uses. Some P35 mobos do support DDR3. So, to verify compatibility, we need the actual model number. CPU-Z can help identify your mobo.
For your other question, changing from DDR2-800 to DDR2-1200 isn't worth the money. To take advantage of the faster memory, you would have to overclock your system. Overclocking memory only offers marginal overll performance gains and often lead to an unstable system.
How much memory do you have in your current system?