1. Remove stock fan and thermal paste
2. Add thermal paste of about the size of a pea
3. Put the new CPU cooler on the CPU and it will squish the paste down across the CPU
NOTE: If you have a CPU cooler like a Hyper 212 there will be trenches on the copper plate that touches the CPU. Put some thermal paste in between those trenches for maximum cooling.
As for overclocking the RAM, you are taking it pretty far and overclocking RAM can corrupt things such as the BIOS causing serious issues, so be careful.
What are your RAM timings/voltages and what does it say under your SPD tab in CPU-Z?