compared to the core 2 duo I'm running now, even without OC this should be fine for a good while. However seeing how I don't plan on OC could I remove the cooler? Or should I have that anyyway to keep temps down.
That's where it gets a bit tricky - even on stock speeds Ivy runs a bit hotter than Sandy does - a fan like a Hyper 212 will greatly reduce system temperatures but you can always add one later - the case you chose makes it really easy to do so. I think as long as you set up your cooling solution properly you should have no problems.