Help on building small pc for kids?

Most kids educational games could probably be pulled off with integrated graphics (reading rabbit and so forth), but I would splurge for some kind of dedicated card so as he gets older he can play some more advanced games like Civ 5 or whatever.

I'd suggest an intel 1155 so you could give him your 2500k if you upgraded to Ivy Bridge. I'm also going for cheap since a 4-year old kid will not appreciate the difference between a $300 build and a $1000 build.

(All prices from newegg.com)
CPU: Intel Celeron G530 - $56.99
MoBo: ASRock H61M-VS - $53.99
RAM: 4GB (single stick so you can expand later) DDR3 1333 1.5V CAS 9 ~$23
PSU: CORSAIR CX430 V2 - $20 after all rebates and discounts
GPU: PowerColor AX5570 1GBD3-H - $39.99 after rebates
HDD: Seagate Barracuda ST3160316AS 160GB 7200 RPM 8MB Cache SATA 6.0Gb/s 3.5 - $34.99
Optical: whatever - $20
Case: APEX TX-381-C (has a handle, but you could pick something else) - $32.98

Total - $311.94 after rebates including shipping.

You still need an OS and Windows 7 Home Edition OEM costs ~$99

This could play Civ 5 and should handle any educational game you throw at it. You could swap the case for anything you want, that one just seemed convenient and cheap.