They already do and on a much grander scale. Shader units or "stream processors" are the "cores" that are present in a GPU. CPU cores are huge and designed to handle a large number of complex instructions while the cores on a GPU are much smaller and designed to handle a limited number of tasks specific to graphic processing very quickly working in parallel on a massive scale. For example the GTX 580 has 512 and the HD6970 has 1536 on each card(Nvidia uses larger, more complex stream processors and AMD a larger number of smaller, simpler ones.)