If they are the same processor and only the cache differs, then the one with the larger cache will be a bit faster. If they are different processors, then check benchmarks to find out which one is faster, i.e., the one with the smaller cache isn't necessarily the slowest.
Caches are very important for performance, but there's a law of diminishing returns for them. The first MB gives you the most benefit, the next MB not nearly as much.
GhislainG is right in that performance depends on the processor architecture and clock speed as well the cache.