Larger cache will reduce access time assuming data has been cached.
If it is not in the cache, then the cache needs to be flushed which takes a little bit of time for new data to be written to the cache. The more cache, the more time it takes to flush and write new data.
Therefore, it is a "catch 22", if the data is in the cache, then you lucky and access time is low. If the data is not in the cache, then access time can be relatively high.