Map it out for yourself and find out!
x1% of core overclock = y1% higher frame rates or score in benchmarks a/b/c
x2% memory overclock at x1% overclock = y2% higher results
Then pick the combination of x1 and x2 that gives you the highest overall performance improvement. As others have said above, how much of an improvement you get will depend on how heavily the scenes you are using are stressing GPU memory bandwidth.