But CL6 is slower than CL5 right? So upgrading to CL6 can slow down my current ram too.
Can you please state whether there is going to be a noticable change or can I just ignore it?
Yes, CL6 is slower than CL5 but the difference is minute and you wouldn't notice it (we are talking milli and micro seconds here!) You can ignore it. I've never heard that CL slowing down both the memory modules but the frequency or clock speed will be downclocked when higher speed mixed with lower.