What resolutions are supported by your security camera?
What resolutions are supported by your flat screen TV?
If both security camera/DVR and flat screen TV are set to the same resolution then that should resolve the mismatch.
Check the respective User Guides/Manuals (or go online) to learn how to find and select the resolutions.
As for the connectivity: HDMI is digital, RCA is analog. You will need the applicable converter.
E.g.:
Camera DVR -----RCA cable ------>[converter] <--------HDMI cable --------> TV