The SPDIF jack was designed around people doing that on accident and on purpose, the SPDIF jack is longer than a 3.5mm headphones jack so you will not end up damaging the LED or sensor. Some systems are setup that they will allow copper connections (normal 3.5mm) in the SPDIF (optical) port, as they are the same size.
I think you may be referring to TOSLINK.