Light only moves through fibre at the speed of light... in the case of glass, that's around 200,000 kilometers a second. Given there's about 10,000km between you and Japan, that means there's a minimum of 50ms latency each way, or 100ms round trip. Your pings are about right.
I guess the phrase "don't look a gift horse in the mouth" applies regarding the 50Mb/s limit - though you may want to check that it's what you're actually paying for on the bill.