In the purest sense, a programming language should be able to represent any algorithm you can in mathematics, and they are one in the same. In reality, you have to deal with limitations in modern computing architecture. As such, you end up with algorithms that are not easily expressable. You also end up with classes of algorithms that really only make sense when the computing environment is taken into consideration.
Some of these problems can be studied academically (you're likely familiar with the P != NP "problem") and others you'll pick up along the way through experience.
My recommendation is to use a "purer" language if you're looking to closely model mathematical entities. Lisp or ML are probably good candidates. If you want something that's more popular, and thus more widely supported, Python is a good pick.
You certainly don't want to be doing assembly. It has its place for sure (I have to use it quite frequently), but this is not it. I've not used Delphi and have heard good things about it, but it's likely not the most suitable language for what you want to achieve either.