Nope, that is completely normal.
PSU's are comprised of capacitors which verify that the voltage going to the "rails" or the components in your pc are always the same. Capacitors are similar to a battery, in that they work by storing up energy in a cell, and releasing a set amount. This is mostly to protect against voltage spikes or droops coming from your wall.
So, even though you have no electricity coming into your PSU, the capacitors are still charged, which is where you get the minor electrical current when you press the power.
This is further proved by how as time goes on, there is less power remaining when you press the button.
So, long story short. That is normal operation of a PSU.