No, you eat a bush and go up a level of speed, the only way to lose a level is to take damage.

A better More balanced way would be to also lose a level after going a certain amount and or a specific time, like sprinting or speed boosts.

Also there's other great suggestions in the post other than just horse speed.