The smaller the fish, the less likely it is to be harmed by this practice. And, even with larger fish, there are plenty of serious anglers who release them by "torpedoing" them into the water, with the intention being to give the fish a blast of freshly-oxygenated water through the gills to speed up recovery.
But, I gotta say that I just find it distasteful and...disrespectful, for lack of a better word. I won't do it. If I intend to release a fish...which is my intent with the vast majority I catch...I will hold it and make sure it is breathing normally if I have any doubts before gently letting go. I won't even drop it into the water; if I am still physically capable of fishing, then I can still bend over and place the fish back in.
It's reminiscent, to me at least, of the way some hunters grab the antlers of a dead deer and casually swing the head around while taking a trophy pic, then drop the head afterwards like a sack of potatoes. I won't do that either; yes, I know it's dead, but...once again, it's a "respect" thing.