The best way I've found is to get a big pleco. Carefully watch the tank until the fish you're trying to measure swims next to it, make note of the relative length.
Wait for the big pleco to suck onto the side of the tank and then quickly put a measuring tape next him. Marking compared to your arm is also a viable option and your arm is less likely to get startled and swim away while you try to read the measuring tape.