EDIT* The tank is 20 gal and was newly set up so they were in 20 gal of chlorinated water, the filter was already established though, but not anymore I guess
Both chlorine &/or chloramine can cause long term damage & even death to fish, in some cases even at low levels, especially young fish which are typically the most sensitive to these substances. Free chlorine can cause acute necrosis of gill tissue in fish, the younger the fish the worse it can be. Chlorine also has the potential to wipe out most/all of the bio bacteria in your system. In short chlorine is an oxidizer, that can burn a fishes gills & kill cell tissue, sometimes even at low levels.
The amount of chlorine and or chloramine can vary greatly from one municipality to another, as well as from one tap to another, and can also vary greatly throughout the season. There is no way that anyone on a public forum can possibly attempt to tell you what a "safe" amount is in your case unless they have the specs on your local tap water at your user end. Water temp, pH values, etc can also determine how 'safe' certain levels of known toxic substances can be on fish.
The worst part was/is that your new set up was void of organics which could have helped greatly reduce any potential harm to your fish, or at least kept the exposure time much lower. According to Edward Noga, a professor at NC State that is considered an expert in the disease & health of fish, chlorine levels as low as 0.10 mg/l (which is common in many tap water levels) can be accutely fatal in aquaria that has low levels of organics. (as in a new set up) Others have found residual chlorine levels as low as 0.05 mg/l to be toxic to certain species of fish.
There has been a TON of work done in this area over the past few decades, by a number of researchers, such as Tompkins & Tsai (1976) for anyone that is interested in understanding just how toxic both chlorine & chloramine can be in a closed system that contains fish.
The fact that your fish haven't died is positive, but that in no way means that the exposure hasn't caused damage to your fish.
And to Water & Alan, I have personally seen a number of cases where total die off of bio-bacteria as well as fish, was caused by a large water change, where the owner forgot to add dechlorinator. I have also seen total (or at least a massive amount) of BB killed off by rinsing filter media in tap water. Enough to cause a major ammonia spike in a semi established system.
In my non-expert opinion, both of you are most likely killing a certain amount of BB every time that you rinse your media in your tap water. In your cases it may not be enough to throw your system into a mini-cycle, but both chlorine & chloramine have the potential to totally wipe out the bacteria in an aquarium. Again, this will be dependant on numerous factors, and can vary greatly from one hobbyists situation to another, and what may be safe for the two of you, could prove to be a disaster for the next person a few cities or towns over.
With my local tap water (2 mg/l chloramine) I rinse my media in a bucket of clean, dechlorinated water. Takes a bit more time than simply rinsing under the tap, but ensures that I am not killing off one of the most important parts of my closed system.
Unfortunately the LC50 (survival time) of most tropical species of fish placed under various levels of chlorine/chloramine has yet to be established, but the bottom line is both substances are highly toxic to most aquatic organisms.
From the EPA ....
http://www.epa.gov/chemfact/s_chlori.txt
V. ENVIRONMENTAL EFFECTS
A. Toxicity to Aquatic Organisms
Chlorine has high acute toxicity to aquatic organisms; many toxicity
values are less than or equal to 1 mg/L. Twenty-four-hour LC50
values range from 0.076 to 0.16 mg/L for Daphnia magna (water flea)
and from 0.005 to 0.1 mg/L for Daphnia pulex (cladocern)
(AQUIRE 1994); 48-hour LC50 values range from 5.3 to 12.8 mg/L for
Nitocra spinipes (snail); and 96-hour LC50 values range from 0.13
to 0.29 mg/L for Oncorhynchus mykiss (rainbow trout), from 0.1 to
0.18 mg/L for Salvelinus fontinalis (brook trout), and from
0.71-0.82 mg/L for Lepomis cyanellus (green sunfish) (AQUIRE 1994).
Papillomas of the oral cavity in fish have been associated
with exposure to chlorinated water supplies (NTP 1992).
Low level chlorination (0.05 to 0.15 mg/L) results in significant
shifts in the species composition of marine phytoplankton
communities (HSDB 1994).
Best course of action to reduce stress from exposure to chlorine/chloramine is add a small amount of sodium chloride (salt) which will help reduce osmoregulatory stress, increase aeration (the more the better), and reduce the temp of the tank a few degrees to also improve dissolved oxygen levels.
HTH