Should be OK, will make a mess when you put it in unless well rinsed.
Buying bulk gravel from a garden center might be a cheaper option. Stilll dirty, but can be prerinsed in the driveway, if you get a delivery. Or, if you have a truck, get a ton or yard in the back and wash it will still in the truck.
Or, buy bags of pea gravel or wash river gravel from any Menards or such. Stab some holes in the bags and put the hose in it.
The best way to clean it is to put it in the pond during construction (or if possible drain the pond) so you can wash it a couple of times, put a trash pump in the low spot, and pump it out a few times. Repeat if necessary.