The presence of insects does not necessarily indicate that water is safe to drink. In general, you should assume that any water in nature is unsafe to drink without boiling unless you don't have another option. (Your options will sometimes depend on where you live; many societies drink unboiled well or river water, but this is not advisable in areas with large human or farm animal populations for reasons of disease). Indicator species like these can be found in purified waterways, the water is clean but NOT sterile. It might be pumped to rivers or lakes or the ocean fairly shortly after this step but to go into the drinking water supply it needs to be further sterilised. It will be treated via heat, chemicals, filtration or all three, and tested for the presence of known local pathogens before being pumped into homes and soforth. (When I went to the USA a decade or so ago I found the water there absolutely fucking disgusting, because the places I went to sterilise their water with high concentrations of chlorine and leave it in there. I don't know how people can live on the stuff.)
The standards of quality will depend on where you live, and how good an indicator one of these species is for safety will vary depending on the water source; river water far away from humans and farms will be safer (although not really safe) than town runoff being purified, in terms of the likely presence of pathogens, even if indicator species are present at both. But these species are very helpful in places that lack the infrastructure to test or purify their water to our standards, as well as being a handy eyeball test for water purification efficacy. You have to work with what you have, and sometimes what you have isn't perfect.