I swim competitivley so, I'm always in the pool, somtimes everyday and I never have a problem with my infusion sites faling off. I use Cleo 90 infusion sets and use Tegaderm over them. I use an Ping pump so I can take it in the pool with me and there's never been any problems. Other than being able to see the bump on my stomach from the set and having to explain to people what "that bump under my suit" is.
I use Cleo 90's, but I've never tried putting tape over it. Do you cut a hole in the Tegaderm and slip it over the small center part? Do you find that the sets don't stay in when you don't use the tegaderm? I ask b/c I've yet to have a Cleo fall out from sweatiness/wetness. They have been great for me in all conditions.
Personally I've never had any more trouble with sites coming out from being in the pool or ocean than I have from just being active outdoors. When I was in my teens I used to have a lot of trouble keeping sites in during the summer. There are a lot of fixes and I tried everyone: IV3000, Tegaderm, Mastisol, shaving the area, antiperspirant on the area, different types of sets. You just have to find what works for you. As I got older for some reason it just got easier to keep sets in (sweat less?) and I don't put any type of tape over sets anymore even in water. I don't even wear the plastic cap on the infusion sets.
Bottom line is if the tape on the set is loose before you get in water it'll be more likely to fall off, otherwise if it's on snug before getting in it should stay on fine.